Mapping your data for AI: structured and unstructured
Most AI projects stumble upstream, on the data, not on the model. Here is the method for reconstructing your real processes, inventorying your sources and scoring how usable they are before you write a single line of prompt.
The real starting point
Why do AI projects fail upstream, on the data?
When an AI assistant hallucinates, when an automation breaks in production, when a promising use case sits stuck in POC limbo for six months, the team almost always looks in the wrong place: the model. The problem is most often upstream, in the raw material. Foundation models (Claude, Gemini, ChatGPT, Mistral, or hosted models) are now more than good enough. What is missing is precise knowledge of what the data you feed them is actually worth, and of the processes you claim to automate. Sector studies converge on this point: the majority of data/AI project failures come down to the quality, availability and governance of upstream data, not to algorithm performance (recurring analyses by Gartner and other analyst firms on enterprise AI adoption).
You automate a process as it is described in a procedure, while the field actually runs three variants of it, with manual rework and exception cases. You plug a RAG pipeline into a document repository assuming it is up to date, when it contains two contradictory versions of the same policy. You assume a right of use where the supplier contract forbids it. Data mapping for AI is the step that makes these blind spots visible before they cost you a project. It is also the prerequisite for a credible AI roadmap prioritised by ROI: without knowing how usable your data really is, any prioritisation remains speculation.
A field example. At an insurer, a use case for automatically sorting inbound mail looked ready: the corpus existed, the model classified well in the demo. The data map revealed that 38% of the PDFs were non-OCRed scans and that two category nomenclatures coexisted depending on the year. Three weeks of data preparation preceded any move to production — a detour that avoided a deployment that would have misfiled one letter in three.
What is a data map for AI?
A data map for AI is a reasoned inventory of every source a use case will consume or produce, linked to the process steps those sources feed, and paired with a usability score. It is neither a technical data dictionary nor an architecture diagram. Its central question is not “where is the data?” but “can this data be used by an AI, at what preparation cost, and under which rights?”.
What is the difference between structured and unstructured data for AI?
Structured data lives in explicit schemas: ERP and CRM tables, relational databases, APIs, the data warehouse. It can be queried in SQL and naturally feeds analytics and governed conversational BI (GenBI). Unstructured data accounts, according to the sector estimates regularly cited by IDC and Gartner, for around 80 to 90% of a company’s data volume: contracts, emails, meeting minutes, support tickets, scanned PDFs, images, recordings. That is where a large share of the value for generative AI lies, and it is also where the traps hide: multiple versions, poor-quality OCR, personal data scattered everywhere, missing metadata.
Process mining: why work from real logs rather than declarative accounts?
Process mining is the reconstruction of a process from the traces it leaves in your systems: event logs, timestamps, status transitions. From a minimal triplet — case ID, activity, date/time — you reconstruct how a process actually unfolds, with all its variants, its rework loops and its bottlenecks. The difference with a declarative approach (workshops, interviews, procedures) is radical: the declarative describes the process as people believe it works; process mining shows how it really works. On a single order-handling process, it is not unusual to discover 15 to 40 execution variants where the official procedure allowed for only one. For AI, this distinction is decisive: you cannot reliably automate a path you have never observed.
What are the prerequisites and limits of process mining?
Process mining is not magic. It requires complete, timestamped logs: a poorly instrumented process, whose steps leave no usable dated trace, cannot be reconstructed correctly — you inherit a partial, sometimes misleading view. Log quality conditions everything: missing case IDs, timestamps that are inconsistent across systems, or poorly labelled activities all degrade the result. On the compliance side, logs often contain agent or customer identifiers: processing them falls under the GDPR and requires minimisation, a legal basis and controlled retention periods. Finally, tooling matters: platforms such as Celonis or Disco, or open-source libraries such as PM4Py, serve different maturity levels. Our process modelling & data mapping offer treats both strands — process mining and source inventory — as a single foundation, prerequisites and limits included.
The method
How do you map your data for AI, step by step?
Start from the use case, not the data lake
An exhaustive map of the entire IT estate is endless and useless. Start from one or two target use cases and trace back only the sources they consume or produce. The scope stays under control and every inventoried source has a reason to be there.
Reconstruct the as-is processes through process mining
From the real logs of the ERP, CRM or ticketing system, reconstruct how the process actually runs: variants, rework, exceptions, lead times. Link each step to the data it reads and writes.
Inventory the structured and unstructured sources
List tables, APIs and warehouses on one side; documents, contracts, emails, PDFs and images on the other. For each source: owner, volume, freshness, format, presence of personal data.
Score the AI usability of each source
Five axes: availability, quality, usage rights, governance, technical usability by AI. The score reveals what is ready, what needs preparation, and what is a blocker.
Prioritise and cost the data-preparation work
Cross the value of the use case with the cost of preparing the data. You get a sequence: quick wins usable straight away, and data prerequisites to clear before industrialisation.
How do you score how usable a data source is for AI?
Usability scoring is the heart of the data map. It avoids the classic trap: believing that because a piece of data “exists” it can be used. A source can be available but without usage rights, or clean but ungoverned. Score each source on five axes, ideally from 1 to 5, and only retain, for a given use case, the sources that reach a minimum threshold on every axis — a single axis at 1 can be enough to disqualify a source.
| Scoring axis | Question asked | Warning signal (low score) |
|---|---|---|
| Availability | Is the data technically accessible, continuously, through a stable channel? | Manual exports, one-off access, undocumented API |
| Quality | Is it complete, fresh, free of duplicates and contradictions? | Empty fields, multiple versions, unreadable OCR |
| Usage rights | Do we have the right to use it for this specific AI processing (inference or training)? | Restrictive contract clause, missing consent |
| Governance | Identified owner, classification, traceability, GDPR? | No owner, unmapped personal data |
| AI usability | Does the format lend itself to extraction, RAG, classification? | Non-OCRed scanned PDFs, closed proprietary formats |
A worked scoring example, from score to decision
Take a repository of supplier contracts considered as the corpus for a legal assistant. Score it on the five axes, then decide.
| Axis | Score /5 | Finding |
|---|---|---|
| Availability | 4 | Document management system queryable via API, but a few contracts outside the system |
| Quality | 3 | Duplicates, and amendments not linked to their parent contract |
| Usage rights | 2 | Contractual confidentiality: internal use allowed, model training excluded |
| Governance | 4 | Identified owner (legal department), classification in place |
| AI usability | 3 | 70% as text-based PDFs, 30% as scans still to OCR |
Decision: the source is usable for RAG/internal inference after two preparation work streams (deduplication and OCR), but disqualified for any fine-tuning because of usage rights (an axis at 2). It goes into the indexed corpus, the preparation is scheduled, and the training ban is documented in the register.
The most underestimated axis is usage rights. Data that is technically perfect but legally barred from training or inference use is unusable data. This is the direct link with AI governance (GDPR, AI Act and the trust layer): the scoring builds compliance into the data map from the start, not after the fact.
How does the data map serve AI Act compliance?
The European AI Act classifies systems by risk level — unacceptable, high, limited, minimal — and imposes concrete obligations on high-risk systems: technical documentation, governance of training and test data, traceability and logging. The data map feeds these requirements directly: it provides the inventory of the sources that feed a system, their provenance, their usage rights and their quality — precisely the elements expected in an AI system register and in compliance documentation. Mapping upstream means assembling the substance of your AI Act file, instead of reconstructing it in the rush of an audit.
How do you prepare your data for AI?
Preparing your data for AI does not mean the same thing depending on the intended use. Two regimes coexist, with markedly different stakes in terms of rights and preparation:
- RAG and inference: your documents are indexed so that an LLM answers from them. The data does not enter the model’s weights; the stakes are corpus freshness, chunking, metadata and a read-only right of use.
- Training and fine-tuning: the data is durably absorbed by the model. The rights requirements are far stricter (consent and contract clauses must explicitly cover training), and the quality of a training set is judged on representativeness and labelling, not just readability.
Most enterprise use cases today rely on RAG rather than fine-tuning, because it is faster to industrialise and simpler to govern. That is the regime where preparation effort most often goes.
How do you prepare unstructured data for a reliable RAG?
RAG (Retrieval-Augmented Generation) is the architecture that lets an LLM answer from your documents rather than from its training memory alone. Its quality depends almost entirely on the quality of the indexed corpus — the most direct illustration of the “garbage in, garbage out” principle. A RAG pipeline does not fix bad data: it amplifies it by giving it the authority of a generated answer. If your repository contains two contradictory versions of a policy, the RAG will cite one or the other depending on which chunk happens to be retrieved, with the same confidence.
- Deduplicate and version: a single source of truth per document, with obsolete versions removed from the index
- OCR and clean scanned PDFs before indexing; unreadable text yields useless chunks
- Enrich with metadata (date, scope, owner, confidentiality level) to filter retrieval
- Chunk coherently by logical section, not by an arbitrary character count
- Exclude from the corpus any personal data not needed for the use case, in line with the scored usage rights
- Put evaluations (evals) in place that measure answer accuracy against a reference question set
The data map is what makes this work fundable and plannable: it identifies precisely which repositories enter the corpus, which need cleaning, and which must be excluded. Taking the RAG to production — versioned index, evals, monitoring — then falls under industrialising an AI POC.
The orders of magnitude to keep in mind
The percentages above are orders of magnitude commonly cited by sector analysts (IDC, Gartner), not measurements taken on any given scope; they serve to calibrate the effort, not to ground a decision.
Lessons from the field
Which pitfalls should you avoid when mapping your data?
Mapping the entire IT estate
A “big bang” map never finishes and informs no decision. Scope by use case, then extend. Two well-understood sources beat fifty catalogued superficially.
Trusting the declarative
Workshops describe the idealised process. Without process mining on real logs, you automate a path that does not exist and miss the variants that will break production.
Confusing “exists” with “usable”
Data that is present is not data you can use. Without scoring on the five axes, you discover rights, quality or governance problems once development is already under way.
Treating compliance as an afterthought
Usage rights, GDPR and AI Act bolted on at the end force costly rework. The scoring assesses them at inventory time, which secures the move to production.
Forgetting the unstructured
Sticking to ERP/CRM tables means ignoring most of the material — and the highest value for generative AI: contracts, emails, tickets, PDFs.
A map with no owner
A map without an owner per source goes stale within months. Data governance is part of the deliverable, not of some later phase.
What are the deliverables, timeline and budget of a data map?
A useful data map is more than a diagram that looks good in a meeting. It produces artefacts that drive the decisions of the following weeks and serve as a reference for the whole AI team.
- The as-is process maps reconstructed through process mining, variants and bottlenecks included
- An inventory of structured and unstructured sources, linked to process steps
- The AI usability score per source across the five axes, with decision thresholds
- The prioritised list of data-preparation work streams, sized in effort
- The compliance prerequisites (usage rights, GDPR, AI Act) to clear before industrialisation
- A use-case sequence: usable immediately vs blocked by a data prerequisite
Indicative timeline and budget. Scoped to one or two use cases, a data map generally takes two to four weeks. The budget sits in the range of a short, fixed-price scoping engagement — a few consulting person-days, not a multi-month programme. The exact cost depends on the number of processes to mine and sources to score; it is set in the quote before any commitment, with no time-and-materials drift.
This is exactly the scope we equip in the Process modelling & data mapping offer. To start lighter and test the approach on one use case, the AI assessment & scoping offer — a short, fixed-price engagement with no commitment — sets the first milestone.
Frequently asked questions
How long does a data map for AI take, and how much does it cost?
Scoped to one or two use cases, a data map generally takes two to four weeks. Budget-wise, it sits in the range of a short, fixed-price scoping engagement — a few consulting person-days, set in the quote before any commitment, not a multi-month time-and-materials programme.
What data do you need to provide to start process mining?
The minimum is one event log per process in scope: case ID, activity label and timestamp. These extracts come from the ERP, CRM or ticketing system. The exact scope is defined during scoping, within a strict confidentiality framework and in line with the GDPR for logs containing identifiers.
Why map unstructured data too?
Because, according to IDC and Gartner sector estimates, it accounts for around 80% of a company’s data volume and concentrates a large share of the value for generative AI: contracts, emails, tickets, PDFs, images. Its usability (extraction, RAG, classification) is assessed on the same footing as ERP or CRM data.
How do you score how usable a data source is for AI?
Each source is scored on five axes: availability, quality, usage rights, governance and technical usability by AI, ideally from 1 to 5. A low score on a single axis — often usage rights — can be enough to disqualify a source for a given use case, or to restrict it to RAG while ruling out fine-tuning.
Should data be prepared differently for RAG and for fine-tuning?
Yes. With RAG, the data is read at inference time: the stakes are freshness, chunking, metadata and a read-only right of use. With fine-tuning, the data enters the model’s weights: the rights requirements are far stricter and quality is judged on representativeness and labelling. Most enterprise use cases today rely on RAG.
What is the difference between declarative mapping and process mining?
The declarative approach (workshops, procedures) describes the process as people believe it works. Process mining reconstructs it from real logs and reveals the actual variants, rework and exceptions. To automate reliably, you must observe the real path, not the idealised one — provided you have complete, timestamped logs.
Move from experimentation to AI in production
Start with a short, fixed-price diagnostic: maturity, high-ROI use cases, and a prioritised roadmap. No commitment.