Data foundations

Why do AI projects stand or fall on data?

Because AI is a magnifier, not a filter. Earlier generations of software mostly refused bad data: the report broke, the field validation complained, a human noticed. A language model refuses nothing. Feed it contradictory, stale, or incomplete material and it produces polished prose from exactly that, delivered with the same confidence as truth. Organisations discover this in the first month of an AI programme, at which point the AI project reveals itself to have been a data project wearing a fashionable coat.

The pattern repeats across sectors and sizes: the model was fine, the demo was impressive, and production stalled on questions no vendor can answer for you. Which system is the truth about a customer? Who owns fixing the duplicates? Why do finance and operations report different totals for the same thing? These questions are answerable, and they are organisational before they are technical, which means they can be settled ahead of any purchase. Organisations that settle them early find every AI decision afterwards is cheaper, because each one inherits the settlement instead of reopening it.

What are lakes, warehouses, and lakehouses?

Once you know which data the work needs, it has to be organised somewhere. There are three storage philosophies, told apart by one question: does structure come before or after arrival?

A data warehouse demands structure first. Data enters only after being cleaned and shaped into agreed tables, which makes reporting fast, consistent, and trustworthy, and makes the warehouse rigid: adding a new source is a project, and anything unstructured, documents, emails, transcripts, has no natural home.

A data lake accepts everything as it comes, raw files at low cost, structure applied later by whoever reads them. That flexibility suits exploration and machine learning, and curation is what protects it: a catalogued lake is an asset, and an uncatalogued one is a swamp with good marketing.

The lakehouse layers warehouse-style guarantees, transactions, quality rules, fast queries, on top of lake storage, applying structure only to the data that needs it and leaving raw material raw. It has become the default pattern for new builds, and it matters for AI specifically because assistants and retrieval pipelines consume the unstructured material, policies, contracts, case notes, that warehouses were designed to exclude.

Very few organisations choose from a blank sheet. If you already have a warehouse, or several that disagree with each other, the realistic path is extension rather than replacement: lake storage alongside it for the unstructured material, with lakehouse-style guarantees layered on where they are needed. The warehouse carries on doing the reporting it is good at, and rip-and-replace migration is usually the expensive route to the same destination.

Text description of this diagram

The diagram shows three panels. Warehouse: a tidy grid of identical blocks, labelled "structure before entry: fast, consistent, rigid". Lake: scattered shapes of different kinds and angles, labelled "everything raw: flexible, cheap, wants a catalogue". Lakehouse: a row of tidy blocks above a dashed line with scattered raw shapes beneath it, labelled "raw underneath, structure where it is needed".

What is a data pipeline?

Extending a warehouse means building pipelines, and so does every other move in this stack. A pipeline is the connective tissue: an automated process that moves data between layers and applies meaning on the way.

How often it runs is a decision rather than a detail. A nightly run suits reporting and anything a person reads in the morning. A continuous pipeline carries events as they happen, which fraud checks and live personalisation need and a board pack does not. Freshness is priced accordingly, and buying it everywhere is a common and expensive reflex.

The engineering is mostly mature. The recurring failures are human, and there are three:

• Definitions drift. A pipeline encodes what "active customer" meant on the day it was written, and nothing tells it when the business changes its mind.
• Breakage is silent. A dashboard fed by a pipeline that stopped on Tuesday still renders beautifully, which is why monitoring belongs in the build rather than after the first incident.
• Ownership is vague. Pipelines have owners, or they have nobody, and "nobody" is the most common root cause in any data incident review.

When you fund a data platform, most of what you are funding is pipelines and the people who keep them honest.

Rabbit hole What actually happens when a pipeline runs

A pipeline is rarely one job. It is a set of steps with an order, run by an orchestrator that knows which steps depend on which, starts what it can side by side, and holds back the rest when a step fails.

Two properties do most of the work in keeping that safe. The first is idempotency: a well-built step produces the same result whether it runs once or five times, so an automatic retry after a network wobble is harmless rather than a source of duplicate rows. The second is the backfill, the ability to re-run a stretch of history after a fix. Without one, correcting a transformation leaves the old error sitting in everything the pipeline already produced.

Text description of this diagram

The diagram has two parts. The upper part, "one run: order, and what runs side by side", shows two extract steps drawn alongside each other, one from the CRM and one from finance, both feeding into "match records: the same person, one row". That feeds into "apply definitions: active means one thing", which feeds into "load: into the warehouse". The load step is highlighted and carries a loop arrow returning into itself, labelled "safe to retry: run it twice, still one set of rows".

The lower part, "many runs: the same pipeline over time", shows twelve small squares in a row, one per nightly run. The first three are plain and the remaining nine are filled, with a bracket beneath the filled stretch labelled "backfill: re-run over history after a fix".

Both words surface in rooms where engineers are explaining why a fix is quick or slow. When somebody says a job is idempotent, they mean it is safe to run again. When somebody says the backfill will take a week, they mean the fix was the easy part and the history is the expensive one.

Is ETL still relevant in an AI architecture?

Yes, almost all of it. Extract, transform, load knowledge transfers nearly wholesale to AI architecture: retrieval ingestion is an ETL pipeline that ends in a vector store instead of a warehouse, and the surrounding disciplines, scheduling, monitoring, ownership, lineage, apply unchanged. If you spent years around ETL, you are not starting again; you are pointing an old discipline at new material.

Two habits do need retraining. Transforms are now model-versioned: change the embedding model and the corpus needs re-embedding, a species of schema migration with no direct ancestor. And testing became statistical: retrieval quality is measured on samples of real questions, not proved with row counts and checksums. Everything else in your ETL instincts, especially the suspicion of silent failure, transfers at full value.

Rabbit hole ETL to AI pipeline term mapping

You knew it as	It is now	What carried over, what changed
ETL jobs	Retrieval ingestion pipelines (Module 2)	Extract documents, transform by chunking and embedding, load into a vector store. Scheduling, incremental refresh, monitoring, idempotent re-runs, and ownership apply unchanged.
ELT and staging raw data	The lakehouse pattern	Keep the raw material, recompute the derived. Re-embedding a corpus when the chunking strategy changes is the same move as re-running transforms over staged data.
OLAP cubes and data marts	Vector stores	A store purpose-built to serve one query shape fast. Cubes answered slice-and-aggregate; vector stores answer "what is nearest to this?".
MDM and the golden record	Definition settling before AI	"Which system is the truth about a customer?" is master data management by another name, and it still comes before any model sees the data.
Enterprise search, TF-IDF, BM25	Neural embeddings	The same job, finding relevant text, upgraded from word-matching to meaning-matching. The Module 2 demo deliberately runs the older method so you can watch the mechanics.
Batch schedulers and DAG orchestration	Still exactly that	Deterministic pipelines still run on orchestrators. The genuinely new thing is the agent (Module 4), which chooses its path at runtime instead of following a fixed graph.

What does "good data" involve?

Five properties, and each one is the result of a decision somebody can take before a single record moves:

• Accuracy. The value is true, because it was validated where it was entered, while the person who knew the answer was still there.
• Completeness. The value is present, because somebody decided which fields are genuinely required instead of making everything optional.
• Timeliness. The value is current enough for the decision it serves, which is the run-frequency question from the previous section answered one use at a time.
• Consistency. Systems agree, because each was pointed at one settled definition rather than carrying its own.
• Lineage. You can trace where a number came from and what transformed it, because the pipeline recorded that as it ran.

Beneath all five sits agreement on definitions. Ask a university how many students are enrolled and three systems can answer truthfully and differently, because each was built for a different purpose. The work is choosing which number the organisation means and giving that choice a named owner. Definitions are settled by people with the authority to settle them; the platform enforces the settlement rather than producing it. Taken before the platform, this is a series of short meetings. Taken afterwards, it is the same meetings with more systems in the room.

Text description of this diagram

The diagram shows one question, "how many enrolled students?", asked of three systems. Student records answers 14,120, counting anyone registered. Finance answers 13,860, counting fee-paying students only. The library answers 14,450, counting active accounts. Every answer is truthful and every definition is different. All three feed into one box: a single definition, settled with authority, which the platform then enforces.

Explore: the layered architecture

The stack below is the architecture this module describes, top layer closest to the people using it. Select a layer to read what it does and what keeps it sound. Every AI ambition in this course sits in the top layer, inheriting everything beneath it.

1. Consumption Where value appears

   Dashboards and reports, analysts running queries, machine learning models making predictions, and AI assistants answering questions over documents, the pattern Module 2 built. Everything at this layer inherits the layers beneath it. An assistant that answers well is usually evidence of sound work two or three layers down, which is where the effort that produces a good answer actually belongs.

2. Quality and governance Whether anyone can trust it

   The layer with nothing to buy and the largest call on people: agreed definitions, data quality measurement, lineage that traces where a number came from, and access control deciding who may see what. This is the layer that turns everything below it into something people act on. Settle a definition once here and every consumer above inherits the agreement, which is why an assistant reading from three systems is exactly as coherent as the work done at this layer. What it costs is analysis time to reach cross-organisational agreement and engineering time to codify that agreement in the platform, which is why it has both the longest reach and the longest lead time of anything in the stack.

3. Storage: lake, warehouse, or lakehouse Where data settles

   A data warehouse stores clean, structured, agreed-shape data built for fast reliable reporting; it demands discipline before entry. A data lake stores anything in its raw form, documents and logs included, cheaply and flexibly; a catalogue and a named owner are what keep it navigable. The lakehouse pattern combines the two: raw flexibility underneath, with warehouse-style structure and guarantees applied only to the data that needs them. AI work made lakes considerably more valuable, because models consume exactly the messy, unstructured material warehouses were built to exclude.

4. Ingestion and pipelines How data moves and changes shape

   A pipeline is automated plumbing that copies data out of source systems, reshapes it, and delivers it somewhere useful, on a schedule or continuously. The reshaping is where meaning is applied: matching records that describe the same person, standardising dates, resolving the three different definitions of "active". Monitoring and a named owner belong in the architecture from the first pipeline onwards, because a report fed by a pipeline that stopped on Tuesday still looks exactly like a report. Specify both when the pipeline is commissioned and the question never arises.

5. Sources Where data is born

   Operational systems doing their day jobs: the student records system, finance, the CRM, ticketing, spreadsheets kept by one determined administrator. Nobody designed these to feed AI; they were designed to enrol students or pay invoices. This is the cheapest layer to improve, and improvements here reach everything above: a required field instead of an optional one, a picklist instead of free text, one agreed code list shared between two systems. Small changes at the source outperform any amount of cleaning downstream.

When should you not build a data platform?

Start from decisions the organisation needs to make, trace each back to the data it requires, and build only the layers that path demands. That sequence is what keeps a platform proportionate, and it works at every scale. If your organisation has a handful of systems and questions a disciplined analyst answers in a spreadsheet each month, buy the smallest managed tooling that removes the manual copying, and revisit when the questions outgrow it. Eighteen months of lakehouse construction ahead of validated use cases is the expensive route to the same place. A modest, owned, trusted warehouse beats an ambitious swamp every time it is measured.

Next: Module 4, Agents and orchestration: what changes when AI systems act rather than answer.