Retrieval-Augmented Generation

Why do AI models invent answers?

A large language model is a system trained on an enormous amount of text to predict what words should come next. Training compresses everything it read into statistical patterns. The result is a strong grasp of the general and a weak grip on the specific. It knows how refund policies tend to read. It has never seen yours, and it cannot look anything up on its own.

Ask a question beyond its knowledge and the model rarely stops to say so. It completes the pattern anyway, producing a fluent, confident, wrong answer. This failure is called hallucination, and it follows directly from how these systems work. No vendor will patch it away next quarter. The practical question for any organisation is architectural: how do you connect a model's language ability to facts you can trust?

What is retrieval-augmented generation?

Retrieval-augmented generation changes the exam conditions from closed book to open book. Instead of asking the model to answer from memory, the system first searches your own content, policies, manuals, contracts, case notes, for passages relevant to the question. It then hands those passages to the model along with the question and an instruction: answer from this material, and say so if the answer is not in it.

The model still writes the reply, which is why answers stay readable and conversational. Your documents supply the facts. When the pattern works, every claim in the answer can be traced to a retrieved passage, which also means the system can cite its sources and a human can check them.

Text description of this diagram

The diagram shows two connected flows. The preparation flow, run once per document: your documents are split into chunks (one idea per passage), the chunks are embedded into lists of numbers capturing meaning, and the embeddings are stored in a vector store, a database built to answer "what is nearest to this?". The query flow, run on every question: the question is embedded the same way, a similarity search scores it against every chunk with the vector store returning the nearest chunks, the top passages are re-ranked by a careful second pass, and the passages are assembled with the question into a prompt from which the model writes an answer grounded in, and citing, the passages. Retrieval decides what the model sees; the model never searches anything itself.

How does a document become searchable?

The searchable library is built ahead of time, before anyone asks anything. Three steps, and each involves a design decision rather than a purchase.

Chunking. Documents are split into passages called chunks, ideally one idea per chunk. Size is a trade-off you tune rather than a setting you look up. Chunks that are too large dilute the search, because a passage about ten things matches nothing precisely. Chunks that are too small lose the context that makes a passage answerable, like a clause separated from the policy it belongs to.

Embeddings. Each chunk is converted into an embedding: a long list of numbers that captures what the passage is about, produced by a model trained so that texts with similar meaning land near each other numerically. "Refund window for damaged goods" and "returning a faulty item" sit close together as numbers even though they share almost no words. That property is what lets retrieval work by meaning rather than by keyword.

The vector store. Embeddings live in a vector store, a database built to answer one question quickly: which stored items are nearest to this one? (Industry jargon: approximate nearest neighbour search.) This preparation runs once per document and again whenever documents change. Give that refresh a named owner and a trigger when the pipeline is first built, because until it re-runs the system keeps answering from the old version of a policy.

A vector store is a job, not necessarily a purchase. Vector search has arrived inside general-purpose databases, so the system your organisation already runs may do this without a new component, and that is worth checking before adding one. A separate store means a second copy of your content to secure, to back up, and to keep in step with the original, plus a second place where somebody can be granted access they should not have. Add a dedicated store when scale or search features justify it, rather than because the pattern appears to demand one.

How does a question find the right chunks?

When a question arrives, it travels the same road: the system embeds the question with the same model used for the documents, then asks the vector store for the chunks whose embeddings sit closest to it. Closeness is usually measured with cosine similarity, which asks how strongly two of these numeric summaries point in the same direction, scored between zero and one. The top few chunks, often three to five, move forward. Everything else is ignored, which is the point: the model will only see what retrieval selects, so retrieval quality sets the ceiling on answer quality. Production systems usually run hybrid search, pairing this meaning-based retrieval with old-fashioned keyword matching, because product codes, names, and exact phrases defeat embeddings alone.

Rabbit hole How the search stays fast, and what that costs you

Scoring a question against every stored chunk is fine at the scale of the demo below, which compares against fifteen. It stops being fine somewhere in the millions, where checking everything on every question would be too slow and too expensive to run.

Production vector stores avoid it by building an index: a structure that lets the store jump straight to a small neighbourhood of likely candidates instead of walking the whole collection. The design that won, if you hear somebody name it, is HNSW: the index is built in layers, so a search crosses the whole collection in a few long hops and then refines locally, in the way you would use motorways before A-roads before the residential street. The saving is enormous. The compromise is that the index is approximate, so what comes back is the nearest chunks it found rather than the nearest chunks that exist. Usually those are the same. Occasionally they are not.

Two consequences follow, and neither asks you to choose an index yourself:

• Retrieval can miss a passage that was genuinely the best match, which is one more reason retrieval quality sets the ceiling on answer quality.
• The balance between speed and thoroughness is tunable, so a change made for performance can quietly reduce answer quality. That is the kind of regression a golden question set catches and nothing else does.

What is re-ranking, and why add it?

The first search is built for speed. It compares compact numeric summaries so that it can sweep millions of chunks in milliseconds, and it pays for that speed with bluntness. Re-ranking adds a second, slower pass over the shortlist only: a more careful model reads the question and each candidate passage together and re-scores them, promoting passages that answer the question over passages that merely resemble it.

The trade-off is cost and latency for precision. Over a small collection, or where the first pass already performs well, re-ranking can be complexity you do not need. Over large or repetitive collections, where many chunks look alike, it often decides whether the right passage reaches the model at all.

Try it: a retrieval pipeline in your browser

The demo below runs the whole pipeline in your browser: no server, no AI model, nothing leaves the page. The knowledge base is the student services handbook of Harwick University, which does not exist: fifteen chunks drawn from six documents, embedded ahead of time exactly as a production system would do it.

One honest simplification. Real systems use neural embeddings, learned from data. To run instantly in a browser, this demo uses word-frequency vectors, the TF-IDF method that powered an earlier generation of enterprise search, plus a small hand-written table of related terms standing in for learned meaning, so "afford" also searches for "hardship". The steps, their order, and the failure modes are the real thing; the embedding arithmetic is simplified. The fourth question below is one the handbook cannot answer: run it and watch the scores collapse, which is exactly the signal a production system uses to say "I don't know" instead of guessing.

Written walkthrough of a full run (text equivalent of the demo)

Question typed: "Can I have more time for my assignment because I was ill?"

1. The question becomes a vector. Everyday words are dropped, and the remaining terms are weighted by how unusual they are in the handbook: assignment, ill, time. The related-terms table expands the query: assignment also searches assessment, extension, and submission; ill also searches extenuating, circumstances, and evidence.
2. Every chunk is scored. Cosine similarity against all fifteen chunk vectors. Top scores: Extenuating circumstances policy part 2 (0.195), part 3 (0.184), part 1 (0.140), then Library services part 1 (0.117), a lookalike that mentions "term time" without being about deadlines.
3. The top four are retrieved and re-ranked. No exact phrases from the question appear in the chunks, so this query's order stands. A query using the policy's own words, such as "accepted grounds for extenuating circumstances", is promoted sharply by the phrase pass; that is the case where re-ranking changes the result.
4. The prompt is assembled. The retrieved passages are placed above the question with the instruction: "Answer using only the passages below. If they do not contain the answer, say so." That assembled prompt, not the question alone, is what a production system would send to the model. Notice the order: the passages arrive before the model writes a word, so it is writing the answer out of them rather than checking itself against them.
5. The model writes the answer. "Yes. If illness affected your ability to complete an assessment you can apply for extenuating circumstances, and you should apply before the deadline where possible. An approved claim usually results in an extension of up to ten working days, or a deferral to the next assessment period for examinations. You will need independent evidence, such as a medical note dated within the affected period, although self-certification is accepted once per academic year for absences of up to five days." Every claim traces back to a retrieved passage, which is what makes the citation possible.

That answer is illustrative rather than generated: this page contains no model, which is why nothing in the demo leaves your browser. The model supplies the language; your documents supply the facts.

How much does this reduce hallucination?

A great deal, and never entirely. Grounding helps for three reasons:

• The model answers from text in front of it rather than from memory.
• The system can cite which passages it used, making answers checkable.
• Low retrieval scores are a signal that the honest answer is "that is not in the knowledge base", which a well-built system says instead of improvising.

The limits deserve equal billing. Retrieval quality is the ceiling: if the right passage is not retrieved, the model cannot use it, and may improvise from whatever arrived instead. Wrong or outdated documents produce confidently cited wrong answers, because citation proves provenance, not truth. And a model can still drift beyond its sources when passages half-answer a question. Teams that succeed with RAG measure retrieval quality on real questions from real users, and treat the knowledge base as a product with an owner, not a folder.

When should you not use RAG?

Every pattern in this series comes with the conditions where it is the wrong choice. For RAG:

• The corpus is small and stable. If everything fits comfortably into the model's context window, the working memory a model can read at once, then paste it in directly. A retrieval pipeline in front of forty pages of policy is machinery without a purpose.
• The problem is behaviour, not knowledge. Retrieving your policies will not make a model write like your organisation. That is a job for what you put in front of it: explicit style rules in the standing instructions, and a handful of examples of good output, which teach tone and structure far better than any number of adjectives. Retrieval can serve this too, by fetching the three closest previous letters of a type rather than the policy that governs them. Fine-tuning is the last resort here rather than the first, and usually for cost rather than quality: a long style instruction paid on every request across millions of requests is real money, and training the habit in removes it.
• The questions are calculations. "How many students deferred this year?" is a database query. Retrieval finds passages of prose; it does not count, sum, or reconcile. Those questions belong against structured data, and nobody should expect the person asking to know that. Deciding which way a question should go, then turning it into a query, is itself a job for a model, and it is the routing work Module 4 describes: one box to ask in, several places the answer might come from, and something in the middle choosing between them.
• Access control comes second. Retrieval reaches into your documents, so the pipeline must respect who is allowed to see what, per user and per query. An assistant that surfaces HR records or unpublished financials to anyone who phrases the right question is a data breach with a chat interface. If permission-aware retrieval cannot be built yet, the corpus must contain only what everyone may see. Module 5 takes up governance in full.

Text description of this diagram

A decision flowchart. Start: AI should answer from your organisation's knowledge. Does it all fit in the context window? If yes, paste it into the prompt; no pipeline is needed. If no: are the questions counts and totals? If yes, query structured data and report from it. If no: is the need behaviour rather than facts? If yes, that is fine-tuning territory, covered in Module 1. Otherwise, build retrieval-augmented generation, with permission-aware retrieval and retrieval quality measurement from day one.

Next: Module 3, Data foundations: why AI projects stand or fall on data decisions made years before anyone said "AI".