In the AI Primer, we bumped into two stubborn limitations of language models. They only know what was in their training data (so nothing about your company, your documents, or anything recent), and they can confidently make things up when asked about what they don’t know. For a huge share of what people actually want to do with these models — “answer questions about our product manual,” “summarise this contract,” “look through our support tickets” — those two limitations are the whole ball game.
There’s a technique that fixes both problems, and it’s arguably the most important pattern in practical AI today. It goes by the initials RAG — Retrieval-Augmented Generation. This track is entirely about it, and this article is the on-ramp: what RAG is, why it works, and why it has quietly become the default way to build serious AI applications.
The core idea, in one sentence
Let me start with the whole idea, then unpack it. RAG means: before the model answers your question, we fetch the relevant information from your own documents and put it directly into the prompt — so the model answers from that, instead of from memory.
Nothing about the model changes. What changes is what’s in front of it at the moment it answers. Instead of relying on its fuzzy general memory, it’s now reading from real, specific source material you supplied. It’s the difference between asking a knowledgeable friend a question from memory and asking them to answer after reading a page from your manual. Same friend, dramatically better answer — and dramatically more trustworthy.
Open book, not closed book
Here’s the analogy I find most helpful. Think of an ordinary language model as a student sitting a closed-book exam. It has to answer purely from what it remembers — no notes, no textbook. It’s read enormously, so it does surprisingly well, but its memory is fuzzy on details, out of date, and occasionally invents plausible-sounding fictions. Sound familiar? It should — this is the model we met in the Primer.
RAG turns that exam into an open-book exam. Before answering each question, the model gets to look up the relevant pages of the actual textbook — the pages you provided. Its reasoning and language skills are the same, but now it’s reading from real material rather than groping in memory. Its answers become specific, accurate, and, crucially, traceable — it can even tell you exactly which page it got them from. Figure 1 makes this contrast concrete.
Figure 1: A plain language model answers from memory alone — vague, potentially out of date, sometimes invented. RAG lets it read from the real source material you provide first, so its answer is grounded, specific, and citable.
Why not just fine-tune it on your data?
This is the natural first question, and it’s worth answering right away because it reveals why RAG has won for this kind of problem. Fine-tuning (the topic of another track) means retraining the model on your data so its weights actually change. It sounds like the “proper” answer — surely you want the knowledge inside the model?
But fine-tuning turns out to be a poor way to teach a model facts. It’s slow to update, so when your policy changes on Tuesday, your fine-tuned model doesn’t know. It’s expensive, so you can’t afford to redo it every time a document changes. And a fine-tuned model still can’t cite where an answer came from — the fact is baked into its weights, not linked to a source.
RAG is the mirror opposite on every dimension. You update it by simply changing the documents. It’s cheap. And every answer comes with a pointer to its source. So for the huge, common case of “keep the model current on our facts,” RAG wins by a mile. Fine-tuning has its place (it’s brilliant for changing behaviour and style, and we’ll cover exactly when to use it in the Fine-Tuning track), but for knowledge, RAG is almost always the right answer.
What RAG actually looks like
At its most basic, a RAG system does two things every time you ask a question. Figure 2 shows the flow.
Figure 2: When a question arrives, the system first retrieves the most relevant passages from your document collection, then hands both the question and those passages to the model, which produces a grounded answer.
Step 1 is retrieve: search your document collection and pull out the passages most likely to contain the answer. This is the “R” in RAG. How this search actually works — matching by meaning rather than by exact words — is a lovely idea that gets its own article next.
Step 2 is generate: hand both the original question and the retrieved passages to the language model, with an instruction like “answer the question using only the information below.” The model reads, thinks, and produces its answer — grounded in the real material rather than in memory. This is the “AG” in RAG: augmented generation.
That’s the whole pattern. Everything else in this track — embeddings, chunking, vector databases, re-ranking, and so on — is really about making those two steps work well in practice.
Why this pattern quietly changed everything
Once you internalise RAG, you’ll see it everywhere. The customer support bot that answers from your help documentation? Almost certainly RAG. The tool that lets you “chat with a PDF”? RAG. The internal assistant that answers company policy questions? RAG. It has become the default way to build any AI application that needs to be accurate on specific information, because it directly addresses the two biggest weaknesses of raw language models.
The strategic beauty of it is worth naming. RAG gives you the fluency and reasoning of a powerful language model and the accuracy and up-to-date-ness of your own documents — without having to retrain anything. Update a document, and your assistant knows the new answer. Add a document, and it now covers a new topic. That combination of capability and control is why almost every serious “AI over our own data” project you’ll encounter is a RAG project underneath.
Where we go from here
This track is going to build RAG up piece by piece so you understand every part of it — not just what to do, but why. Next, we open up the surprisingly elegant idea that makes retrieval actually work: how machines can search by meaning rather than by exact words. It’s called semantic search, and once you see it, a lot more than just RAG will click into place.
Next in the series: Embeddings & Semantic Search Explained — how a machine finds relevant text by meaning, not just keywords.