If you followed the RAG track earlier on this site, this article is where two threads braid together. You learned there how retrieval-augmented generation gives a model access to knowledge it wasn’t trained on: you fetch relevant documents, stuff them into the prompt, and let the model answer grounded in real information. It’s one of the most useful patterns in all of applied AI. And here in the agentic track, you’ve learned that agents decide what to do and use tools to do it. Agentic RAG is what happens when those two ideas meet, and the result is meaningfully better than either alone.
The short version: in classic RAG, retrieval is something that happens to the model on a fixed schedule. In agentic RAG, retrieval becomes a tool the agent chooses to use, on its own terms, as many times as the problem needs. That one shift, from a fixed pipeline to a decision the agent makes, fixes most of the frustrations people hit with basic RAG.
The problem with classic RAG
Let me remind you how classic RAG runs, because its limitation is the whole motivation here. A user asks a question. The system takes that question, searches a knowledge base for relevant chunks, pastes the top few into the prompt, and asks the model to answer. Retrieve, then generate. Always once, always in that order.
That works beautifully for simple, direct questions. It falls apart in three common situations.
It falls apart when the question is poorly phrased for search. A user asks “why did it break?” and the raw query retrieves nothing useful, because the good documents talk about “system failures” and “root causes,” not “it” and “break.” Classic RAG has no way to rephrase and try again.
It falls apart on multi-part questions. “Compare our refund policy with our warranty policy” needs two separate retrievals, but classic RAG does one search and hopes the top chunks happen to cover both.
And it falls apart when the first retrieval isn’t good enough. The chunks come back thin or off-target, and classic RAG has no mechanism to notice and go looking again. It just answers with what it got, thin chunks and all.
Figure 1: Classic RAG runs retrieve-then-generate exactly once in a fixed order. Agentic RAG puts retrieval inside the agent loop, so the agent can search, judge, rephrase, and search again as needed.
Figure 1 puts the two side by side. The left path is a straight line. The right path is a loop, and that loop is the entire difference.
Retrieval as a tool
Here’s the mental flip. Instead of hard-wiring “always search once before answering,” you give the agent a search_knowledge_base tool and let it decide. Everything you learned about tool use in article 4 now applies to retrieval.
This unlocks behaviours classic RAG simply cannot do. The agent can rewrite the query before searching, turning a vague user question into a sharp search query. It can search multiple times, once for each part of a compound question, and combine what it finds. It can read the results and judge them, and if they’re thin or irrelevant, search again with different terms rather than answering from bad material. It can even decide not to search at all when the question is one it can answer directly, saving a pointless round-trip.
That last one matters more than it looks. A classic RAG system retrieves even when someone just says “thanks, that helps,” wasting a search and sometimes polluting the answer with irrelevant chunks. An agent that owns the decision simply skips retrieval when it isn’t needed.
What a run looks like
Let’s trace a real example so the loop feels concrete. A user asks: “Does our enterprise plan include the audit-log feature, and how does that compare to what the pro plan offers?”
The agent reads the question and recognises two parts. It rewrites the first part into a clean search query, something like “enterprise plan audit log feature,” and searches. It gets back a clear answer: yes, enterprise includes audit logs. It notes that, then rewrites the second part, “pro plan audit log logging features,” and searches again. This time the top chunk is vague, so the agent judges it insufficient and searches once more with “pro plan feature list comparison.” Now it has solid material for both halves. It stops searching, because more wouldn’t help, and writes an answer that genuinely compares the two plans.
Figure 2: One walk through agentic RAG: rewrite the query, search, judge the results, decide whether to search again, and only then answer. The judge-and-decide step is what classic RAG lacks.
Figure 2 traces exactly that. Three searches, each a deliberate choice, driven by the agent reading its own results and deciding what to do next. Classic RAG would have done one search and given a half-answer.
The cost of the upgrade, stated plainly
Agentic RAG is not a free win, and I’d be doing you a disservice to pretend otherwise. It’s slower and more expensive than classic RAG, because it may run several searches and several model calls where the simple version runs one of each. For a high-volume, latency-sensitive product answering mostly simple questions, plain RAG is often the right, cheaper choice.
There’s also a familiar risk: an agent that keeps searching, never satisfied, burning through calls. The stopping conditions from article 3 are non-negotiable here. Cap the number of retrievals, and make the agent’s instructions clear that it should answer once it has enough, not once it has everything imaginable.
The honest framing is the same one from the multi-agent article. Start simple. Use classic RAG when questions are direct and single-part. Upgrade to agentic RAG when your questions are genuinely messy, multi-part, or when first-pass retrieval quality has been letting you down. Match the tool to the problem, not to the fashion.
In the next article we look at something that’s been quietly reshaping how agents connect to tools and data sources: the Model Context Protocol, a standard that’s making the tool-use machinery from article 4 far less painful to build and reuse.