The end of the last article left a breadcrumb. If a system reads its first retrieval, decides what it still needs, formulates a new query, and searches again, then retrieval isn’t really a fixed step anymore. It’s starting to reason about itself. And once you let the language model make those decisions, you cross into what people call agentic RAG: retrieval with a brain in charge of it.
This article is about that shift, why it matters, and where it earns its keep. It’s also the natural bridge between the RAG track and the Agentic AI track, because agentic RAG is really the first place where the ideas of those two worlds meet.
From a pipeline to a decision-maker
Everything we’ve built up so far is essentially a pipeline: a fixed sequence of steps the system runs on every question, regardless of what the question is like. Ingest, embed, retrieve top-k, prompt, generate. Same shape, every time. That works remarkably well for most cases and is exactly what you should start with.
But a pipeline has an inherent limit: it can’t tell one question from another. It runs the same retrieval whether the question is a trivial lookup, a compound query, a multi-hop follow-up, or a general knowledge question that doesn’t need retrieval at all. It treats them all the same.
Agentic RAG changes that by handing the reins to the language model. The model itself decides:
- Do I actually need to retrieve anything? (Sometimes the answer is no.)
- What should I search for? (Rephrase, decompose, or reformulate the query as needed.)
- Are these results any good, or should I search again?
- Is one round of retrieval enough, or do I need to hop?
- Do I have enough now to answer?
Figure 1 shows the shift in the shape of the whole system.
Figure 1: A pipeline runs a fixed sequence of steps every time. Agentic RAG turns retrieval into a loop the model controls: it decides whether to search, what to search for, evaluates results, searches again if needed, and stops when it has enough to answer.
Nothing about your embeddings, chunks, vector database, or re-ranker has to change. What changes is who is in charge of when and how they get called.
Retrieval as a tool the model can use
The cleanest way to think about agentic RAG is that retrieval becomes a tool the language model can choose to call, rather than an unavoidable step it’s forced through. This will sound familiar if you’ve peeked at the Agentic AI track: giving a model tools it can decide to use is exactly the definition of an agent. Agentic RAG is what you get when one of those tools is your retrieval system.
In practice, you describe your retrieval to the model as a tool it can invoke, roughly like this in pseudocode:
tools = [{
"name": "retrieve",
"description": "Search the internal knowledge base and return the top passages.",
"input": {"query": "string"}
}]
# The model reads the user question and decides whether/how to call the tool.
# It can call it once, or several times with different queries, or not at all.
answer = agent.run(user_question, tools=tools)The model reads the question, and if it decides it needs to look something up, it emits a call to the retrieve tool with a search query it chose. It gets the passages back, reads them, and either answers, or calls the tool again with a better query. This flexibility is what makes it powerful.
The behaviours you get for free
Once retrieval is a tool the model can call at will, several of the tricks from the previous article stop being separate techniques and just… happen naturally. Figure 2 shows a few of the behaviours that emerge.
Figure 2: With retrieval as a decision, the system naturally decides whether to search at all, chooses a good query, judges whether results are sufficient, searches again with a better query if not, and hops across sources for multi-part questions. Behaviours the pipeline had to be hard-coded to do, an agent can just do.
Some of the useful patterns that fall out of this:
- Skip retrieval when it isn’t needed. Ask “What’s the capital of France?” and the model can just answer, without wasting a retrieval on it.
- Rewrite the query itself. No need for a separate rewriting stage; the model can rephrase before searching.
- Iterate when results are weak. If the first search doesn’t return anything useful, the model can try a different query, or a narrower one, or a broader one.
- Multi-hop naturally. Multi-step retrieval is just the model calling the tool a second time based on what it read.
- Choose among tools. Give it more than one retrieval tool (say, one for internal docs and one for the web), and it picks the right one for the question.
None of this is magic. It’s the direct consequence of moving from “always retrieve like this” to “decide, based on what you know so far, what to do next.”
The honest trade-offs
Agentic RAG is genuinely powerful, but the trade-offs are real and they’re the same ones that show up whenever you give a model more autonomy. Worth being frank about:
- It’s more expensive and slower. Every decision the model makes about retrieval is another model call. A question that would have taken one retrieval and one generation might now take several of each.
- It’s harder to predict. Because the model decides its own path, two runs of the same system on the same question won’t necessarily do the same thing. This is a feature (flexibility) and a headache (harder testing and debugging).
- It has more failure modes. The model can pick a poor query, misjudge whether results are sufficient, or spiral in unhelpful loops. You’ll want the debugging habit from earlier in the track, plus the evaluation habits we’ll come to shortly.
- It needs guardrails. Give a model a tool and it can misuse it. In this case that mostly means calling retrieval more than necessary or with unhelpful queries. Setting sensible limits (a maximum number of retrieval calls per question, for example) keeps costs and behaviour in check.
Where agentic RAG earns its keep, and where it doesn’t
A useful rule of thumb: agentic RAG is worth reaching for when the variety of questions in your application is high, when many of them are multi-hop or context-dependent, or when your users mix genuine lookups with general chatter that doesn’t need retrieval. In those situations, the flexibility pays for itself.
For narrower use cases (a support bot answering questions from one policy document, say), a good pipeline is often cleaner, cheaper, and more predictable. Don’t upgrade for the sake of upgrading. The point of every technique in this track has been to solve a real problem you can name, and that stays true here. Agentic RAG solves a real problem when the shape of your questions genuinely varies. It’s overkill when they don’t.
Bridging to the next track
Notice what we ended up with: a language model that decides when to use a tool, evaluates the tool’s output, iterates until it has what it needs, and stops when the job is done. That is the definition of an agent, and the retrieval-as-a-tool pattern we just described is one of the most common and useful shapes an agent can take. Which is exactly why the Agentic AI track picks up from here. If you found this article interesting, that’s the natural next place to explore.
For now, we have two remaining articles to close out this RAG track. Everything so far has focused on building good RAG. The last two are about proving it works and keeping it working. In the next article, we look at how you actually evaluate a RAG system so you know it’s improving instead of hoping it is.
Next in the series: Evaluating RAG Systems, measuring retrieval and answer quality so improvements are real, not imagined.