We’ve spent the last few articles making the retrieval step itself smarter: cleaner chunking, tuned top-k, hybrid search, re-ranking. All of that helps enormously, but it shares one quiet assumption: that a single retrieval, done well, is enough to answer the user’s question. And for many questions, that’s true.
But some questions break the pattern. They’re vague, or they hide two questions in one, or the real answer only becomes findable after you’ve retrieved some initial information. For those, no amount of tuning your one-shot search will save you. You need to change the shape of retrieval itself. That’s the world of advanced RAG, and it comes in two big ideas: making the question better before searching, and letting retrieval happen in more than one step.
Why one search sometimes isn’t enough
Two everyday examples make this vivid.
First, imagine a follow-up question in a chat: “And what about for enterprise customers?” On its own, that string is nearly meaningless. Search on it and you’ll get irrelevant results. To retrieve well, the system needs to rewrite the question using the earlier context (“What is the standard refund policy for enterprise customers?”). A better search question produces a better search.
Second, imagine a compound question: “How did our Q3 revenue compare to Q2, and what did the report say caused the difference?” That’s really two questions, each needing different chunks. A single embedding search for the whole thing tries to be about both at once, and ends up finding not much about either. To retrieve well, you’d want to break the question apart and search for each piece separately.
Both examples share a lesson: the way the user asks isn’t always the way you should search. Advanced RAG is largely about closing that gap.
Idea 1: Rewrite the question before searching
The first big move is query rewriting: take the user’s raw question, transform it into one or more better search queries, and then run retrieval on those. Figure 1 shows the shape.
Figure 1: A cleanup step reshapes the user’s raw question into something better suited to retrieval. A vague or context-dependent query becomes clear; a compound query becomes several focused ones; the resulting searches are far more likely to find the right chunks.
There are three flavours of rewriting worth knowing, and they cover most real cases:
Context expansion for follow-ups. In a chat, rewrite the follow-up question into a stand-alone one using the earlier turns. “And for enterprise?” becomes “What is the standard refund policy for enterprise customers?” This dramatically fixes retrieval in ongoing conversations, and it’s often the single most valuable rewrite you can add.
Query decomposition for compound questions. Break a multi-part question into separate sub-questions, retrieve for each, then combine what you found. The Q3-vs-Q2 example above becomes two searches: “Q3 revenue” and “Q2 revenue”, or “causes cited for revenue change between Q2 and Q3” and so on. Each search is now clean and targeted.
HyDE (Hypothetical Document Embeddings). A cleverer trick with a scary name and a simple idea. Instead of searching with the question, first ask the language model to draft a possible answer (even a made-up one), then use the embedding of that answer to search. Why? Because a hypothetical answer often uses the vocabulary of the real answer, so its embedding ends up closer to the right chunks than the question’s embedding does. It’s a subtle idea that quietly outperforms plain question search for many domains.
You don’t have to use all three. Pick the ones that match the problems you’re seeing. Most systems start with context expansion for chat and query decomposition for compound questions, because those two cover the majority of frustrating misses.
Idea 2: Retrieve in more than one step
The second big move is multi-step retrieval. Some questions can’t really be answered from a single search because the first result is what tells you what to look for next. Two examples:
- “Which of our competitors offers the longest free trial, and how long is it?” First you need to find the list of competitors’ trials, and only then can you compare and answer.
- “What did the CEO say about the situation described in the risk section of the annual report?” First you find the risk section; then, informed by what it says, you search for the CEO’s remarks about that specific situation.
These are called multi-hop questions, because you need to hop from one piece of information to another. The right response is to break the retrieval itself into multiple steps: retrieve, read, formulate a follow-up query based on what you found, retrieve again, and combine. Figure 2 shows this.
Figure 2: A one-shot retrieval tries to find the answer in a single search and often fails on multi-hop questions. Multi-step retrieval fetches an initial piece, reads it, formulates a follow-up query based on what it found, and searches again, chaining hops until the full answer is assembled.
In pseudocode, a two-hop retrieval looks like this:
# Hop 1: find the initial piece
chunks1 = retrieve(question)
partial = summarise_or_extract(chunks1)
# Hop 2: use what we learned to formulate a targeted follow-up query
followup_query = build_followup(question, partial)
chunks2 = retrieve(followup_query)
# Combine everything for the model
answer = generate(question, chunks1 + chunks2)It’s more moving parts than a one-shot retrieval, but the payoff on questions that actually need it is huge. And it opens the door to something you may have started to sense already: if the system can decide for itself how many hops to take, what to search for next, and when it has enough to answer, we’re starting to describe something with real autonomy. Which is exactly the bridge to the next article.
The trade-off you’re now used to
There’s an honest cost. Query rewriting means extra model calls before the search. Multi-step retrieval means multiple rounds of retrieval and generation for a single question. Both slow things down a little and cost a little more per query. Neither is free.
The mature move is the one that’s run through this whole track: use the simplest approach that solves your problem. For most everyday questions, the single-shot pipeline you built earlier is completely sufficient. Reach for query rewriting when you notice retrieval struggling with vague follow-ups or compound queries. Reach for multi-step retrieval when you have genuinely multi-hop questions that a single search can’t touch. Don’t apply these techniques universally just because they’re more sophisticated. That’s how you end up paying more for the same answer, or worse, adding failure points to an otherwise-reliable system.
Where this leads
Look at what we just described in the second half of this article. A pipeline that reads its first retrieval, decides what it still needs, formulates a new query, and searches again. That’s no longer really a “pipeline” in the fixed-flow sense. It’s starting to reason about its own retrieval, on the fly, guided by what it finds along the way.
Give a language model the ability to decide whether to retrieve, what to search for, and when it has enough, and you’ve made the leap from retrieval-as-a-fixed-step to retrieval-as-an-active-decision. That leap has a name: agentic RAG. And it’s next.
Next in the series: Agentic RAG: Retrieval With a Brain, letting the model decide whether, what, and how to retrieve.