Here’s a habit that quietly ruins otherwise-serious RAG projects. Someone tweaks a chunk size, runs one question, glances at the answer, thinks “yes, that’s better,” and ships the change. The next tweak: same routine. Rinse and repeat. Six months in, the team has changed dozens of things, each one supposedly an improvement, and nobody can tell you with a straight face whether the current system is actually better than the one they had at the start.
This is the trap of judging RAG by feel. The escape is boringly simple and enormously powerful: measure it. Not in a heavyweight, formal way, but in a lightweight, honest way that turns “I think this is better” into “here’s the number that says so.” This article is about how to do that.
Why evaluating RAG is a bit special
You already met the general habit of testing prompts in the Prompt Engineering track. That’s the foundation. But evaluating a RAG system has an extra wrinkle worth naming: there are two things you have to judge, not one.
There’s the retrieval: did the system fetch the right passages for this question? There’s the generation: did the model produce a good answer, given whatever passages it got?
Those two questions are separate, and it really matters to look at them separately. Because if you only measure the final answer, you can’t tell whether a bad answer came from retrieval failing (wrong passages) or generation failing (right passages, but the model messed it up). Which, you’ll remember from the “why your RAG keeps hallucinating” article, is precisely the distinction you need to know to fix anything. Figure 1 shows the two-part framing.
Figure 1: A RAG evaluation has two halves. Retrieval quality asks whether the right passages were fetched. Answer quality asks whether the final response was correct, faithful to the passages, and complete. Measuring both separately tells you exactly where to focus improvements.
What “good” looks like on each half
Let’s put words to what we’re actually measuring on each side.
For retrieval quality, a few practical measures cover most of the ground:
- Was the truly relevant chunk fetched at all? The simplest and most useful check. Across your test set, on what fraction of questions does the right chunk appear in your top-k results? If the right chunk isn’t retrieved, no downstream fix can rescue the answer.
- Where in the ranking did it appear? If the right chunk is fetched but ranked eighth, that’s a very different problem than not being fetched at all. Watching where the winning chunk lands tells you whether you need better retrieval or better re-ranking.
- How much of the top-k is actually useful? If nine of your ten fetched chunks are irrelevant, your prompt is being polluted with noise, and that hurts the model’s focus and your bill.
For answer quality, four qualities matter, and it helps to score each on its own:
- Correctness. Is the answer actually right?
- Faithfulness (or “groundedness”). Does every claim in the answer come from the retrieved passages? This is the specific measure of hallucination, and it’s the one to watch most carefully.
- Completeness. Did the answer cover the full question, not just part of it?
- Relevance. Did it answer the question that was asked, rather than a slightly different one?
You may not measure all four every time; often faithfulness plus correctness is enough. But knowing the full set means you’ll never miss a whole dimension of quality.
The one habit you need: a real test set
Everything in RAG evaluation rests on one deceptively simple thing: a test set of representative questions, ideally with the ground truth for each. It doesn’t need to be big. Twenty to fifty carefully-chosen questions is plenty for most projects, and even ten is a lot better than none.
A good test set covers the range of what real users will ask:
- The common, straightforward questions your system should nail.
- The tricky, ambiguous, or compound questions that stretch it.
- Questions from different topic areas your documents cover.
- Questions you know are not in the documents (so you can check that the system honestly says so instead of inventing an answer).
- Any specific question you’ve seen fail badly in the past. Once a failure is on the list, it stays on the list, forever, so you can’t quietly regress.
For each question, jot down what a good answer looks like, and if you can, which chunk or source contains it. You don’t need to be exhaustive; you just need enough of a reference to grade against. Figure 2 shows this evaluation cycle.
Figure 2: Every change to your RAG system runs against the same test set. You measure retrieval quality and answer quality, compare the scores to the previous version, and only ship if the numbers say you improved. This turns tuning from guesswork into a real feedback loop.
The whole point of the test set is comparability. When you change chunk size, or add re-ranking, or swap embedding models, you run the same test set through the new configuration and the previous one, and you compare the scores. Improvements you can see replace improvements you feel. Two configurations differing by 3 out of 25 correct answers is a real, defensible improvement. A vague sense that “it seems sharper today” is not.
How to actually grade the outputs
A fair question: how do you decide if an answer is “correct” or “faithful”? Three practical approaches, from lightest to heaviest.
Judge by eye. For a modest test set, just read the results against your notes. Simple, honest, and quick, and you learn a lot as you read. Don’t underrate this. Twenty minutes with a printed test set often reveals more than any dashboard.
Automate the easy checks. Some checks are objectively measurable and easy to script: “did the right chunk appear in top-5?” is just a lookup. “Did the answer contain the specific phrase we expected?” is a substring check. These automated checks let you rerun the test set effortlessly on every change.
Use a model as a judge. For larger test sets or subtler qualities like faithfulness, you can have a capable language model score each answer against a rubric (“rate 1 to 5 for whether every claim is supported by the passages”). This scales well and is used a lot in practice. But: spot-check the model’s judgements against yours occasionally, so you can trust the scores. A model judging models isn’t magic, but it’s a legitimate way to widen your bandwidth.
You don’t have to pick one. Most mature setups use all three: quick automatic checks on every change, model-judged scores on a larger sweep, and human review on a sampled subset.
The lightweight version you can start today
If all of this sounds like a lot, here’s the smallest useful version you can start with, and honestly it’s what I’d recommend for most teams as their first step:
- Write down 15 to 20 questions covering the shape of what users actually ask.
- For each, note the right source (which document/page/section) and what a good answer looks like.
- Every time you change something in your RAG system, run all the questions. Log which retrieved the right source, and how the answer scored on correctness and faithfulness.
- Only ship changes where the numbers improved.
That’s it. No fancy tooling required for step one. You can build up from there, but even this modest habit will put you well ahead of anyone tuning by feel.
The mindset shift
The bigger reframe underneath all of this is that evaluation isn’t a chore you do at the end of the project. It’s the feedback loop that drives your improvements. Without it, you’re steering blind; with it, every change becomes a small experiment you can learn from. The teams that ship reliable RAG aren’t the ones with the most sophisticated pipelines. They’re the ones who know, at any moment, exactly how their system is performing, and whether that number is going up or down.
We have one article left in this track. Once you’re building well and evaluating well, the last frontier is running your RAG system in the real world, where scale, cost, monitoring, and change all come into play. That’s the topic of the final article.
Next in the series: RAG in Production: Cost, Latency & Monitoring, running a RAG system reliably at real-world scale.