RAG Evaluation: Measuring Retrieval Quality Before You Trust It
TL;DR
- A RAG demo that answers five questions well tells you almost nothing. You need a measured baseline across a real query set before you trust the system in production.
- Score retrieval and generation separately. Retrieval gets recall@k, precision, MRR, and hit rate. Generation gets faithfulness, answer relevance, and correctness. Conflating them hides the actual fault.
- Build a golden eval set from real user queries, use LLM-as-judge carefully with human spot checks, and wire the whole thing into CI so a bad prompt or chunking change fails the build instead of shipping.
Why 'it looks good in the demo' is not evaluation
Every RAG project hits the same happy moment. You wire up a retriever, drop in a generation model, type a few questions you already know the answers to, and the thing responds beautifully. Someone says 'ship it.' We've watched this exact scene play out on more client projects than we can count, and the demo almost never survives contact with real traffic.
The problem is selection bias. When you test by hand, you ask questions you understand, phrased the way you'd phrase them, about documents you remember being in the corpus. Real users do none of that. They ask sideways questions, use the wrong words, reference things that aren't in the knowledge base at all, and stack three sub-questions into one sentence. A demo measures whether the system can work. It says nothing about how often it does.
Evaluation is the practice of replacing that vibe check with numbers you can defend. Before you let a RAG system answer anything that matters, you want a baseline score on a representative set of queries, a way to tell whether a given change made things better or worse, and an alarm that goes off when quality drops. None of that comes from staring at the demo.
- Hand-testing favours questions you already know are answerable, so it overstates real-world accuracy.
- A demo proves capability once, not reliability across hundreds of varied queries.
- Without a baseline you can't tell whether your next change helped or quietly hurt.
- The cost of a wrong answer in production is paid by users, not by you at your desk.
Separating retrieval quality from generation quality
The single most useful habit in RAG evaluation is refusing to score the system as one black box. A RAG answer is the output of two stages: retrieval pulls candidate chunks from your store, and generation turns those chunks into prose. When an answer is wrong, it's wrong for one of two very different reasons, and the fix is different in each case.
If retrieval failed, the right context never reached the model. No prompt tweak will save you, because the facts aren't in the window. If retrieval succeeded but generation still produced a bad answer, the model ignored the context, misread it, or padded it with invented detail. Those are prompt, model, and grounding problems, not retrieval problems. Score them together and you get a blended number that tells you something broke without telling you where.
Here's the practical version. First, evaluate retrieval in isolation: given a query, did the correct chunks come back in the top k? Then, evaluate generation conditioned on good context: given the right chunks, did the model produce a faithful, relevant, correct answer? Only after both pass independently do you measure end to end. This is the difference between a debugging session that takes an afternoon and one that takes a week.
- Retrieval failure means the answer was impossible regardless of the model.
- Generation failure means the model had the facts and still got it wrong.
- Test generation with known-good context to isolate it from retrieval noise.
- A single end-to-end score hides which of the two stages is actually broken.
- Fixing the wrong stage is the most common waste of time in RAG work.
Retrieval metrics: recall@k, precision, MRR, and hit rate
Retrieval metrics all answer variations of one question: did the relevant documents show up, and how high? You need a golden set first, meaning a list of queries each labelled with which chunks or documents actually contain the answer. Once you have that, the metrics fall out cleanly.
Recall@k is usually the one that matters most for RAG. It asks: of the chunks that should have been retrieved, how many appeared in the top k results? If recall@5 is 0.6, then four times out of ten the model is working with incomplete context before it writes a single word. Precision measures the inverse pollution: of the chunks you returned, how many were actually relevant? Low precision means you're stuffing the context window with noise, which both costs tokens and gives the model more ways to go off track.
MRR, mean reciprocal rank, rewards getting the right chunk near the top. Hit rate is the blunt instrument: did at least one relevant chunk appear anywhere in the top k? Hit rate is forgiving and useful as a floor, while MRR and recall@k tell you whether ranking and coverage are good enough to feed generation. In practice we lead with recall@k for coverage and watch MRR for ranking, because a relevant chunk buried at position nine often gets ignored even when it's technically retrieved.
- Recall@k: fraction of relevant chunks captured in the top k. The coverage metric.
- Precision: fraction of retrieved chunks that are actually relevant. The noise metric.
- MRR: rewards relevant results that rank near the top, where the model reads most carefully.
- Hit rate: did any relevant chunk make the top k? A coarse pass-or-fail floor.
- Choose k to match how many chunks you actually feed the model, not an arbitrary round number.
Answer metrics: faithfulness, relevance, and correctness
Good retrieval is necessary but not sufficient. The model can still misuse perfect context, so generation needs its own scorecard. Three metrics carry most of the weight here, and they measure genuinely different failures.
Faithfulness, sometimes called groundedness, asks whether every claim in the answer is supported by the retrieved context. This is your hallucination detector. An answer can be fluent, confident, and completely unsupported, and faithfulness is the metric that catches it. Answer relevance asks a separate question: does the response actually address what the user asked? A model can produce a perfectly grounded paragraph that answers a question nobody posed. Correctness compares the answer against a known ground truth, which you only have when your golden set includes reference answers.
These pull in different directions, which is the point. An answer that says 'I don't have enough information to answer that' scores high on faithfulness and low on relevance. An answer that confidently restates an industry truism scores high on apparent relevance and low on faithfulness because nothing in the context supports it. Watching all three at once keeps you from optimising one metric into a corner. The pattern we see most: teams chase relevance, the model gets chattier and more confident, and faithfulness quietly collapses.
- Faithfulness or groundedness: is every claim supported by the retrieved context? Your hallucination check.
- Answer relevance: does the response address the actual question, not a nearby one?
- Correctness: does the answer match a reference answer? Needs labelled ground truth.
- A safe 'I don't know' scores high on faithfulness and low on relevance, by design.
- Track all three together so improving one doesn't silently wreck another.
Building a golden eval set from real queries
Every metric above depends on a golden set: queries paired with the right answers, the right source chunks, or both. This is the unglamorous work that decides whether your evaluation is worth anything, and it's where most teams cut corners. Synthetic questions generated from your own documents are a fine way to start, but they share a fatal flaw with hand-testing. They're phrased the way the documents are phrased, so they make retrieval look easier than it is.
The better source is your real query logs. Pull the questions users actually asked, including the messy ones, the typos, the half-sentences, and the questions about things you don't even cover. Label a few hundred of them. For each query, record which document or chunk holds the answer, and ideally write a reference answer. Yes, this takes human hours. There's no shortcut that produces a trustworthy number, and a trustworthy number is the entire point.
Make the set representative, not just large. Include easy questions, ambiguous ones, multi-part ones, and out-of-scope ones where the correct behaviour is to decline. Tag each query by type so you can read scores by segment. A system can post a strong overall number while failing every out-of-scope question, which is exactly the kind of failure that erodes trust fastest. Aim for a few hundred well-chosen, well-labelled queries over thousands of careless ones.
- Source queries from real logs, not just questions generated from your own docs.
- Label each query with its source chunks and, where possible, a reference answer.
- Cover easy, ambiguous, multi-part, and out-of-scope cases so segments are testable.
- Tag by query type so a strong average can't hide a weak segment.
- A few hundred carefully labelled queries beat thousands of sloppy ones every time.
LLM-as-judge and where it falls down
Scoring faithfulness and relevance by hand across hundreds of queries on every change doesn't scale, so the common move is to use a strong language model as the judge. You hand it the question, the context, and the answer, and ask it to rate groundedness or relevance against a rubric. Done well, this correlates reasonably with human judgement and runs in minutes instead of days. We use it on every RAG engagement, but we use it with our eyes open.
The failure modes are real and worth naming. Judges show position bias, favouring whichever answer comes first in a comparison. They show length bias, rating longer answers as better even when they're padded. They tend to favour outputs from the same model family, which matters if your judge and your generator share a lineage. And they're inconsistent: ask twice and you can get two scores. A judge that drifts is worse than no judge, because it gives you confident numbers that aren't measuring anything stable.
The discipline that makes LLM-as-judge usable: pin the judge model and version so scores stay comparable over time, write a tight rubric with explicit criteria rather than 'rate this 1 to 5', score on narrow dimensions one at a time instead of one vague overall grade, and run a small set of cases the judge has already scored back through it to confirm it still agrees with itself. Treat the judge as an instrument that needs calibration, not an oracle.
- Position bias: judges favour the first answer shown in a pairwise comparison.
- Length bias: longer answers get rated higher even when padded with filler.
- Self-preference: judges tend to favour outputs from their own model family.
- Inconsistency: the same input can earn different scores on repeat runs.
- Mitigate by pinning the model version, using narrow rubrics, and calibrating against human labels.
Human review and regression testing in CI
Automated metrics carry the day-to-day load, but humans stay in the loop, just not on everything. The efficient pattern is to let the automated pipeline score the full set, then route a sample to people: the cases where the judge was uncertain, the ones where automated and reference scores disagree, and a random slice for sanity. A few dozen human-reviewed cases per change keeps your automated scores honest without burning a person on every run.
Once you trust the pipeline, put it in CI. Treat evaluation like a test suite, because that's what it is. Every change to chunking, embeddings, the retriever, the prompt, or the model runs the golden set and reports recall@k, precision, MRR, faithfulness, relevance, and correctness against the committed baseline. Set thresholds. If recall@5 drops below a line you've agreed on, or faithfulness falls more than a couple of points, the build fails and the change doesn't merge.
This is the step that turns evaluation from a one-time exercise into a safety net. RAG systems rot quietly. Someone tweaks a chunk size to fix one query and breaks twelve others. A model version updates underneath you and faithfulness slips. Without regression testing in CI, you find out from users. With it, you find out from a red build. The first time a CI run catches a regression nobody would have spotted by hand, the whole practice pays for itself.
- Route uncertain and disagreeing cases to humans, plus a random slice for sanity.
- Run the full golden set on every change to retrieval, prompts, embeddings, or model.
- Commit a baseline and set thresholds that fail the build when scores drop.
- Catch regressions from a chunking tweak or a silent model update before users do.
- Treat the eval set as a versioned test suite that grows as new failure cases surface.
What to fix when each metric is low: start here
Metrics are only useful if they point at an action. Once you've separated the two stages and have numbers in hand, the diagnosis becomes mechanical. Low recall@k almost always means a retrieval problem upstream of the model: your embeddings don't capture the query intent, your chunks are the wrong size, or you need hybrid search to catch keyword matches that pure vector similarity misses. Low precision with decent recall points the other way, toward a reranker or tighter chunking to cut the noise you're feeding in.
On the generation side, low faithfulness with good retrieval is a prompt and grounding problem. Instruct the model to answer only from the provided context and to say when it can't, and confirm the context is actually fitting in the window rather than being truncated. Low relevance with high faithfulness usually means query understanding is off, so look at query rewriting or expansion before the retrieval step. Low correctness with everything else healthy often traces back to the golden set itself, where stale or wrong reference answers make a good system look broken.
If you take one thing from this: start by scoring retrieval and generation separately on a few hundred real, labelled queries, then wire that scoring into CI with thresholds. That single foundation tells you where the system breaks, proves whether each change helped, and stops regressions from reaching users. Everything else is refinement on top of it. If you'd rather not build that harness from scratch, this is the kind of groundwork our team sets up early on a RAG engagement so the rest of the build rests on numbers instead of hope.
- Low recall@k: fix embeddings, chunk size, or add hybrid search.
- Low precision: add a reranker or tighten chunking to cut noise.
- Low faithfulness with good retrieval: tighten the prompt and check for context truncation.
- Low relevance with high faithfulness: improve query rewriting or expansion before retrieval.
- Low correctness with everything else healthy: audit your reference answers for staleness.
Want this built for your business?
We map the highest-leverage place to start and ship a first live system within two weeks.
Book a strategy callCommon questions
What is the difference between retrieval evaluation and generation evaluation in RAG?
Retrieval evaluation checks whether the right source chunks were pulled from your store for a given query, using metrics like recall@k, precision, and MRR. Generation evaluation checks whether the model turned that context into a good answer, using faithfulness, relevance, and correctness. Scoring them separately tells you which stage is broken; a single end-to-end score hides it.
How many queries do I need in a golden eval set?
A few hundred carefully chosen and well-labelled queries beat thousands of careless ones. What matters more than raw count is representativeness: include easy, ambiguous, multi-part, and out-of-scope questions sourced from real user logs, and tag them by type so you can read scores by segment instead of trusting one average.
Is recall@k or precision more important for RAG?
For most RAG systems recall@k matters most, because if the relevant chunks never reach the model, no amount of prompting can produce a correct answer. Precision matters once recall is solid, since low precision floods the context window with noise that costs tokens and gives the model more ways to go off track. Watch both, but fix coverage first.
Can I trust an LLM to grade my RAG outputs?
Yes, with discipline. LLM-as-judge correlates reasonably with human judgement and scales to hundreds of cases, but it shows position bias, length bias, self-preference toward its own model family, and run-to-run inconsistency. Pin the judge model and version, use narrow rubrics scored one dimension at a time, and calibrate against a set of human-labelled cases.
How do I stop a RAG system from quietly getting worse over time?
Put your evaluation in CI as a regression test. Run the golden set on every change to chunking, embeddings, the retriever, the prompt, or the model, compare against a committed baseline, and fail the build when key metrics like recall@k or faithfulness drop below agreed thresholds. That way you catch regressions from a tweak or a silent model update before users do.
Retrieval evaluation checks whether the right source chunks were pulled from your store for a given query, using metrics like recall@k, precision, and MRR. Generation evaluation checks whether the model turned that context into a good answer, using faithfulness, relevance, and correctness. Scoring them separately tells you which stage is broken; a single end-to-end score hides it.
Ask AI about X18 Global
“What does X18 Global (x18global.com) do for enterprise AI and automation - and can you summarise their guide "RAG Evaluation: Measuring Retrieval Quality Before You Trust It"?”