Re-ranking: the underrated step that fixes most RAG problems
TL;DR
- Your first-stage retriever (vector or keyword search) is built for recall, not precision. It is good at pulling the right document into a big pile and bad at putting it at the top.
- A re-ranker is a second pass that reads each candidate against the actual query and reorders them. It is the cheapest, highest-leverage fix for the common 'right doc was retrieved but ranked 8th' failure.
- Pull 50-100 candidates, re-rank, keep the top 3-5. Add a hosted re-ranker API first, measure the lift with recall@k before and after, and only self-host once volume justifies it.
Why your RAG answers are wrong even when the right doc is in the index
Here is the pattern we see on almost every RAG project that lands on our desk in trouble. Someone asks the system a question, gets a vague or wrong answer, and assumes the document was never indexed. So they re-chunk, re-embed, swap the vector database, and burn two weeks. Then we run a quick check: was the correct passage actually retrieved? Almost always, yes. It was sitting at position 8 in a list of 10, and the model only ever saw the top 4.
That is the whole problem in one sentence. The right answer made it into the candidate set, but it did not make it to the top, so it never reached the prompt. The retriever did its job. The ranking failed.
Once you see this distinction - retrieval versus ranking - most RAG debugging gets a lot calmer. You stop blaming the index and start fixing the order.
- Failure looks like a bad answer, but the cause is usually bad ordering, not a missing document.
- If you only feed the model the top 3-5 chunks, anything ranked below that is invisible no matter how relevant it is.
- Re-chunking and re-embedding rarely fix a ranking problem - they just move the noise around.
- The first diagnostic question is always: 'was the correct passage in the retrieved set at all?' Check before you rebuild.
Why first-stage retrieval optimises for recall, not precision
Vector search and keyword search are both built to be fast over millions of chunks. To stay fast, they compare a single query vector against a single chunk vector with a cheap similarity score, or they match terms. They never read the query and the chunk together. That design choice is what makes them scale, and it is also what makes them blunt.
So the honest job of first-stage retrieval is recall: get the relevant material somewhere into the candidate pile, even if the ordering inside that pile is rough. It is optimised to not miss the answer, not to rank it first. A semantic match on a paraphrase, a document that mentions your key term once in passing, a chunk that is topically close but answers a different question - all of these can score higher than the passage you actually need.
That is fine. It is the contract. You are supposed to pull a generous candidate set and let a second, smarter stage sort out precision. The mistake is treating the first stage's top 3 as the final answer.
- Vector similarity compares embeddings independently - it never sees query and document side by side.
- Keyword search (BM25 and friends) matches terms and term frequency, blind to meaning and intent.
- Both are tuned to be fast and high-recall over huge corpora, which trades away fine-grained precision.
- Treat the first stage as a net that catches the answer, not a ranking you can trust as-is.
What a re-ranker actually is: cross-encoder vs bi-encoder in plain terms
Your embedding model is a bi-encoder. It encodes the query into a vector once, encodes each document into a vector once, and compares them with a quick distance calculation. The two pieces of text never meet inside the model. That separation is exactly why it is fast enough to search millions of chunks in milliseconds - and why it misses nuance.
A re-ranker is usually a cross-encoder. It takes the query and one candidate document together, as a single input, and runs them through the model so every word in the query can attend to every word in the document. The output is one relevance score for that exact pair. Because the model reads both texts jointly, it catches things the bi-encoder cannot: negation, qualifiers, whether the document actually answers the question or just mentions the topic.
The trade is obvious once you say it out loud. A bi-encoder is cheap because it compares pre-computed vectors. A cross-encoder is expensive because it runs a full model pass for every query-document pair. You cannot run a cross-encoder over a million chunks per query. You can absolutely run it over the 50-100 candidates the first stage already narrowed down. That is the entire trick.
- Bi-encoder (your embedder): encodes query and docs separately, compares vectors - fast, scales to millions, less precise.
- Cross-encoder (the re-ranker): reads query and one doc together, outputs a single relevance score - slow, far more precise.
- Cross-encoders catch negation, intent, and 'mentions vs answers' distinctions that vector distance flattens.
- You never run a cross-encoder over the whole index - only over the shortlist the first stage produced.
The two-stage retrieve-then-rerank pattern, step by step
The pattern is simple enough to fit on a napkin, and it is the architecture behind most RAG systems that actually work in production. Stage one casts a wide net. Stage two reads carefully and reorders. Then you trim to what fits in the prompt.
In practice the flow looks like this: take the user query, run your usual vector or hybrid search, and pull a generous candidate set - more than you intend to use. Pass every candidate, paired with the query, through the re-ranker. The re-ranker returns a fresh relevance score per candidate. Sort by that score, keep the top few, and hand only those to the model.
The beauty is that you bolt this on without ripping anything out. Your index, your embeddings, your chunking - all stay. You are inserting one step between 'search returned results' and 'build the prompt.' If you have ever wanted a RAG fix that does not mean rebuilding the pipeline, this is the one. When teams ask us where to get the most quality per hour of engineering, this is nearly always the first thing we point at.
- Step 1: run vector or hybrid search and retrieve a wide candidate set (think 50-100, not 5).
- Step 2: score each (query, candidate) pair with the cross-encoder re-ranker.
- Step 3: sort by the new score, keep the top 3-5, discard the rest.
- Step 4: build the prompt from only the survivors - the model now sees a clean, precise context.
- It is additive: no need to re-index, re-embed, or change your chunking to adopt it.
How many candidates to pull and how many to keep
This is where most of the tuning lives, and it is two separate numbers people tend to collapse into one. The first number is how many candidates you retrieve from the first stage and feed into the re-ranker. The second is how many you keep after re-ranking and put in the prompt. They pull in opposite directions.
Retrieve too few candidates and the re-ranker can only reorder what it was handed - if the right doc was at position 60 and you only pulled 20, the re-ranker never sees it and cannot save you. Retrieve too many and you pay more latency and more re-ranker cost for diminishing returns. A sensible starting point is to retrieve 50-100 and keep 3-5, then adjust based on what your measurements show.
The keep number is bounded by your prompt budget and by a subtler effect: more context is not always better. Stuffing 15 chunks into the prompt often makes answers worse, because the genuinely relevant passage gets diluted by near-misses. A tight top 3 from a good re-ranker usually beats a loose top 10 from raw vector search.
- Two knobs, not one: candidates retrieved into the re-ranker vs results kept after it.
- Retrieve wide (50-100) so the re-ranker has a real chance to find the buried answer.
- Keep narrow (3-5) so the model gets a clean, high-signal context window.
- More kept chunks can hurt - dilution from near-misses drags answer quality down.
- Tune the retrieve number against recall@k; tune the keep number against answer quality and prompt budget.
The cost and latency trade-off, and how to keep it sane
Nothing is free, and the re-ranker adds a step. A cross-encoder pass over 50-100 candidates adds latency - typically tens to low hundreds of milliseconds with a hosted API, more if you self-host on undersized hardware. If you are calling a hosted re-ranker, you also pay per query and per candidate scored. So the cost scales with how wide you cast the net.
The good news is that this is one of the better-value milliseconds in your whole pipeline. The re-ranker step is almost always small next to the language model's own generation time, which is usually the dominant latency in a RAG response. Adding 80ms of re-ranking in front of a 2-second generation is noise, and the answer quality jump is large. You are spending a little time up front to stop wasting a lot of tokens on irrelevant context.
If latency does bite, you have levers. Retrieve fewer candidates. Use a smaller or distilled re-ranker model. Cache re-rank results for repeated queries. Run the re-ranker in parallel batches rather than one pair at a time. Most teams find the default settings are already fast enough and never touch these.
- Re-ranking adds latency proportional to the candidate count - more candidates, more cost and time.
- It is usually small next to LLM generation time, so the user barely notices it.
- Hosted APIs bill per query and per candidate scored - the retrieve-wide number drives your bill.
- Levers if it hurts: fewer candidates, smaller model, result caching, batched parallel scoring.
- Cleaner context also means fewer wasted prompt tokens, which claws back some of the cost.
Hosted re-ranker APIs vs self-hosting your own
You have two roads. A hosted re-ranker API gives you a managed cross-encoder behind an endpoint - you send the query and candidates, you get scores back. Self-hosting means running an open-source cross-encoder on your own infrastructure, on a GPU or a well-specced CPU box, and owning the serving.
Start hosted, almost every time. It is one API call, there is no model to manage, and it lets you prove the lift before you commit any infrastructure. You will know within an afternoon whether re-ranking helps your data, and that answer is worth far more than the infra savings you might eventually get. We have watched teams spend a month standing up a self-hosted re-ranker before confirming it even moved their numbers - do it the other way around.
Self-hosting earns its keep later, under specific pressure: high query volume where per-call pricing stops being cheap, strict data-residency or privacy rules that forbid sending content to a third party, or latency targets you can only hit on co-located hardware. When two or three of those are true at once, owning the re-ranker makes sense. Until then, the hosted call is the right default.
- Hosted API: one endpoint, no infra, fastest path to a yes/no on whether re-ranking helps you.
- Self-hosted: open-source cross-encoder on your own GPU or CPU, you own latency and serving.
- Choose self-hosting for high volume, data-residency or privacy constraints, or tight latency targets.
- Prove the lift on a hosted API first - never build serving infra for a benefit you have not measured.
- The decision is mostly about volume and compliance, not about model quality - both roads use similar models.
When you do not need a re-ranker, and how to measure the lift
Re-ranking is high-leverage, but it is not mandatory. If your corpus is small and your queries are simple keyword lookups, plain vector or hybrid search may already put the right chunk first. If your retrieval is already returning the correct passage at position 1 most of the time, a re-ranker has little to reorder and you are adding latency for almost nothing. And if your real problem is that the answer is not in the index at all, re-ranking cannot conjure it - that is a chunking or coverage problem, and you fix it upstream.
So measure before and after, do not guess. Build a small evaluation set of real queries paired with the chunk or chunks that should answer them - even 30 to 50 examples is enough to see a clear signal. Run retrieval without the re-ranker and record recall@k and the rank of the correct chunk. Then run it with the re-ranker and compare. The metric that matters most is whether the correct passage moves into your keep window (top 3-5).
The first time you run this you will usually see one of two clean outcomes. Either re-ranking lifts your correct chunk from the middle of the pack into the top three and your answer quality jumps, or it barely moves the numbers because your first stage was already good - in which case you have just saved yourself a dependency. Both results are useful. Guessing is not.
- Skip it when: the corpus is small, queries are simple, or position 1 is already right most of the time.
- Re-ranking cannot help if the answer was never retrieved - that is a coverage or chunking fix, not a ranking one.
- Build a 30-50 query eval set with known-correct chunks before you change anything.
- Track recall@k and the rank of the correct chunk, before vs after re-ranking.
- The key signal: does the right passage move into your top 3-5 keep window? If yes, ship it.
Start here
If your RAG system gives shaky answers, resist the urge to rebuild the index. Do the cheap check first: pull your candidate set wider and look at whether the correct passage is in there but ranked low. If it is - and it usually is - you have a ranking problem, and a re-ranker is the fix.
Wire up a hosted re-ranker, retrieve 50-100 candidates, keep the top 3-5, and run it against a small eval set so you can see the lift in numbers rather than vibes. That is an afternoon of work for what is often the single biggest quality jump in the whole pipeline.
Most RAG problems are ranking problems wearing a retrieval costume. Re-ranking is the underrated step that pulls the mask off. Start there before you start anywhere more expensive.
- First move: widen your candidate set and confirm whether the right doc is present-but-buried.
- If it is buried, add a hosted re-ranker before touching chunking, embeddings, or your vector DB.
- Default settings to start: retrieve 50-100, re-rank, keep 3-5.
- Measure recall@k and correct-chunk rank on a small eval set so the lift is provable.
- Reach for the expensive rebuilds only after re-ranking has not closed the gap.
Want this built for your business?
We map the highest-leverage place to start and ship a first live system within two weeks.
Book a strategy callCommon questions
What is the difference between retrieval and re-ranking in RAG?
Retrieval is the first stage - vector or keyword search pulls a wide set of candidate chunks from your index, optimised for recall so it does not miss the answer. Re-ranking is a second stage that reads each candidate against the query with a cross-encoder and reorders them for precision, so the genuinely relevant chunks rise to the top before they reach the model.
Why is the right document retrieved but ranked too low to use?
Because first-stage retrieval compares the query and each document as separate vectors and never reads them together. That makes it fast but blunt, so a topically-close-but-wrong chunk can outscore the passage you actually need. The answer ends up in the candidate set but at position 8, below the top 3-5 you feed the model. A re-ranker reads each pair jointly and fixes the order.
How many documents should I retrieve and keep when re-ranking?
A good starting point is to retrieve 50-100 candidates from the first stage into the re-ranker, then keep the top 3-5 after re-ranking for the prompt. Retrieve wide so the re-ranker has a real chance to surface a buried answer, and keep narrow so the model gets clean context. Tune the retrieve number against recall@k and the keep number against answer quality and your prompt budget.
Does re-ranking slow down my RAG system?
It adds some latency - usually tens to low hundreds of milliseconds with a hosted API, scaling with how many candidates you score. In practice that is small next to the language model's own generation time, which dominates the response. If latency does become an issue, you can retrieve fewer candidates, use a smaller model, cache results, or batch the scoring.
Should I use a hosted re-ranker API or self-host one?
Start with a hosted API. It is a single call with no infrastructure to manage, so you can prove whether re-ranking actually helps your data within an afternoon. Move to self-hosting later only when high query volume, data-residency or privacy rules, or strict latency targets justify owning the serving. Both roads use similar cross-encoder models, so the choice is about volume and compliance, not quality.
Retrieval is the first stage - vector or keyword search pulls a wide set of candidate chunks from your index, optimised for recall so it does not miss the answer. Re-ranking is a second stage that reads each candidate against the query with a cross-encoder and reorders them for precision, so the genuinely relevant chunks rise to the top before they reach the model.
Ask AI about X18 Global
“What does X18 Global (x18global.com) do for enterprise AI and automation - and can you summarise their guide "Re-ranking: the underrated step that fixes most RAG problems"?”