Agentic RAG Explained: When Retrieval Needs to Reason
TL;DR
- Classic RAG runs one retrieval per question and trusts whatever comes back. That works for simple lookups and falls apart on questions that need more than one hop or more than one source.
- Agentic RAG wraps retrieval in a reasoning loop: it can rewrite the query, retrieve in steps, route across sources, check its own work, and decide whether to retrieve at all.
- It costs more latency and more tokens. Use it when questions are genuinely multi-step or span systems. For single-fact lookups, plain RAG is faster, cheaper, and usually correct.
Why single-shot RAG breaks on hard questions
Classic RAG has one move. You take the user's question, turn it into an embedding, pull the top few chunks from a vector store, stuff them into the prompt, and let the model answer. One query, one retrieval, one shot. When the question is a clean lookup and the answer sits in a single chunk, this is great. It's fast, it's cheap, and you can reason about what it did.
The trouble starts the moment the question needs more than one piece of context. Ask 'how does our refund policy differ between the US and the EU plans' and a single retrieval grabs whatever is closest to that exact phrasing. If the US policy and the EU policy live in different documents, you'll often get one and not the other - and the model answers confidently with half the picture.
We've watched this fail in production more than once. The retrieval looks fine in isolation, the chunks are relevant, the answer is fluent. But it's wrong because the system never asked itself whether it had everything it needed. Single-shot RAG can't notice a gap, because noticing requires a second look - and there is no second look.
- One embedding of one query means the retriever only sees the user's phrasing, not the intent behind it.
- Multi-part questions get one retrieval, so the second and third parts often go unanswered.
- There's no check on whether the retrieved context is sufficient before the model commits to an answer.
- Vague or under-specified queries return loosely-related chunks, and the model papers over the gap with fluent guessing.
What does agentic RAG actually add?
Agentic RAG keeps the same building blocks - embeddings, a vector store, a generation step - but it puts a reasoning loop around them. Instead of retrieve-then-answer, the model gets to act: look at the question, decide what to fetch, look at what came back, and decide what to do next. Retrieval stops being a fixed step and becomes a tool the model chooses to use.
The shift sounds small and isn't. Once the model can decide, you get behaviours that single-shot RAG simply can't produce: rewriting a clumsy query into something the retriever handles well, breaking one question into several, pulling from a database for one part and documents for another, and reading its own draft to spot what's missing.
The most underrated capability is the one that does nothing. A good agentic system can decide not to retrieve at all. If the question is 'rewrite this paragraph to be shorter,' there's nothing to look up - and a system that retrieves anyway just adds noise, latency, and a chance to drag in irrelevant context.
- Query rewriting - reshaping the user's words into a query the retriever actually matches well.
- Multi-step retrieval - fetching, reading, then fetching again based on what the first pass revealed.
- Self-correction and reflection - checking the draft answer against the evidence and retrying when it falls short.
- Tool use and routing - choosing among a vector store, a SQL database, a search API, or no retrieval at all.
- Retrieve-or-not - skipping retrieval entirely when the task doesn't need outside knowledge.
Query decomposition: breaking one question into many
Decomposition is the pattern most people meet first, because it maps onto how a person would answer. Faced with 'which of our enterprise customers churned last quarter and what did their support tickets say,' you wouldn't run one search. You'd find the churned accounts, then for each one go read the tickets. Agentic RAG does the same: the model splits the question into sub-questions and retrieves for each.
The wins are concrete. Each sub-question gets its own focused retrieval, so the chunks are tighter and more relevant than one broad query would ever return. And because the steps are explicit, you can see what the system looked for - which makes debugging a wrong answer a matter of reading the trace, not guessing.
The cost is just as concrete. Three sub-questions mean three retrievals and usually three or more model calls. That's roughly triple the latency and token spend of a single shot. Decomposition earns its keep on genuinely compound questions and wastes money on simple ones, so the model needs to decide when to decompose rather than always doing it.
- Split a compound question into independent sub-questions, retrieve for each, then synthesise.
- Use it for comparisons ('A versus B'), aggregations ('all customers who...'), and multi-entity lookups.
- Each sub-retrieval is narrower, so relevance per chunk goes up.
- Watch the multiplier - every sub-question is another retrieval and another generation call.
- Let the model gate decomposition; forcing it on every query is the fast path to a slow, expensive system.
Iterative retrieve-then-reason loops
Some questions can't be split up front because you don't know the second step until you've done the first. 'What's the latest version of the integration our biggest client uses' needs you to find the client, find their integration, then find that integration's version. Each answer unlocks the next query. This is the iterative loop: retrieve, reason about what you learned, retrieve again, stop when you have enough.
The defining feature here is the stopping decision. The model has to judge, each turn, whether the evidence so far answers the question or whether one more retrieval is warranted. Done well, this is the closest RAG gets to how a careful analyst works - and it's where reflection earns its place, because the model is effectively grading its own progress at every step.
Here's the first-hand bit: the stopping logic is where these systems go wrong most often. Too eager to stop and you get the same shallow answer as single-shot RAG. Too reluctant and the loop runs five, eight, ten times, burning tokens to add nothing. We put a hard cap on iterations on every loop we build, no exceptions - it's the cheapest insurance against a runaway agent you'll ever write.
- Each iteration's result shapes the next query, so the path is data-dependent, not fixed in advance.
- The model must decide 'do I have enough' at every turn - that judgement is the whole game.
- Reflection fits naturally here: grade the current answer, then retry or stop.
- Always set a hard iteration cap so a confused loop can't run forever.
- Log every step; an iterative system you can't trace is one you can't fix.
Routing across multiple sources
Real organisations don't keep everything in one tidy vector store. Policy lives in a docs site, customer records live in a SQL database, recent activity lives behind an API, and last week's decisions live in a chat archive. A question like 'is this customer on a plan that covers the feature they just asked about' needs structured data for the plan and unstructured docs for the feature - two different sources, two different query styles.
Routing is the agentic pattern that handles this. The model classifies the query, picks the right source or sources, and translates the question into whatever each one speaks - SQL for the database, a semantic search for the docs, a parameterised call for the API. To the user it's one question; underneath it's a small orchestration.
Routing is also where the biggest accuracy gains hide, because so many 'RAG is wrong' failures are really 'we searched the wrong place.' The flip side is that each new source is another thing to authorise, rate-limit, and reason about. Add sources because a real question needs them, not because you can.
- Classify the query, then send it to the source best suited to answer it.
- Mix structured (SQL, APIs) and unstructured (vector search) retrieval in one flow.
- Translate the question into each source's native query form rather than forcing one shape on all.
- Many 'bad retrieval' bugs are routing bugs - the data existed, the system looked elsewhere.
- Every added source is added surface area for latency, auth, and failure - add them deliberately.
When is the added latency and cost worth it?
Agentic RAG is not a free upgrade. A single-shot pipeline answers in one retrieval and one generation. An agentic one might rewrite the query, retrieve three times, reflect once, and synthesise - that's several model calls and several round-trips to your data, which users feel as seconds and you feel on the bill. The question isn't 'is agentic better,' it's 'does this workload need it.'
Here's the decision framework we actually use. Score the workload on three things: how often questions need more than one hop or one source, how costly a confidently-wrong answer is, and how much latency budget you have. High multi-hop, high cost-of-wrong, generous latency points straight at agentic. Low on all three points at plain RAG, and you should feel no guilt about it.
A practical middle path: start with plain RAG, log the questions it gets wrong, and look at why. If the failures cluster around multi-part or cross-source questions, add the specific agentic pattern that fixes that cluster - decomposition, or routing, or a loop. Buying the whole agentic stack before you've seen the failure modes is how you end up with a slow system solving problems you didn't have.
- Lean agentic when questions are genuinely multi-hop, span sources, or carry a high cost of being wrong.
- Stay with plain RAG for single-fact lookups, FAQ-style answers, and anything latency-sensitive.
- Score workloads on multi-hop frequency, cost-of-wrong, and latency budget before deciding.
- Start simple, measure the failures, then add only the agentic pattern that addresses them.
- Treat every extra model call as a cost you're choosing to pay, not a default.
How do you evaluate an agentic RAG system?
You can't manage what you don't measure, and agentic RAG gives you more to measure than the classic kind. Beyond 'was the final answer right,' you want retrieval quality at each step (did we fetch the relevant context), faithfulness (is the answer grounded in what we retrieved rather than invented), and the cost profile (calls, tokens, and wall-clock time per question).
Build a fixed evaluation set of real questions with known good answers, and include the awkward ones - the multi-hop, cross-source, and 'this needs no retrieval' cases that single-shot RAG flubs. Run every change against it. Without that harness you're tuning prompts by vibes, and agentic systems have too many moving parts for vibes to hold up.
If you want a second pair of hands building that harness before you ship, that's exactly the kind of thing our team sets up early - because the eval suite is what tells you whether the extra machinery is paying for itself. The honest test is comparative: run the same set through plain RAG and through your agentic version. If the agentic one isn't clearly better on the questions that matter, you're paying for complexity you don't need.
- Measure per-step retrieval quality, not just whether the final answer reads well.
- Track faithfulness - is the answer grounded in retrieved evidence, or quietly made up?
- Record cost per question: model calls, tokens, and latency, so regressions show up.
- Keep a fixed test set heavy on multi-hop and cross-source questions.
- Always benchmark agentic against plain RAG on the same set - complexity has to earn it.
Failure modes: loops, runaway cost, and over-retrieval
The failure that scares people most is the runaway loop. A model that keeps deciding it needs one more retrieval, never satisfied, can spin until it hits a timeout or a frightening bill. The fix is boring and non-negotiable: a hard cap on iterations, a token budget per request, and a fallback answer for when the cap is hit. Build these in from the first version, not after the first incident.
The quieter failures cost you in ways that don't show up as an outage. Over-retrieval pulls in chunks nobody needed and dilutes the prompt, so answer quality drops even though more context went in. Reflection loops can talk themselves out of a correct answer. And a router that picks the wrong source produces a confident answer from the wrong data - the hardest kind of wrong to catch, because everything downstream looks healthy.
Most of these trace back to giving the model too much rope with too little instrumentation. The defence is the same posture every time: caps on everything that can iterate, logs on every decision the agent makes, and an eval set that includes the failure shapes you've already seen. An agentic system you can observe is one you can keep honest.
- Runaway loops - cap iterations, cap tokens, and define a fallback when the cap trips.
- Over-retrieval - more context isn't better context; irrelevant chunks lower answer quality.
- Reflection drift - a self-critique loop can reject a right answer; bound the retries.
- Mis-routing - the wrong source gives a confident wrong answer that looks fine downstream.
- Blind operation - without per-decision logging you can't tell which of the above just happened.
Start here
Don't build agentic RAG because it's the interesting version. Build plain, single-shot RAG first, put it in front of real questions, and write down every answer it gets wrong. That log is the most useful document you'll have, because it tells you exactly which agentic pattern - if any - you actually need.
When the failures cluster, add one pattern, not all of them. Multi-part questions failing? Add decomposition. Answers pulling from the wrong place? Add routing. Questions that need a chain of lookups? Add a bounded iterative loop. Re-run your eval set after each addition and keep only what clearly pays for itself.
The teams that get this right treat agentic RAG as a set of targeted upgrades to a working baseline, with caps and logs from day one - not a grand architecture they commit to before seeing a single real failure. Start simple, measure honestly, and let the questions tell you how much reasoning your retrieval really needs.
- Ship plain RAG first and collect a real log of its wrong answers.
- Add agentic patterns one at a time, each one tied to a failure cluster you've observed.
- Put iteration caps, token budgets, and per-step logging in from the very first version.
- Re-run a fixed eval set after every change and keep only what beats the baseline.
- Let the workload, not the trend, decide how much reasoning retrieval needs.
Want this built for your business?
We map the highest-leverage place to start and ship a first live system within two weeks.
Book a strategy callCommon questions
What is the difference between RAG and agentic RAG?
Classic RAG runs one retrieval per question and answers from whatever comes back - one query, one shot. Agentic RAG wraps retrieval in a reasoning loop so the model can rewrite the query, retrieve in multiple steps, route across sources, check its own answer, and even decide not to retrieve at all. Same building blocks, but retrieval becomes a tool the model chooses to use rather than a fixed step.
When should I use plain RAG instead of agentic RAG?
Use plain RAG when questions are single-fact lookups, FAQ-style answers, or anything where the answer lives in one place and latency matters. It's faster, cheaper, and easier to reason about. Reach for agentic RAG only when questions are genuinely multi-hop, span several data sources, or carry a high cost of being confidently wrong.
Does agentic RAG cost more than standard RAG?
Yes. A single-shot pipeline is one retrieval and one generation. An agentic flow might rewrite the query, retrieve several times, reflect, and synthesise - that's multiple model calls and multiple round-trips to your data, which shows up as higher latency and token spend. The extra cost is worth it only when the workload genuinely needs multi-step reasoning.
How do you stop an agentic RAG system from looping forever?
Put hard limits in from the first version: a cap on the number of retrieval iterations, a token budget per request, and a defined fallback answer for when a cap is hit. Pair those with per-decision logging so you can see when and why the agent kept going. Caps and logs are the cheapest insurance against a runaway loop and a surprise bill.
How do you evaluate an agentic RAG system?
Measure more than the final answer. Track per-step retrieval quality, faithfulness (is the answer grounded in retrieved evidence), and cost per question in calls, tokens, and latency. Keep a fixed test set heavy on multi-hop and cross-source questions, and always benchmark the agentic version against plain RAG on the same set - if it isn't clearly better on the questions that matter, the added complexity isn't earning its place.
Classic RAG runs one retrieval per question and answers from whatever comes back - one query, one shot. Agentic RAG wraps retrieval in a reasoning loop so the model can rewrite the query, retrieve in multiple steps, route across sources, check its own answer, and even decide not to retrieve at all. Same building blocks, but retrieval becomes a tool the model chooses to use rather than a fixed step.
Ask AI about X18 Global
“What does X18 Global (x18global.com) do for enterprise AI and automation - and can you summarise their guide "Agentic RAG Explained: When Retrieval Needs to Reason"?”