A client once asked me to make their support bot “agentic” because a competitor’s marketing said the word a lot. Their actual system was a one-shot RAG pipeline over 400 help-center articles, answering FAQ-shaped questions in under a second for a few cents a day. It was correct around 92% of the time. They wanted to spend a quarter of engineering time turning that into a multi-step retrieval loop.
I talked them out of it. Here is the reasoning, because it applies far more often than the agentic-RAG enthusiasm online would suggest.
The default should be the simple thing
I build the simplest thing that works and I make it earn every bit of added complexity. An agentic loop is complexity with teeth: more LLM calls, higher latency, a new class of failure modes, and a system whose behavior is harder to predict because the model is now deciding its own control flow. None of that is free. You should reach for it only when one-shot retrieval genuinely cannot answer the question, not because a loop sounds more sophisticated.
Regular RAG retrieves once and answers. That single pass is a feature, not a limitation, for a large and boring category of work — and most production traffic is boring in the best way.
What one-shot RAG is genuinely good at
The pattern fits when three things are true at once.
The answer lives in one place. A policy lookup, a product spec, an API parameter, a definition, a single clause in a contract. If the fact is contained in a chunk or two that a single semantic search will reliably surface, you do not need a loop. You need a good index and a good embedding model.
The question and the answer share vocabulary. “What’s your refund window?” retrieves the refund policy because the words overlap and the concepts cluster. Where the query language matches the document language, one embedding gets you there. The hard cases — answers spread across documents with no shared keywords — are exactly where I’d switch to a multi-hop loop. But those are a minority of real traffic for most products.
Latency and cost matter. This is the one people forget. A support widget, an in-app assistant, anything a human waits on, lives or dies on responsiveness. One-shot RAG lands answers in roughly 600 to 900 ms. An agentic loop with three to eight model calls turns that into 3 to 15 seconds. For a user staring at a spinner, that is the difference between “this is fast” and “this is broken.”
The cost gap is the same story. Here is the rough shape at scale:
| One-shot RAG | Agentic loop | |
|---|---|---|
| LLM calls / query | 1 | 3-8 |
| Latency | ~0.7s | 3-15s |
| Relative cost | 1x | 4-10x |
| At 100k queries/day | manageable | a budget conversation |
Multiply that 4-10x by serious volume and the loop is not an architecture decision anymore. It is a line item your CFO will ask about.
Predictability is underrated
The argument I find most persuasive has nothing to do with money. A one-shot pipeline has a fixed shape. The same question takes the same path every time, with the same number of calls and a tight latency band. You can load-test it, you can put a hard SLA on it, you can reason about it when it breaks.
An agentic loop is the model deciding its own control flow at runtime. That is power, and it is also a system that can take a different route every time you run it.
A loop can reformulate a query in a direction that drifts away from the original intent. It can declare thin context “sufficient” and stop early, or never be satisfied and keep retrieving. It can take three calls on Monday and seven on Tuesday for the same question because the model’s intermediate judgment shifted. When you operate these systems on call at 2am, deterministic shape is worth more than marginal accuracy. You cannot page someone for a loop that is merely slow today.
The honest test
Before adding an agentic layer, I run a cheap experiment. Take a representative sample of real questions, run them through plain one-shot retrieval, and read the retrieved chunks by hand before looking at the generated answers. Ask one question: was the fact needed to answer this actually in the retrieved context?
If the answer is yes 90%+ of the time, you do not have a retrieval problem that a loop will fix. Your failures are coming from generation, or prompting, or chunking — and a loop fixes none of those. It just adds cost on top of a system that was already finding the right context. I have watched teams add elaborate planning and self-correction machinery to fix what was, on inspection, a bad chunking strategy. The loop kept looping over chunks that had been mangled at index time. No retrieval cleverness recovers a fact that was destroyed before it ever entered the store.
If instead you find the needed fact is missing from the retrieved context in a meaningful fraction of cases, now you have a real decision. Sometimes the fix is still not a loop — it is better chunks, hybrid search, a reranker. Only when the fact is missing because it requires reasoning across multiple retrievals do you actually need agentic RAG.
Where I draw the line in practice
For most products I ship, the architecture is a hybrid, and the regular path carries the load. One-shot retrieval handles the bulk of traffic — the FAQ-shaped, single-document, shared-vocabulary questions that are most of what users ask. A lightweight relevance check sits on its output. Only when that check flags thin context does the request escalate into a fuller loop.
That is exactly how I’m wiring retrieval into Vontra, the AI-native operating system I’m building. Most of what a business asks of its own data is one fact in one place, and forcing those through an agentic loop would only make a fast product feel slow.
The result is that 70 to 80% of queries never touch an agent. They take the fast, cheap, predictable path, because that path is correct for them. The loop exists for the minority of questions that earn it.
Regular RAG is not the training-wheels version of agentic RAG. It is the right tool for a specific and very common shape of problem: one fact, one place, shared words, and a user who is waiting. Use the loop when the question genuinely spans documents and reasoning. Use the straight line everywhere else, and feel no need to apologize for it. The most sophisticated thing you can do is not add a loop. It is knowing precisely when you don’t need one.