Here is a question that breaks regular RAG completely, and it’s worth sitting with why: “What’s the email address of the contracting officer for the agency that awarded the most cybersecurity contracts in our NAICS code last year?”
Embed that whole sentence, search your vector store, and you get back a pile of chunks vaguely about contracts and cybersecurity and email. None of them contain the answer, because the answer doesn’t exist in any single document. It has to be assembled:
hop 1: cybersecurity contracts in NAICS 541512, last year, grouped by agency
-> answer: Agency X awarded the most
hop 2: contracting officer assigned to Agency X for that vehicle
-> answer: a specific person
hop 3: contact record for that person
-> answer: the emailThree hops. Each one’s query is the previous one’s answer. And here’s the part that kills one-shot retrieval dead: the three documents involved share almost no keywords with the original question or with each other. The contract data, the officer assignment, and the contact record live in different places, in different vocabularies. No single embedding could point at all three. There is no “top-k” that contains the answer, because the answer was never co-located.
Why one embedding can’t do this
Vector search finds documents semantically near a query. That works beautifully when the answer lives somewhere near the question in concept-space. It fails structurally when the answer requires a chain, because each link in the chain is near a different point, and the points you need to visit are only knowable in sequence.
Think about hop 2 above. You cannot write the query “contracting officer for Agency X” until hop 1 has told you that Agency X is the answer. At the moment the user asks the question, “Agency X” is unknown. The query that would retrieve it does not yet exist. No amount of clever embedding of the original sentence conjures it, because the information needed to form the query is itself the output of a prior retrieval.
Regular RAG asks one question. Multi-hop questions are a question whose later halves are written by the answers to their earlier halves.
This is the cleanest example of why agentic RAG isn’t a fancier flavor of regular RAG. It’s the only architecture that can produce a query after learning something. The loop is not an optimization. For these questions it is the difference between an answer and a hallucination.
The shape of a multi-hop loop
Mechanically, multi-hop is the planner and the self-correction grader working together across iterations:
state = { question, facts_gathered: [] }
while not can_answer(state):
next_query = plan_next_hop(state) # uses facts_gathered so far
chunks = retrieve(next_query)
relevant = grade(chunks, next_query) # drop the noise
state.facts_gathered += extract(relevant)
if no_progress(state): # guard against infinite loops
break
answer = generate(state)Every piece here has a job. plan_next_hop decides what to ask given what’s known — that’s query planning, and in multi-hop it runs once per hop rather than once per question. grade keeps each hop’s results clean so junk from hop 1 doesn’t poison hop 2’s query. extract pulls the specific fact (the agency name, the officer’s name) that becomes the bridge to the next hop. can_answer checks whether the gathered facts now cover the full question — that’s gap detection deciding when to stop.
The bridges matter most. A multi-hop system lives or dies on extracting the right entity from each hop to carry forward. Get “Agency X” wrong in hop 1 and every subsequent hop is confidently retrieving facts about the wrong agency. The error compounds. By hop 3 you have a fluent, fully-grounded, completely wrong email address.
What it costs
Multi-hop is the most expensive pattern in retrieval and you should know the bill before you sign up:
| One-shot | 3-hop loop | |
|---|---|---|
| Retrievals | 1 | 3 |
| LLM calls (plan + grade + generate) | 1 | 7-10 |
| Latency | ~0.7s | 5-12s |
| Relative cost | 1x | 6-10x |
| Error compounding | none | each hop can derail the next |
That error-compounding row is the real cost, not the dollars. A one-shot system fails independently per query. A multi-hop system can fail cumulatively — a small mistake in hop 1 is amplified by hops 2 and 3. This is why self-correction isn’t optional here. You grade every hop precisely because an ungraded bad hop doesn’t just give one wrong fact, it corrupts the entire remaining chain.
When to actually build it
Don’t reach for multi-hop because it’s impressive. Reach for it when your real questions have this property: the answer requires a fact that can only be retrieved using another fact you must retrieve first. That’s the test. If you can’t write the final query at the moment the question arrives — if part of the query is itself a thing you have to look up — you need hops.
The question that opens this piece isn’t hypothetical for me. I built a federal-contractor intelligence platform whose whole job was to answer exactly that shape of question — chaining award data, officer assignments, and contact records that lived in entirely separate sources. That system would have been impossible as one-shot RAG. The chain was the product.
But most traffic does not have this property. “What’s your refund policy?” is one hop. “What did we decide about retention?” is one hop. For those, a multi-hop loop is pure waste: extra latency, extra cost, extra failure surface, to answer a question one lookup already nailed. I run multi-hop as an escalation path, not a default. A request only enters the hop loop when the planner recognizes the question references an entity that isn’t yet known — otherwise it takes the fast single-shot path and answers in under a second.
And the usual caveat, which is never not true: hops can’t reach a fact that chunking destroyed. If hop 1 needed a table that got split across two chunks at index time, the chain breaks at the first link and no loop length saves it. The cleverest multi-hop reasoning in the world still bottoms out on what survived indexing.
The point
Multi-hop retrieval is the clearest case for why agentic RAG exists at all. There’s a whole class of real questions — the ones that span documents sharing no vocabulary, where each lookup depends on the last — that one-shot retrieval can’t answer and will never tell you it can’t answer. It’ll just hand you something fluent and wrong. The loop’s value isn’t speed or polish. It’s that it can walk a chain of facts no single embedding could ever point at, and stop when the chain is complete. Build it for exactly those questions, grade every hop because the errors compound, and leave everything else on the fast path where it belongs.