People think the intelligence in agentic RAG lives in retrieval. It doesn’t. Retrieval is a solved-ish mechanical step: embed a string, search a store, get chunks back. The intelligence lives one step earlier, in deciding what string to embed — and the cruel part is that you have to decide it before you know the answer. That is query planning, and it is the part that actually separates a system that works from one that loops uselessly.
The chicken-and-egg problem
A user asks: “Did the vendor we onboarded after the Q2 incident pass their latest security review?”
To retrieve the right document, you’d need to already know which vendor that is. But you only learn which vendor by retrieving first. The query you’d write if you had the answer is not the query you can write at the start. This is the core tension of planning. The good query depends on information you don’t have yet.
There are three moves a planner makes to break out of this, and a real system uses all three.
Move one: reformulation
The raw user question is often a terrible search query. It carries conversational framing, pronouns, and assumptions that pollute the embedding. “Can you remind me what we decided about that?” embeds to noise. Reformulation rewrites the question into something a retriever can actually use, pulling in context from the conversation so “that” becomes “the data retention period for EU customer records.”
This is the cheapest, highest-return planning step there is. Before any fancy decomposition, just ask the model to turn the user’s message into a clean, self-contained search query. I’ve seen this single step lift retrieval relevance noticeably on conversational systems, because the original phrasing was sabotaging the embedding the whole time.
Move two: decomposition
Some questions are secretly several questions. “Which agencies that funded our competitor also have open solicitations we qualify for?” is three retrievals wearing one sentence:
sub-query 1: contracts awarded to <competitor> in the last 12 months
-> extract: list of awarding agencies
sub-query 2: open solicitations from <those agencies>
-> filter: matching our NAICS codes
sub-query 3: our qualification criteria / past performance
-> reconcile against the filtered solicitationsNotice that sub-query 2 cannot be written until sub-query 1 returns. The plan is not a static list you generate up front. It unfolds. Each step’s query is built from the previous step’s result. This is the same multi-hop structure I cover separately, and planning is the engine that drives it. The planner’s job is to recognize that a question decomposes at all, and then to sequence the pieces so each one feeds the next.
A static plan generated before any retrieval is a guess. A plan that adapts after each result is reasoning. The second one is what you actually want, and it is harder to build.
Move three: routing
Not every sub-query should hit the same place. “What’s our current cloud spend?” is not a vector-search question — it’s a structured query against a billing API or a database. “What did we decide about retention?” is a document-retrieval question. A planner that treats everything as semantic search over one vector store is leaving the easy wins on the table.
Routing is the planner deciding which tool answers each sub-query: vector store, SQL, a live API, a keyword index. On a federal-contractor intelligence engine I architected, a single user question routinely fanned out to public federal award data for solicitation history, a Postgres store for our internal records, and a vector index over past proposals. The planner decided the split. Get routing wrong and you semantic-search a question that wanted an exact number, and you get a fluent paragraph instead of the figure the user needed.
Static plans versus adaptive plans
There are two schools, and the gap between them is where most agentic RAG systems succeed or fail.
A plan-then-execute approach generates the full sequence of sub-queries up front, then runs them. It’s predictable and cheap — one planning call, then a batch of retrievals. It works when the decomposition doesn’t depend on intermediate results. “Compare the pricing of product A and product B” is two independent lookups; you can plan both immediately.
An interleaved approach plans one step, retrieves, looks at what came back, then decides the next step. It’s more expensive — a model call between each hop — but it handles the chicken-and-egg case, where step two genuinely cannot be written until step one returns. The vendor-security question demands this. You don’t know the vendor until you’ve retrieved it.
My rule: start with plan-then-execute because it’s simpler and you can reason about it. Move to interleaved only for the queries that provably need it, the ones where a sub-query references an entity you can only learn by retrieving. Most decompositions don’t. The ones that do are worth the extra calls precisely because nothing cheaper can answer them.
Where planning goes wrong
The failure modes are specific and worth naming, because they don’t look like failures — they look like a system working hard.
Over-decomposition. The planner splits a simple question into six sub-queries when one would do, turning a 700ms lookup into a six-call, eight-second ordeal. A blunt guardrail helps: cap the number of hops, and make the planner justify going past one.
Drift. Each reformulation moves slightly away from the original intent until, three hops in, the system is confidently retrieving documents about something the user never asked. Anchoring every sub-query back to the original question in the prompt keeps this in check.
False decomposition. The planner sees structure where there’s none and invents sub-questions that don’t map to retrievable facts. This burns calls and pollutes the final context with irrelevant chunks. A relevance grader on each sub-result — the self-correction step I treat on its own — catches it, but it’s better to not generate the bad sub-query in the first place.
The thing to internalize
Good retrieval is mostly good question-asking. The model that drives an agentic loop spends most of its real intelligence not on reading documents but on deciding which documents to even look for, in what order, against which tool. When my retrieval systems fail, the post-mortem almost never says “the vector search returned the wrong chunks.” It says “we asked the index the wrong question.”
So when you build the loop, spend your time on the planner, not the retriever. Instrument the sub-queries it generates and read them like a code review — are these the questions a sharp analyst would ask to answer this? If the sub-queries are good, the chunks take care of themselves. If they’re bad, no reranker, no bigger model, no longer context window saves you. The system is only ever as smart as the questions it knows to ask.