Regular RAG has no idea whether it’s about to lie to you. It retrieves whatever the vector search returns, hands it to the model, and the model produces a fluent answer regardless of whether the context actually supports it. The pipeline never asks “is this enough?” because there is no step whose job is to ask. That missing step — the system checking its own work — is the single capability that makes agentic RAG worth its cost.
Let me break down what “checking its own work” actually means in code, because it’s three distinct moves and people blur them together.
Move one: grade the retrieval
The first and cheapest self-correction is grading the chunks before you generate anything. Cosine similarity tells you a chunk is semantically near the query. It does not tell you the chunk is relevant. Those are different things, and the gap between them is where bad answers come from.
So you add a grader: a small, fast LLM call that looks at each retrieved chunk and the question and answers one thing — does this chunk actually help answer this question, yes or no?
for chunk in retrieved:
verdict = grade(question, chunk) # cheap model, binary
if verdict == "relevant":
keep.append(chunk)
if len(keep) == 0:
# nothing we retrieved is actually useful
reformulate_and_retry()This is a guard against the quiet failure: the top-k came back, every chunk scored 0.8 on similarity, and none of them contain the answer. Without the grader, all eight go into the prompt and the model hallucinates a plausible answer from adjacent-but-wrong context. With it, the system notices the context is empty of real signal and goes back for more instead of guessing.
The grader is cheap relative to what it prevents. A binary relevance check on a small model runs in a couple hundred milliseconds and a fraction of a cent. The wrong answer it stops can cost you a customer’s trust or a contract.
Move two: detect the gap
Grading individual chunks tells you what’s irrelevant. It doesn’t tell you what’s missing. Gap detection is the harder question: given the relevant context I now hold, can I actually answer the full question, or is there a hole?
The most dangerous answer a RAG system gives is the one that’s 80% grounded and 20% invented, because the grounded 80% makes the invented 20% sound true.
This is where decomposition and self-correction meet. If the question asks for three things and the retrieved context only covers two, a system that just generates will fabricate the third in the same confident voice as the first two. Gap detection means the model inventories what the question requires, checks that inventory against what it holds, and flags the missing piece as a new retrieval target instead of papering over it.
In practice I prompt the grading model to do exactly this: “List the facts needed to fully answer this question. For each, mark whether the current context contains it.” Anything unmarked becomes a sub-query. That’s the loop closing — a detected gap turns directly into the next retrieval.
Move three: check the answer against its sources
The last move happens after generation. You have an answer; now verify that every claim in it is actually supported by the retrieved context. This is grounding verification, and it’s the one I refuse to ship financial or contractual systems without.
The check is mechanical: take the generated answer, take the source chunks, and ask the model claim by claim — is this statement entailed by the sources, or did the model add it? Unsupported claims get flagged. Depending on the stakes, you either strip them, regenerate with a stricter instruction, or surface the answer with the unsupported parts marked so a human knows where to look.
Here’s the discipline that makes this real rather than theatrical: the verifier must see only the retrieved sources, not the model’s own reasoning. If you let it grade the answer against the same context the generator already used in the same breath, it rubber-stamps. Separate the call. Give it the sources and the claim and nothing else. A verifier that shares the generator’s blind spots verifies nothing.
What it costs and where to put it
None of this is free, and I won’t pretend otherwise. Every self-correction step is at least one more model call. A full loop — grade retrieval, detect gap, retrieve again, generate, verify grounding — can turn a single 700ms call into a 4-to-8 second sequence costing several times the baseline. Autonomy is a cost, and self-correction is autonomy spending your latency and token budget on caution.
So you place it by stakes, not uniformly:
| Self-correction step | When it earns its cost |
|---|---|
| Retrieval grading | Almost always — cheap, catches the loudest failures |
| Gap detection | Multi-part questions, decomposed queries |
| Grounding verification | High-stakes domains: money, legal, medical, federal |
For a casual support bot, retrieval grading alone is probably enough; full grounding verification on every answer would just burn budget proving what’s already fine. For a system that tells a contractor whether they qualify for a federal solicitation — which is exactly the kind of high-stakes retrieval I’ve shipped — I want all three, because a confident wrong answer there has consequences a few extra cents and seconds are trivial against.
The honest limit
Self-correction is powerful and it is not magic. A grader is an LLM, and LLMs are fallible graders — a too-lenient one waves thin context through, a too-strict one loops forever rejecting context that was fine. You tune that boundary by reading real transcripts, not by guessing. And no self-correction step can recover a fact that was never retrievable in the first place. If your chunking destroyed the answer at index time, the grader will correctly report that nothing relevant came back, and the loop will spin looking for something that no longer exists in your store. Self-correction tells you the answer isn’t there. It can’t put it back.
That’s the right frame to end on. The defining move of agentic RAG isn’t that it answers — regular RAG answers fine. It’s that it knows when it shouldn’t have. A system that can look at its own retrieved context and say “this isn’t enough, I’m going back” is doing the one thing a fluent language model will otherwise never do on its own: doubt itself before it speaks. Build that doubt in deliberately, place it where the stakes justify it, and you turn a confident liar into a system you can actually put in front of a decision that matters.