The question I get asked least often, and the one that decides whether a RAG system survives contact with production, is: how do you know it’s working? Not in the demo. In the wild, across thousands of real questions, when nobody is watching and the model just told a contractor they qualify for a solicitation they don’t. How do you measure that before it costs you?
Most teams reach for a number — “our RAG is 87% accurate” — and that number is almost always a lie, because it was computed on a tiny golden set of questions someone wrote while building the thing, questions that look nothing like what users actually ask. Worse, the moment you add an agentic loop, the standard metrics stop describing the system at all. You can’t put one accuracy number on a thing that decides its own control flow. Let me lay out how I actually measure these systems, regular and agentic, and why the second is so much harder.
Separate the two failures
The first discipline is refusing to measure RAG end-to-end as a single number. A RAG system can fail in two completely independent places: retrieval brought the wrong context, or generation mishandled the right context. Collapse them and you can’t fix anything, because a single accuracy score doesn’t tell you which half to work on.
So I measure them apart.
Retrieval quality asks: did the right information make it into the context, regardless of what the model did with it?
- Recall@k — of the chunks that actually contain the answer, how many showed up in the top k? This is the one that matters most and the one people skip because it requires knowing which chunks are the right ones. It’s worth the labor. Low recall means the answer never even reached the model. No prompt fixes that.
- Precision@k — of the k chunks retrieved, how many are relevant? Low precision floods the context with noise and the model drowns the answer in it.
- MRR — how high up did the first relevant chunk land? Position matters; models weight early context more.
Generation quality asks: given the context it got, did the model produce a faithful, complete answer?
- Faithfulness / groundedness — is every claim in the answer supported by the retrieved context, or did the model invent some of it? This is the hallucination metric and it’s the one I never ship a high-stakes system without.
- Answer relevance — did it actually address the question, or did it produce a faithful, well-grounded answer to a slightly different question?
- Completeness — for multi-part questions, did it cover all parts or quietly drop one?
Faithfulness without recall is a system that honestly reports it doesn’t know. Recall without faithfulness is a system that has the answer and lies anyway. You need both, and one number hides which one you’re missing.
The payoff of separating them is diagnostic. High retrieval scores and low generation scores means fix the prompt or the model. Low retrieval scores means fix chunking, embeddings, or search — and no amount of generation tuning will help, because the answer was never in the context to begin with. I’ve watched a quarter of engineering time get spent tuning prompts on a system whose real disease was 40% recall. Separated metrics would have caught that in an afternoon.
Why agentic RAG breaks these metrics
Now add the loop, and watch the metrics fall apart.
Recall@k assumes there’s a retrieval to measure. Agentic RAG retrieves three, five, eight times, each query generated from the last one’s results. Which retrieval do you compute recall on? The first one might be intentionally broad, the third one sharp. A low recall@k on hop one isn’t a failure — it might be the planner correctly casting a wide net before narrowing. The metric that was a clean signal for one-shot RAG is now ambiguous.
Faithfulness gets harder too. The final answer might be grounded in context the system gathered across multiple hops, including intermediate facts it extracted and carried forward. Verifying the answer against “the retrieved context” now means verifying against an accumulated, multi-source pile, and a bridge fact extracted wrong in hop one can make a perfectly faithful-looking final answer completely wrong. The grounding check passes; the answer is still false, because the error was upstream in the chain.
And there’s a whole dimension that doesn’t exist for regular RAG at all: was the loop itself correct? Did the planner decompose the question sensibly? Did it stop at the right time, or loop three extra times burning budget, or stop early on thin context? Did each hop’s query follow from the last? You can’t see any of that in an input-output accuracy score. The answer can be right by luck while the reasoning was a mess, or wrong despite a sound plan that hit a chunking wall.
Measure the trajectory, not just the answer
This is the core shift, and it’s the same principle I apply to every autonomous system I build: when a system reasons, you have to instrument the reasoning, not only the result. I log the full trajectory of every agentic RAG run — the plan, each sub-query, what each hop retrieved, what the grader decided, when and why the loop stopped.
run_id: a91f
question: "CO email for the agency that awarded the most cyber contracts in our NAICS last year"
hop_1: query="cyber contracts NAICS 541512 last year by agency" | retrieved=12 | graded_relevant=4 | extracted="Agency X"
hop_2: query="contracting officer Agency X vehicle Y" | retrieved=8 | graded_relevant=2 | extracted="J. Rivera"
hop_3: query="contact record J. Rivera" | retrieved=5 | graded_relevant=1 | extracted="j.rivera@..."
stop_reason: can_answer=true after 3 hops
latency_ms: 6840 | llm_calls: 9 | final_grounded: trueWith trajectories logged, I can ask questions an accuracy number can never answer. Where do chains break — which hop most often extracts the wrong bridge entity? How many hops do real questions take, and is the planner over-decomposing simple ones? When the final answer is wrong, was it a planning failure, a retrieval failure, or a generation failure? The trajectory tells you which, and that’s the difference between fixing the problem and guessing at it.
The operational metrics matter as much as the quality ones here, because the loop spends real money and time:
| Metric | Why it matters for agentic RAG |
|---|---|
| Hops per query (distribution) | over-decomposition shows up as a fat tail |
| LLM calls per query | the cost driver; tracks directly to budget |
| End-to-end latency (p50/p95) | the loop’s variable depth lives in the p95 |
| Loop termination reason | early-stop vs. exhausted vs. answered — a health signal |
| Escalation rate | what fraction took the expensive path vs. one-shot |
That last row is the one I watch most on hybrid systems. If 80% of traffic is supposed to take the cheap one-shot path and your escalation rate is creeping toward 50%, either your relevance grader got too strict or your traffic shifted, and your costs are about to surprise someone.
Build the eval set from production, not imagination
None of these metrics mean anything computed on questions you made up. The golden set you write while building reflects how you think about your data, and users do not think like you. They ask malformed questions, multi-part questions, questions with typos and pronouns and assumptions. Your hand-written set has none of that, so it scores high and tells you nothing.
So I build eval sets from real traffic. Sample actual production questions, including the ones that failed. Label them — what was the correct answer, which chunks were the right context. It’s tedious and it’s the highest-leverage work in the whole project. A few hundred real, labeled questions beat a few thousand synthetic ones every time, because they fail the way production fails. Then, crucially, you re-run that set on every change. A chunking tweak that helps one query class often quietly wrecks another, and only a regression set catches it before users do.
LLM-as-judge is the practical tool for scoring faithfulness and relevance at scale, since you can’t human-label thousands of answers. It works, with one discipline: validate the judge against human labels on a sample first, and keep the judge’s view narrow. A faithfulness judge sees only the answer and the sources, never the question’s expected answer, never the model’s reasoning. A judge that shares the generator’s blind spots rubber-stamps. Give it one job, sources and claim, supported or not.
What measurement is really for
The reason all of this matters comes back to a principle I hold for every autonomous system: you don’t earn the right to add autonomy until you can measure it. An agentic RAG loop is autonomy — the model deciding what to retrieve, when it’s satisfied, when to stop. If you can’t see the trajectory, you’ve shipped a system that makes its own decisions in the dark, and the first time you’ll learn it’s been deciding badly is when a user does.
Regular RAG you can measure with recall, precision, and faithfulness, separated so they’re diagnostic. Agentic RAG you measure the same way plus the trajectory, because the reasoning is now part of the system and an answer-only score can’t see it. Build the eval set from real failures, score it on every change, log every hop, and watch the escalation rate like it’s a cost center, because it is. This trajectory-level instrumentation is exactly what I bake into the retrieval layer of Vontra — if the system is going to answer questions a business actually runs on, I have to be able to see every decision it made to get there.
The demo will always look fine. The demo is not the system. The system is the thousandth real question answered when no one is watching, and the only way to know whether that one was right is to have been measuring all nine hundred ninety-nine before it. Measurement isn’t the boring part you do after building. It’s the thing that tells you whether you built anything at all.