Two agents I built did nearly identical work, and one cost about ten times more than the other to run. Not because of the model. Because of where I put the words. The cheaper one was structured so the model could reuse most of its input across calls. The expensive one re-paid for the same thousands of tokens on every single step. The lever was prompt caching, and the thing I want to convince you of is that caching is not a flag you flip at the end. It is a decision about prompt layout that you have to design for from the start.
Why an agent loop is the perfect caching target
Think about what an agent prompt actually contains on step ten of a loop. There’s a big stable block — the system prompt, the tool definitions, the task framing — that has not changed since step one. And there’s a small volatile tail — the latest tool result, the next decision. The stable block might be 6,000 tokens. The volatile tail might be 300.
Without caching, every step re-processes all 6,300 tokens from scratch. You pay full input price for the 6,000 tokens of stable prefix you already sent nine times. That’s the waste. An agent loop sends the same prefix over and over by its very nature, which makes it close to the ideal case for caching.
With caching, the model stores the processed state of that stable prefix after the first call. Subsequent calls that share the exact prefix read it from cache at a steep discount instead of re-processing it. The volatile tail still pays full price, but it’s small. The big block goes cheap.
Caching turns the most repetitive, wasteful property of an agent loop — re-sending the same prefix every step — into its biggest cost saving.
The one rule: stable stuff first, volatile stuff last
The cache works on prefixes. It matches from the front of the prompt and stops at the first byte that differs from what it saw before. That single mechanism dictates everything about how you lay out the prompt.
Get the order right:
[ system prompt ] <- never changes \
[ tool definitions ] <- never changes | cacheable prefix
[ stable task context ] <- set once per task /
--- cache boundary ---
[ conversation so far ] <- grows each step
[ latest tool result ] <- new every stepNow break it. Put a timestamp, a request id, or a “current time is…” line at the very top of your system prompt. That one volatile token at the front changes the prefix on every call, the cache misses from byte one, and you re-pay for the entire 6,000-token block every step. I have done exactly this and watched costs balloon for a reason that took me an embarrassingly long time to find. A single dynamic value in the wrong place defeats the whole mechanism.
So the discipline is blunt: anything that changes per call goes at the end. Anything stable goes at the front, in a fixed order. Tool definitions are a perfect cache citizen because they’re identical across an entire session. The volatile reality of the loop lives in the tail where it belongs.
What actually moves the bill
The savings are large because of where they land. In an agent loop, input tokens dominate. The model reads a huge accumulated context and writes a small action. Most loops are heavily input-weighted, so cutting the cost of the repeated input slashes the largest part of the bill.
| Setup | Input cost per step (illustrative) | Notes |
|---|---|---|
| No caching | full price on the whole prefix | re-processes everything every step |
| Caching, good layout | cache discount on the stable block | only the volatile tail at full price |
| Caching defeated by a dynamic prefix | full price again | one bad token at the top ruins it |
The shape that matters: the longer your loop runs and the more stable context it carries, the more caching saves, because you amortize that one-time processing of the prefix across more and more steps. Long agent trajectories — exactly the ones I warned are expensive because cost compounds per step — are where caching pays off hardest. The two ideas are the same coin. Long loops cost more per step because of accumulated context; caching is how you stop re-paying for the part of that context that never changed.
Throughput, not just cost
Caching isn’t only cheaper. It’s faster. Reading a cached prefix skips the work of re-processing it, which shaves real latency off each step. For my voice agents that’s the whole game — every hundred milliseconds I trim off a turn is a hundred milliseconds the human isn’t sitting in dead air. Lower per-step latency also means I can run more concurrent agents on the same budget. Cost and throughput improve together here, which is rare enough to be worth designing for deliberately.
How I build for it now
Caching changed a few of my defaults permanently:
- Freeze the prefix. System prompt, tool definitions, and task framing are written once and never touched mid-session. If I’m tempted to inject something dynamic up top, I find another place for it.
- No dynamic values in the stable block. No timestamps, no per-request ids, no “as of now” lines at the front. They go in the tail or into a tool result.
- Stable order for tools. Reordering tool definitions between calls is a silent cache-buster. I serialize them deterministically.
- Design the loop to share prefixes. When I split work across calls, I keep the common context identical so successive calls hit the same cached prefix instead of each carrying a slightly different one.
None of this is exotic. It’s just taking seriously that the prompt has a layout, and the layout has an economic consequence on every call. The same agent, the same model, the same task — laid out one way it’s cheap and fast, laid out another way it quietly costs ten times more and runs slower. I’d rather decide that on purpose. Put the stable things first, keep the volatile things last, and let the loop’s most repetitive habit become the thing that makes it cheap.