The Prompt-Caching Discount Most Agents Leave on the Table

Watch a single agent turn and you will notice something wasteful: your agent sends the same 4,000-token system prompt, the same tool schemas, and the same accumulated context on call after call, and pays full input price for all of it every time. Anthropic and OpenAI already offer a large discount on exactly those repeated prefix tokens through prompt caching, somewhere in the range of 30-60% off the cached portion. The catch is that you have to mark the stable prefix with cache_control and keep that prefix identical across calls. When I read through how self-built agents actually assemble their requests, the caching is usually off or accidentally broken. The tokens are paid for twice, ten times, a hundred times.

The tokens you pay for on every single call

An agent request is not one flat prompt. It is a stack: a system prompt with instructions and guardrails, a block of tool or function schemas, sometimes a few-shot preamble or a retrieved-context section, and finally the live user turn. Everything above that last turn is stable. It does not change between call two and call fifty of the same session, and often not between sessions.

That stable stack is the bulk of your input tokens. If your system prompt plus tool schemas run 5,000 tokens and the user turn adds 200, then 96% of every request’s input is a repeat of the previous request. You are billed for all of it at full rate unless caching is on. This is the same structural waste I walked through in where your agent bill actually goes: the line item that quietly dominates is not the clever reasoning, it is the boilerplate you resend.

Why this is the first lever to pull

Most cost-optimization advice carries a quality risk. Swap to a cheaper model and you gamble on accuracy. Compress a prompt and you might drop the sentence that was holding a behavior together. Prompt caching has neither problem, which is why I reach for it before anything else.

It is also the one lever where the savings math is not a guess. You know exactly how many tokens sit in your stable prefix. You know the published cache-read discount for your provider. Multiply those by your call volume and you have the estimate, grounded in counts you can verify rather than a hopeful projection. For a deeper treatment of why deterministic levers beat speculative ones, see agent token economics.

How the discount actually works

Caching operates on a stable prefix, not on arbitrary chunks anywhere in the prompt. The provider hashes your prompt from the start up to the cache breakpoint. On a repeat request, if that prefix matches a live cache entry, you pay the reduced read rate for those tokens instead of the full input rate.

Three mechanics decide whether you actually get the discount:

The prefix must be identical. One changed byte before the breakpoint (a timestamp injected into the system prompt, a reordered tool list, a per-request session id near the top) invalidates the cache. The next call is a full-price miss.

Order matters. Stable content has to come first and volatile content last. If you interleave the live user turn between two static blocks, everything after that turn is uncacheable even if it never changes.

The cache has a TTL. Anthropic’s default cache window is a few minutes and refreshes on each hit; OpenAI’s automatic caching works on prompts at or above 1,024 tokens. If your calls are spaced further apart than the window, the entry expires and you rewrite it.

Here is the ordering mistake I see most, and the fix:

# Before: volatile content first breaks the cache on every call
messages = [
    {"role": "system", "content": f"Session started at {timestamp}. {SYSTEM_PROMPT}"},
    {"role": "user", "content": user_turn},
    {"role": "system", "content": TOOL_SCHEMAS},  # too late to cache
]

# After: stable prefix first, cache_control on the last stable block,
# volatile user turn last
messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {
        "role": "system",
        "content": TOOL_SCHEMAS,
        "cache_control": {"type": "ephemeral"},  # breakpoint here
    },
    {"role": "user", "content": user_turn},
]

The timestamp in the first version mutates the prefix on every request, so nothing ever caches. Move volatile values out of the prefix, put the breakpoint after the last stable block, and the same tokens now bill at the read rate.

Measure first, then place the breakpoint

TokenJam ships two analyzers for this, and they answer two different questions.

tj optimize cache measures your current cache usage. It reads the traces you have already recorded and reports how much of your prefix is being cached today and your effective hit rate. It also shows what you are saving versus leaving on the table. This is the “where do I stand” pass.

tj optimize cache-recommend recommends where to place cache_control. It hashes the stable prompt prefixes across your recorded calls, finds the longest run that stays identical, and points to the breakpoint that would capture the most repeated tokens. This is the “what do I change” pass.

The two shipped cache analyzers

Feature	tj optimize cache	tj optimize cache-recommend
Question it answers	How much am I caching now?	Where should the breakpoint go?
Input	Recorded traces	Recorded traces
Output	Current hit rate and savings estimate	Suggested cache_control placement
When to run	First, to baseline	After, to act on the gap

Getting to them takes three commands:

pipx install tokenjam        # or: pip install tokenjam
tj onboard                   # wire up trace capture
tj optimize cache            # baseline your current cache usage
tj optimize cache-recommend  # find where to place cache_control

Both are positional subcommands. There are no --finding or --include-bloat flags to remember. If you already run TokenJam under an MCP server, the read-only get_optimize_report surfaces the same findings to your agent; the placement work still happens through the CLI.

Because this is provider-native, it composes with whatever routing you already have. If you front your providers with a gateway (see what is an LLM gateway), caching lives at the provider layer beneath it and keeps working regardless of how the gateway splits traffic.

Common questions

I added cache_control and my bill didn't move. What did I do wrong?

Almost always the prefix is not actually stable. Something before your breakpoint changes on every call: a timestamp in the system prompt, a per-request id, a tool list that serializes in non-deterministic order, or retrieved context that is spliced in ahead of the static block. The provider hashes from the start of the prompt to the breakpoint, so one mutated byte upstream forces a full-price miss. Run 'tj optimize cache' to see your real hit rate. If it is near zero, hunt for what is mutating the prefix rather than adding more cache_control markers.

Is the caching savings number just a guess?

No. It is arithmetic on values you already have, not a projection of future behavior. You know how many tokens sit in your stable prefix, you know your call volume from the traces, and the cache-read discount is a published rate. Multiply them. That is why caching is the lever I trust first: unlike a model swap, where the accuracy impact is unknown until you test, the caching estimate is grounded in counts you can verify.

Do I have to switch models or providers to use this?

No. That is the whole point. Prompt caching is a billing feature that Anthropic and OpenAI already offer on the models you are running today. The model sees byte-identical input and returns byte-identical output. Nothing about quality changes. You are only opting into a discount on the repeated prefix.

Where do I put cache_control if my tools change per request?

Split the stack. Put the tools that are constant for the session into the cached prefix and keep the per-request ones after the breakpoint. If the tool set genuinely changes on every call, that portion is not cacheable and you should not try to force it. 'tj optimize cache-recommend' finds the longest run that actually stays identical across your recorded calls, so it will point you at the real stable boundary rather than an aspirational one.

Is it worth it if my calls are minutes apart?

Maybe not. The cache entry has a TTL. Anthropic's default window is a few minutes and refreshes on each hit, so a busy session keeps the prefix warm cheaply. If your agent fires one call every ten minutes, the entry may expire between calls and you pay to rewrite it each time. Baseline your actual cadence with 'tj optimize cache' before committing; caching rewards frequent reuse inside the window.

Does TokenJam place the cache_control markers for me?

No. The analyzers are read-only. 'tj optimize cache' measures where you stand and 'tj optimize cache-recommend' tells you where the breakpoint should go, but you make the edit in your own request-building code and keep control of it. TokenJam observes and recommends; the placement is yours to apply and review.