Our cost dashboard had been creeping up for months and the usual suspects — model choice, output length, request volume — had all already been optimized about as far as they'd go without hurting quality. The actual fix, once we found it, was almost embarrassing: we had a system prompt with a large block of static context — tool definitions, a long set of formatting instructions, some reference examples — that was identical across nearly every call, and we were paying full input-token price for it every single time.

What prompt caching actually buys you

The mechanics are simple once you see them: mark the static portion of your prompt as cacheable, and on subsequent calls within the cache window, you pay a fraction of the normal input cost for that portion instead of the full rate. The catch, and the reason it took us a while to notice the opportunity, is that it only helps if the cached portion is genuinely static and appears at the start of the prompt, before anything that changes per request. We had our static instructions after a per-request timestamp, which invalidated the cache on every single call without us realizing it. Moving the timestamp to the end of the prompt was a two-line change that unlocked most of the savings.

before: [timestamp][system instructions][tool defs][user query]
after:  [system instructions][tool defs][timestamp][user query]

That reordering alone got our cache hit rate from effectively zero to over 90%, because now the identical prefix — instructions and tool definitions — is what the cache actually sees on every call.

Where this compounds

The bigger the static portion of your prompt relative to the dynamic portion, the more this matters. For us, with a heavy tool-definition block and detailed formatting instructions, the static prefix was close to 4,000 tokens against a typical dynamic query of a few hundred. That ratio is exactly the situation where caching goes from "nice optimization" to "the majority of your bill." If your prompts are mostly dynamic content with little repeated structure, don't expect the same payoff — this isn't a universal 60% discount, it's a discount proportional to how much of your prompt is actually reused verbatim.

What I'd check first if your bill looks too high

  • Is there a static block — tool defs, instructions, few-shot examples — that's identical or near-identical across calls?
  • Is anything dynamic sitting before that static block in the prompt, silently invalidating the cache?
  • What's your actual cache hit rate — most providers expose this, and if you haven't checked it, it's worth five minutes to look.

This was the single highest-leverage change we made to our LLM costs all quarter, and it required no model changes, no quality tradeoff, and about an hour of actual engineering work once we understood what was happening.