Three months after launch, cost per request was up by a third. Nobody had changed the model, the traffic mix, or the price.
What changed was the prompt. Engineering’s back-of-envelope had priced the system prompt at the cached rate on every call, which is how most launch estimates treat prompt caching: as a discount you switch on. Then someone added a line with today’s date to the top of the system prompt, and every request after that paid to write the cache and never got to read it.
The short answer: prompt caching saves in proportion to the cache hit rate, and the hit rate comes from the product shape, not the model. A cached read costs about a tenth of a normal input token. A chat assistant with a stable system prompt serves 70 to 85 percent of its input from cache; a summarizer serves almost none.
That gap is why I don’t trust a caching estimate until I know what kind of product it’s for.
Prompt caching pricing: the three rates that matter
Every vendor’s cache comes down to three prices, all quoted against the normal input rate. As of October 2026, from each vendor’s own documentation:
| Vendor | Cached read | Cache write | Default lifetime |
|---|---|---|---|
| Anthropic (Claude) | about 0.1x input, lower on some newer models | 1.25x for 5 minutes, 2x for 1 hour | 5 minutes, refreshed on every hit |
| OpenAI | 0.1x input, lower on some models | 1.25x on newer models, free on older ones | 30 minutes on newer models |
| Amazon Bedrock | the model’s cache-read rate | higher than input on some models | 5 minutes on most, reset on every hit |
| Google Gemini | discounted, applied automatically | none for implicit caching | managed by the vendor |
I plan on the ratios because they’ve held steady while the prices moved. Check the live page before a number goes into a budget.
Those three numbers give a break-even rule you can apply without a spreadsheet. At a 1.25x write and a 0.1x read, a cached prefix pays back the first time it’s read once. At a 2x write, the one-hour option, it needs two reads inside the hour. A prefix that’s written and never read again costs 25 to 100 percent more than not caching at all.
Cache hit rate by product shape
Cache hit rate here means the share of a feature’s input tokens served from cache over a month. It’s the one number that turns the vendor’s discount into a cost per request. The ranges below come from production AI products at a Fortune-500 financial data company running eight figures of annual AI and cloud spend, rounded and abstracted to shape.
Chat assistant (conversational AI)
What caches: the system prompt, tool definitions, and history up to the latest turn. Users usually reply inside the five-minute window, so each turn reads what the previous one wrote. Hit rates landed at 70 to 85 percent, and caching cut cost per request by 20 to 40 percent.
Caveat: a user who walks away for ten minutes comes back to a cold cache and pays the write again.
Copilot (in-app assistant)
What caches: the system prompt and tool definitions. The document or record the user is working on changes between calls, so it sits after the cache breakpoint. Hit rates of 40 to 60 percent were typical.
Caveat: copilots tend to inject live context near the top of the prompt, which breaks the prefix for everything after it.
Summarizer (document summarization)
What caches: only the instruction block, which is usually short. Each document is read once. Hit rates under 10 percent, and in one case caching made the feature more expensive, because the breakpoint sat after the document and every request paid the write premium on content nobody read twice.
Caveat: the exception is a summarizer asked several questions about the same document, which behaves like a chat assistant.
RAG search (retrieval-augmented generation)
What caches: the system prompt and instructions. The retrieved chunks change with every query, so most of the input never caches. Hit rates of 10 to 20 percent. Context trimming did more for cost per query than caching did.
Caveat: a knowledge base small enough to put whole into the prompt flips this, and the hit rate jumps.
Classifier (routing and tagging)
What caches: the instructions and few-shot examples, if together they clear the vendor’s minimum length. Anthropic’s minimum runs from 512 to 4,096 tokens depending on the model, and OpenAI’s starts at 1,024. Below the line, nothing caches and nothing errors. Above it, hit rates ran past 80 percent.
Caveat: the minimum differs between models, so a model swap can switch caching off silently. It is one more reason the cheapest AI API is rarely the cheapest AI product.
Agent (multi-step tool use)
What caches: tool definitions, instructions, and the growing run history, across the many calls one task makes within a few minutes. Hit rates of 60 to 80 percent within a run. The AI agent cost breakdown covers why that matters: an agent’s input grows with every step.
Caveat: sub-agents fired in parallel all miss. Anthropic’s documentation is explicit that a cache entry exists only once the first response has started.
| Product shape | Typical hit rate | Effect on cost per request |
|---|---|---|
| Chat assistant | 70 to 85% | down 20 to 40% |
| Agent | 60 to 80% within a run | down 15 to 30% |
| Classifier (above minimum) | over 80% | down 30 to 50% |
| Copilot | 40 to 60% | down 10 to 20% |
| RAG search | 10 to 20% | down under 5% |
| Summarizer | under 10% | flat, or up |
The levers that move the hit rate
Seven things decided the hit rate on the products I’ve worked on. None of them is the model choice.
Static content first (prompt order)
Every vendor matches on an exact prefix. Tools, then system instructions, then history, then the new input. Anything that changes per request belongs at the end.
Timestamps and IDs in the system prompt
The most common way I’ve seen a hit rate fall to zero. A date, a request ID, or a user’s name near the top changes the prefix on every call.
Tool definition changes
Anthropic and Bedrock both process the prompt as tools, then system, then messages, so editing one tool invalidates everything after it. OpenAI’s advice is to keep tool lists stable and restrict use per request instead of removing definitions.
Minimum cacheable length
A short prompt never caches, with no warning. Check cached_tokens or its equivalent in the usage response, not the config.
Cache lifetime (5-minute vs 1-hour TTL)
The longer window costs more to write. It pays when users come back between five and sixty minutes later, which is common for chat and rare for batch-like traffic.
Parallel fan-out
Ten requests sent at once each write their own copy. Sending one first and waiting for it to start lets the other nine read it.
History rewrites (summarization and compaction)
Appending turns keeps the prefix intact. Summarizing or trimming older turns rewrites it and starts the cache over.
Pre-build vs in production: which hit rate to plan on
Before launch, I plan on the uncached input price and treat caching as upside. The LLM cost calculator takes a hit rate as an input for that reason: it’s an assumption until it’s measured. After a month in production with a measured rate above 50 percent, re-price on the blended rate, and carry that into the monthly run rate by product shape. The sibling piece on what one AI request actually costs covers how cached reads fit with the other eight drivers.
When to switch caching on, and when to leave it off
Turn it on when most requests reuse a prefix over the vendor’s minimum within its lifetime: chat, agents, and classifiers with long instructions. Leave the breakpoint off content read once, like a summarizer’s document or a retrieved chunk. If you can’t say how many times a prefix gets read before it expires, measure that before paying the write premium.
The mistake: treating a prompt edit as a copy change
Where I’ve watched caching savings disappear, nobody changed vendors or models. Someone edited the prompt: a date at the top, a reordered tool list, a new instruction above the cached block. The feature kept working, so nobody looked, and cost per request climbed for a quarter.
The symptom is a cache-write line growing faster than the cache-read line in the usage export. When the two cross, the cache is costing money.
The rule
Price the cache from the hit rate you’ve measured, and re-check it every time the prompt changes. Cache hit rate is one of the few FinOps KPIs that held up on AI products, and it’s the one most likely to move without anyone deciding to move it.
FAQs
What is prompt caching?
Prompt caching stores the processed start of a prompt so the next request that begins with the same text can reuse it. The reused tokens bill at about a tenth of the normal input rate at the major vendors. It only helps when the same prefix repeats within the cache’s lifetime, usually minutes.
How does prompt caching work?
The vendor matches the start of each new prompt against prefixes it has recently processed. On a match, those tokens are read from cache at the discounted rate; the rest bill normally. Storing a new prefix can cost more than a normal read, so the first request pays a premium that later requests earn back.
How much does Anthropic prompt caching cost?
Per Anthropic’s documentation as of October 2026, a cache read costs about 0.1x the base input price, lower on some newer models. A five-minute cache write costs 1.25x and a one-hour write costs 2x. Every hit refreshes the five-minute window at no extra charge, which is why busy chat products stay warm.
How does Bedrock prompt caching differ from calling the vendor directly?
Amazon Bedrock supports caching for Claude, Nova, and OpenAI models through its own APIs, with per-model minimums and lifetimes listed in its documentation. It isn’t available with Bedrock’s batch inference API, and cross-region routing can raise the number of cache writes. Check the model card for the minimum before you count on savings.
What is the difference between prompt caching and KV cache?
The KV cache is the model’s internal memory of already-processed tokens inside one generation. Prompt caching keeps that work around across separate API requests and bills less for reusing it, which makes it a pricing line you can plan a budget around.
Is semantic caching the same as prompt caching?
No. Semantic caching returns a stored answer when a new question means roughly the same thing, skipping the model call entirely. Prompt caching still calls the model and discounts only the repeated prefix. Semantic caching saves more per hit but risks a wrong answer, so it fits FAQ-style traffic better than open-ended chat.
What to do next
Pull a month of usage for one feature, split input into uncached, cache-write, and cache-read tokens, and compute the hit rate before you plan any savings. If that number needs to turn into cost per subscriber and a forecast your finance team runs on its own, that’s the work behind FinOps consulting for AI products. The rest of my FinOps for AI work is on the home page.
I put a number on what an AI product costs per request, per active user, and per subscriber, early enough to change the model, the architecture, or the price. If you’re shipping something with a model behind it and nobody can tell you what it costs, email me.
Have a number nobody can explain?
Send a note and I will get back to you.