Ask three people what one AI request costs and you get three numbers. None of them is on the cloud bill.
The bill has a total with no unit behind it. The vendor’s pricing page times a token estimate gets closer, and it is what most launch estimates run on, but it prices two of the nine things a request does and is wrong by the end of the quarter, because the token mix moves and the estimate does not. LLM cost only becomes a number you can plan on when it is built from the drivers, not the invoice.
The direct version: one LLM request costs its input tokens, output tokens, and cached reads at the model’s rate, plus the retries, tool calls, retrieval, and eval calls that ride along with it. On the production AI products I have measured, those ride-alongs added 30 to 80 percent on top of the tokens-times-price estimate.
LLM cost per request: the nine drivers
Same profile for each: what it is, what moves it, where it landed, and the caveat. The ranges come from a Fortune-500 financial data company running eight figures of annual AI and cloud spend, abstracted to shape.
Input tokens (prompt tokens)
Everything the model reads: system prompt, history, retrieved documents, the user’s message. Driven by design more than by users; a prompt that grows a paragraph a sprint doubles input with no change in traffic. On chat products, 15 to 25 percent of cost per request; on summarizers, 55 to 70 percent.
Caveat: the prompt you tested with is shorter than the prompt you ship.
Output tokens (completion tokens)
Everything the model writes back. Per Anthropic’s pricing page as of September 2026, output runs three to five times the input rate, and the same shape holds at other vendors. On chat assistants this was the largest driver, 45 to 60 percent of cost per request, and output length crept up every quarter nobody capped it.
Caveat: a prompt edit changes output length, so a prompt edit is a pricing event.
Cached tokens (prompt cache reads)
Input the model has already seen and reads from cache. Per Anthropic’s pricing page as of September 2026, a cache read is priced at roughly a tenth of a standard input token; the discount varies by vendor, the order of magnitude does not. Caching cut cost per request 20 to 40 percent on chat products with long system prompts and almost nothing on summarizers.
Caveat: a cache write costs more than an uncached read, so caching content read once loses money.
Context length (the system prompt tax)
The fixed tokens every request carries before the user types: the part of input the product team controls completely. On one assistant, the system prompt and tool definitions were more than half of all input tokens for the month.
Caveat: the cheapest lever on this list, and the one skipped because it looks like a chore instead of a cost decision.
Retries and re-prompts
Second and third calls made because the first failed a check: malformed JSON, low confidence, a timeout. Each is a full request at full price. On agents and structured-output features, 10 to 20 percent of cost per request; on plain chat, 5 to 10. The rate climbs when a team moves down a model tier, which is how the cheaper model costs more per resolved task.
Caveat: retries are never in the pre-build estimate, so they arrive as unexplained variance at the first invoice.
Tool calls (function calling)
The round trips an agent makes to look something up or run a query, each with its own tokens. One user request can fan out into ten model calls. On agent products, tool calls plus retrieval ran 15 to 30 percent of cost per request, with a long tail: a small share of requests made most of the calls.
Caveat: for agents, cost per request is a distribution, not a number. Report the p90 with the mean.
Retrieval: embeddings and vector search
The embedding call, the vector search, and the reranking pass. Cheap per call, expensive when documents get re-embedded on every index refresh. On RAG search products, under 5 percent of cost per request but a large share of the fixed monthly infrastructure.
Caveat: retrieval usually sits in a different account from the model, which is how it goes missing from unit economics.
Eval overhead (LLM-as-judge)
A second model grading the first model’s answer. Sampled at 5 to 10 percent of traffic, eval overhead stayed at a few percent of cost per request; run on every request, it nearly doubled it.
Caveat: eval spend is charged to the platform team, so it never lands on the product’s unit cost unless someone allocates it.
Provisioned throughput share (PTU)
The slice of a prepaid capacity pool a request consumes. At low utilization, cost per request on a provisioned pool ran two to three times the pay-per-token equivalent; at high utilization, below it. The sibling piece on what changes when the workload is inference covers the allocator that turns the pool into a per-application number.
Caveat: a request on a provisioned pool has no marginal cost until the pool fills, so the unit cost looks like zero until it looks like a resize.
AI token costs by product shape
Share of cost per request by product shape, from production, rounded to the range:
| Driver | Chat assistant | Summarizer | Agent or copilot |
|---|---|---|---|
| Input tokens (uncached) | 15 to 25% | 55 to 70% | 20 to 30% |
| Output tokens | 45 to 60% | 20 to 30% | 25 to 35% |
| Cached reads | 5 to 10% | under 5% | 10 to 15% |
| Retries and re-prompts | 5 to 10% | 3 to 8% | 10 to 20% |
| Tool calls and retrieval | under 5% | under 5% | 15 to 30% |
Read the column, not the cell. A summarizer is an input problem, so context trimming is the lever. A chat assistant is an output problem, so response length is. An agent is a call-count problem; budget the p90 request.
From cost per request to cost per subscriber
Cost per request is the base unit. Four metrics sit between it and the number finance is asking for.
Cost per request (cost per query, cost per call)
Model, retrieval, and eval spend attributed to a feature, divided by the requests it served, measured from the vendor’s usage export rather than inferred from compute. Per feature, because a shared endpoint has one bill and five unit costs; features on the same model differed by ten to one.
Caveat: an all-up cost per request across a product hides the feature losing money.
Tokens per session
Input plus output across every request in one session. Products that replay full history saw tokens per session grow quadratically with turn count, the mechanism behind “long conversations are unprofitable.”
Caveat: the median session is short and the p95 session is where the cost lives.
Cost per active user (cost per MAU)
Cost per request times requests per active user per month. Requests per active user is the input launch estimates skip and the one that moved most: it doubled within two quarters of shipping, so the launch estimate was half the real number by the time the pricing page was public.
Caveat: “active” needs one definition finance and product share, or two teams report two numbers from the same data.
Cost per subscriber (cost per seat)
Cost per active user times the share of subscribers who are active. The number that belongs next to the seat price, because it is in the same unit as revenue. A costly feature at 20 percent activation can be cheaper per seat than a modest feature everyone uses.
Caveat: activation rises after launch, so cost per subscriber drifts upward.
Cache hit rate
The share of input tokens served from cache. Under 30 percent, the cached price is a rounding error. Above 60 percent, pricing on uncached rates overstates cost per request by a third or more.
Caveat: the hit rate in a test harness with one user is not the hit rate with a thousand; entries expire between requests.
Cached versus uncached: which price to plan on
Plan on the uncached price until the feature has run a month in production with a measured hit rate above 50 percent, then re-price on the blended rate. Pricing at the cached rate before launch assumes a discount the product may never earn.
Pre-build versus in production: where the number comes from
Before the feature exists, the number comes from a model: token split, requests per user, an assumed cache hit rate and retry rate. That is what the LLM cost calculator computes, and it is good for choosing a model and an architecture. After launch, the number comes from the usage export joined to product analytics, the only version fit for a forecast. Swapping one for the other is how forecasts miss.
Per request versus per user: which one to send to finance
Send cost per active user and cost per subscriber. Cost per request explains why the number moved; finance needs a number in the same unit as revenue. The FinOps Foundation’s framework calls this capability unit economics, and inference stresses it hardest.
The mistake: reporting the API bill as the unit cost
Take the vendor invoice, divide by requests, call it cost per request. It is too low, because retrieval, evals, and provisioned capacity sit on other bills, and wrong per feature, because it averages the cheap features with the expensive one.
The symptom is a stable-looking cost per request while margin on one feature quietly goes negative. The first feature-level breakdown at a Fortune-500 financial data company showed one of the top three AI products running several times the portfolio average; the invoice number had hidden it for two quarters.
The rule
Price a request from its drivers, report it per subscriber, and re-measure it every time the prompt changes. The re-measure clause is there because a prompt edit moves output length, cache hit rate, and retry rate at once, which makes it the most common pricing event nobody logs as one.
FAQ
How much does ChatGPT cost per query?
For a consumer subscription there is no per-query price; the vendor carries the inference cost behind a flat fee. For a product built on the API, ChatGPT or OpenAI cost per request is input, output, and cached tokens at the published rate, plus retries. Output costs several times input, so the token split decides most of the number before the model choice does.
What is LLM inference cost?
The spend on running a trained model to answer requests, as opposed to training it. It is driven by token volume and by how capacity is bought, pay-per-token or provisioned. In production it moves with prompt length, output length, and cache hit rate rather than infrastructure, which is why cloud cost tools built for compute report it poorly.
Why is LLM cost per request rising when traffic is flat?
A prompt template that grew, output length that crept up, a cache hit rate that fell, or a retry rate that rose after a model change. None show on an infrastructure dashboard. Check tokens per session by feature before the cloud side.
What to do next
Take one feature, pull a month of usage from the vendor export, and build cost per request from the nine drivers instead of the invoice. When that number needs to become cost per subscriber and a forecast finance runs unassisted, that is the work on FinOps consulting for AI products. The rest of the FinOps for AI practice is on the home page.
I put a number on what an AI product costs per request, per active user, and per subscriber, early enough to change the model, the architecture, or the price. If you’re shipping something with a model behind it and nobody can tell you what it costs, email me.