Part VI · Chapter 10 · Models in Production · lesson 2
What one model call costs, and what a thousand a minute costs
14 min
Builds on What one request costs you, in cents
Part VI · Chapter 10 · Models in Production · lesson 2
14 min
Builds on What one request costs you, in cents
The feature works in staging. Before it ships, somebody wants a monthly figure. You have two per-million rates from the provider, a prompt you can count, and an answer length nobody has measured. Three multiplications stand between you and the number — and the one people get wrong is not the arithmetic, it is which of the two token counts is driving the total.
Predict
Inference cost and latency
Hypothetical model, not a forecast
Cost and wall-clock time for one generation request, split into the part driven by the prompt and the part driven by the answer.
It assumes
It ignores
This feature spends most of its money on the prompt. Change the token counts until the answer is the majority of the bill instead, and notice how far you had to move them.
Everything sent in: instructions, retrieved text and conversation history.
The length of the answer, counted in tokens rather than words.
Not there yet — keep moving the controls.
Read the two shares in the summary against each other. The answer is a ninth of the tokens and well over a third of the bill, which is the whole argument for why the rate asymmetry matters — and it still loses, because a third is not a half. Both of those statements are true at once, and people pick whichever one supports the thing they already wanted to do.
Now change one thing. Add retrieval: eight chunks of a knowledge base, pushing the prompt from four thousand tokens to twelve thousand, with the same answer. The cost per call goes from roughly six and a half tenths of a cent to one and a half cents — a bill more than doubled — and the answer's share of it falls from thirty-eight per cent to seventeen. The feature did not get better at writing. It got more expensive at reading, and the term you were watching is now the term that does not matter.
The request count input stops at ten million a month. A thousand calls a minute is forty-three million a month, so this model cannot be asked that question directly — and if you type it in anyway, the input clamps silently and hands you a confident figure for a quarter of your traffic. That behaviour is a property of every input in every model here, and the habit worth forming is to check an input's range before quoting a number built on it.
So do it by hand. Cost per call times calls per month, with the per-call figure from the model above: a thousand a minute at the four-thousand-token prompt is about two hundred and eighty thousand a month, and the same traffic once retrieval is added is about six hundred and twenty-six thousand. That second number is a hiring decision, which is why finance asked before the feature shipped rather than after.
Two things follow that are not obvious from the per-call figure.
The monthly total is linear in traffic and linear in both token counts, so the cheapest lever is whichever one you can halve. Halving the answer length and halving the request count save the same money. One of those is a product decision about verbosity and the other is a decision not to ship.
The prompt term grows without anybody editing it. Conversation history, more retrieved chunks, a longer system instruction, few-shot examples somebody added to fix a quality problem — all of those raise the term you pay on every call, and none of them look like a cost change when they are made. A monthly bill that rises with no traffic change is almost always this.
Count the tokens, do not estimate them. Every provider returns the counts in the response. Log both, per call, with a feature label on them, and you have a cost figure that is a measurement rather than an argument. Estimating from character counts is wrong by enough to flip which term dominates, especially outside English.
Put a cap on the answer length and mean it. The maximum-tokens parameter is the only hard bound you have on the expensive term, and it is also the one that protects you from a loop that will not stop. Set it from the longest acceptable answer in your eval set rather than from the provider's maximum.
Price the feature per unit of the thing the business counts. Cost per support reply, per summary, per candidate screened — not cost per thousand tokens. The token figure cannot be compared to revenue and the per-reply figure can, and the per-reply figure is what makes someone say the feature is too expensive at this margin rather than merely large-sounding.
Cache the prefix if the prompt repeats. This is the one lever that changes the shape of the arithmetic rather than its inputs, and it is the subject of the next section.
The rate asymmetry is not a pricing strategy bolted onto a symmetric cost. It reflects a genuine asymmetry in the work.
Prompt tokens are processed together. The whole prompt can be pushed through in one pass, with the arithmetic for many tokens happening at once, which keeps the expensive hardware busy doing useful work.
Generated tokens cannot be. Each one depends on the one before it, so they come out strictly one at a time, and each step reads the entire accumulated state to produce one token. The hardware spends most of that step moving memory rather than computing, which is exactly the roofline problem from the performance track wearing different clothes.
So one prompt token is cheap because it shares a pass with thousands of its neighbours, and one generated token costs more because it gets a pass of its own. The next lesson is about what that does to the clock rather than the bill, and it is the same mechanism measured on a different axis.
Check
It stops the moment the prompt is cached between calls.
A cached prefix is billed differently — typically much less than a fresh prompt token, sometimes with a separate charge for writing the cache — and that breaks the single assumption this whole model rests on, which is that every prompt token costs the same on every call. Once a long system instruction and a stable set of retrieved documents are being read from a cache, the prompt term stops scaling with its length and starts scaling with how much of it changed. The dominance question flips back towards the answer, and it flips without anybody touching the token counts.
The practical consequence is a design one rather than an accounting one. If the prompt is cached, prompt order becomes load-bearing: everything stable goes first, and the one line that varies per request goes last, because a change early in the prompt invalidates everything after it. Teams discover this by putting the user's question at the top, where it is easy to read, and getting none of the discount.
Recompute the per-call figure the first time you turn caching on, and recompute it again if the cache's own lifetime or pricing changes. The shape of the arithmetic — two counts, two rates, a product each — survives all of that. The figures do not.