Part VI · Chapter 10 · Models in Production · lesson 7
15 min
Builds on Under overload you refuse work, or you lose all of it, Ship a number you measured, and say what would reverse it and What one model call costs, and what a thousand a minute costs
A teammate started an agent on Friday evening to reconcile two data sources. It ran all night and most of Saturday: nine hours, roughly thirty-nine thousand model calls, and an invoice line somebody found on Monday for about four thousand dollars, against a job that was meant to take twenty minutes. Nothing failed. No alert fired. The loop never decided it was finished, and nothing in its own definition said it had to.
Predict
The arithmetic is two multiplications and the only hard part is choosing the inputs honestly.
A loop's step is one model call. Its prompt carries the system instruction, the task, the tools, and everything the loop has done so far — which means it starts small and grows towards whatever limit you allow. Its answer is the model's next move, which stays roughly the same size throughout.
So the honest cost per step is the cost of the largest step: the prompt as big as you will ever let it get. Multiply that by the number of steps you will allow and you have an upper bound that holds whatever the loop decides to do. Build it from the average step instead and you have a forecast, which is a different object and the one that was quietly assumed on Friday evening.
Inference cost and latency
Hypothetical model, not a forecast
Cost and wall-clock time for one generation request, split into the part driven by the prompt and the part driven by the answer.
It assumes
It ignores
Leave the per-step figures alone and find the step budget that holds one whole run under two hundred and fifty dollars. That number is the cap you would put in the code.
Everything sent in: instructions, retrieved text and conversation history.
The length of the answer, counted in tokens rather than words.
Not there yet — keep moving the controls.
Read the labels sceptically. The model calls the last summary line "cost of a month of requests" because that is what it was written for, and here it means "cost of one run of this many steps". Reinterpreting a model's axis is fine as long as you say so out loud; what is not fine is reading the label and forgetting you did.
With a prompt of a hundred thousand tokens and a thousand-token answer at these illustrative rates, one step costs about a tenth of a dollar, of which the prompt is the overwhelming majority — the same dominance the second lesson of this track measured, except that here the prompt grows with every step. Nearly thirty-nine thousand steps brings the run to roughly four thousand and eighty dollars, which is where Monday's invoice line came from.
And then the clock. At these rates one step takes about twenty seconds, so nine hours is about sixteen hundred steps in sequence. Thirty-nine thousand steps in nine hours means roughly two dozen of them were running at once. That is the part people miss when they estimate a loop's cost from how long they expect it to run: concurrency is a multiplier on money and not on time, and a fan-out inside the loop multiplies the bill without lengthening the job.
A budget object, threaded through the whole call tree, that refuses. One object, created per run, holding a remaining token count, a remaining step count and a deadline. Every call decrements it before it goes out, and a call that would take it below zero raises instead. Not a log line, not a warning, not a metric — a refusal, in the code path, that the loop cannot continue past.
Shared, not per-call. This is the mistake worth naming because it is invisible in review: a budget enforced per call, or per sub-agent, multiplies with the fan-out. One parent spawning twelve children each with the parent's budget has thirteen budgets and no ceiling. The budget has to be one mutable thing that every descendant draws from, which also means the fan-out decision becomes a decision about how to divide a fixed amount.
Tokens as the primary unit, steps and seconds as the secondary ones. A step cap is easy to reason about and loose, because step cost grows through the run. A token budget binds on the thing you are actually billed for. Keep the step cap anyway as a cheap sanity bound and the deadline as the protection against a loop that is stuck rather than expensive.
Decide, in advance and in writing, what happens when it trips. This is the feature now, so design it. Return the work done so far plus the state needed to resume, and say clearly that the budget stopped it. The two bad answers are raising an opaque failure after spending the whole budget, and — far worse — summarising the partial work into something that reads like a completed answer. A truncated run that presents itself as finished is a quality failure you have paid full price for.
A ceiling on the retrievable prompt as well. The prompt term is what grows, so cap how much history and retrieved material a step may carry, and compact or drop the oldest rather than letting the step cost climb until the window stops it. Capping the window is what makes the worst-step figure in the model above a real number rather than an aspiration.
Billing alerts, as a backstop and nothing more. Provider spend alerts usually lag by hours and sometimes by a day, which is long enough for the whole weekend. They are the control that tells you the in-process budget was wrong, not the control that stops the run.
A forecast says what the loop will probably cost. A ceiling says what it cannot exceed. Both are useful and only one of them is a control.
The temptation is to build the forecast, because the forecast is a much nicer number: the median run takes a handful of steps with a short prompt and costs a fraction of a cent, and a ceiling built on the worst case is a number several orders of magnitude larger that will make somebody ask whether the feature is viable. That question is the point. A feature whose worst case is unacceptable has an unacceptable worst case whether or not anybody computed it.
The cost of being honest is that the ceiling, written as a cap, will sometimes bite a legitimate run. That is a real product cost and it is the thing to negotiate: raise the cap for a named class of task, let a person authorise an extension, or split the job so each piece fits. What you cannot do is leave the cap off because it would occasionally be inconvenient, which is how a nine-hour weekend run happens — nobody chose it, and nobody chose against it either.
The related arithmetic worth running is the other model next door. Context economics divides a window between a system prompt, retrieved material and accumulated history, and finds the turn at which it overflows. That model tells you how large the worst step's prompt gets and therefore supplies the input this one needs; it just cannot tell you what it costs, because it does not deal in money. Two models, one question each: how big does the prompt get, and what does that many of them cost.
Check
It stops when every step is approved by a person.
With a human confirming each action, the bound on the loop is that person's attention, and attention is a much harder limit than any number you would have written in a configuration file. A run cannot exceed what somebody will sit through, and the nine-hour weekend version cannot happen at all, because there was nobody there. The cost ceiling stops being the constraint and the throughput of the reviewer becomes it.
Two things follow, and both cut against the control in this lesson. The budget object is now ceremony — correct, cheap, and never the thing that binds — so spending design effort on it is spending it in the wrong place. And the failure mode moves: a reviewer approving the fortieth step of the day is the approval fatigue from the previous lesson, which means the real risk is a wrong action rather than an expensive one.
The honest boundary is narrower than "there is a human in the loop", though. Approval on the final action with twenty unreviewed model calls behind it is not this regime; it is the regime in this lesson with a confirmation dialog at the end, and the budget still has to be the control. The test is whether a person sees and authorises each billed call. If they do not, write the ceiling.