Part VI · Chapter 10 · Models in Production · lesson 1
12 min
Builds on What one request costs you, in cents, Why it gets slow at 70% busy, not at 100% and Your diagram has four dependencies. You have nineteen
You have finished seven tracks. Now somebody wants a feature with a language model inside it, and your list of new worries has about two dozen items on it. Most of that list is a list you already know how to work. Five items on it are not, and they have no equivalent anywhere in the first seven tracks. Sorting the two piles is the whole job of this lesson, because the pile you put a problem in decides which tool you reach for.
Predict
One question sorts the pile: if this went wrong right now, would any measurement you already have move?
If yes, it is an old problem. Use the old tool. A model call that takes seconds is a slow dependency; it belongs in the latency budget next to the payment gateway, and it is not more interesting than the payment gateway.
If no, it is new, and the work is to invent the measurement before you can manage the problem at all. That is why four of the next six lessons are about measurement rather than about mechanism.
The reason to be strict about this is cost in attention. A team that treats every model worry as unprecedented rebuilds retry logic it already had, and spends none of its remaining attention on the four things it has never built before.
Fourteen worries, where each one already lives, and what you would reach for. None of these is easy. All of them are familiar.
| The worry as it arrives | What it actually is | Where you met it |
|---|---|---|
| The call takes seconds | A slow dependency inside a latency budget | Tracks 1, 4 |
| Tokens cost money on every call | Cost per request, then unit economics | Track 0 |
| We get rate-limited at peak | A quota on a dependency, needing shed or queue | Track 5 |
| The provider had an outage | One more link in the availability chain | Track 5 |
| Timeouts plus retries made it worse | Retry amplification, entirely unchanged | Track 5 |
| The context limit is a hard wall | A capacity limit with a working set that grows | Tracks 0, 3 |
| We should cache the prompt prefix | Caching, with the same invalidation questions | Track 3 |
| Batching requests raises throughput | Queueing, plus group-forming delay | Track 1 |
| Streaming makes it feel faster | A progressive response, measured at first byte | Track 1 |
| Capacity cannot be added quickly | Provisioning lag, with a warm-up | Tracks 1, 5 |
| The vector store is a new database | A database with an index and a read path | Track 2 |
| The same job was processed twice | At-least-once delivery, needing idempotency | Track 3 |
| Sub-agents overload a downstream service | Fan-out amplification | Track 4 |
| One tenant's traffic starves the rest | A hot partition with noisy neighbours | Tracks 2, 7 |
Two of those get their own lesson here anyway — the next one on cost and the one after it on the two phases of a call — not because they are new, but because their arithmetic has a shape that catches people, and because a cost ceiling you cannot compute is the thing that stops a feature shipping.
One: there is no assertion to write. Equality works on a total, a status code and a row count. It does not work on a summary. Everything you know about testing assumes a function whose correct output you can name in advance, and that assumption has quietly held through every track so far.
Two: the same input does not give the same output. A bug report you cannot reproduce by replaying the request is a new category of bug report. Replay was the first move in every debugging story you have: send it again, watch it fail, bisect. That move now returns a different answer, which is neither the bug nor a refutation of it.
Three: a decline raises nothing. Output quality can fall for weeks with no exception, no alert, no failing test and no change in any dashboard you own. Every other degradation in this subject eventually showed up in a number somebody was already watching.
Four: the data channel is the control channel. Text you retrieved and put in the prompt is read with the same attention as the instructions you wrote. A document is now able to issue a command. There is no precedent for this in the first seven tracks, and the closest analogue outside them is SQL injection, which has a fix this does not have.
Five: the work per request is not set by the caller. A loop that calls a model, reads the answer and decides whether to call it again has no bound in its own definition. Every cost figure you have ever written assumed the request decided how much work it was asking for.
Not the same thing
A worry whose failure already moves a number you collect, and whose fix is a technique from an earlier track.
A worry whose failure moves nothing you collect, so the first piece of work is inventing the measurement.
Both sides accept
Both piles are real engineering problems that will cost you a weekend. Neither pile is harder than the other, and nothing here says the old pile is beneath you — the queueing and the retry arithmetic are the same arithmetic and still the most common cause of a bad launch.
Which is which? Put each one on a side.
The provider starts returning a too-many-requests code during the morning peak
A retrieved support article contains a sentence addressed to the model rather than to a reader
The monthly bill for the feature grows faster than the number of users does
Replaying the exact request from a bug report produces a different answer, and you cannot tell whether the bug is fixed
The first part of the answer arrives slowly enough that users hit reload
A ranking the feature produces has slowly stopped matching what people pick, and nothing has errored
One agent step spawns twelve sub-steps, and the search service behind them falls over
Nothing in the feature's definition says how many model calls one request will make
Before writing any of the feature, write four things down. Each one takes under an hour and each one is the thing a team regrets not having.
The eval set. Thirty to fifty real inputs with a judgement attached to each, in a file in the repository, before the prompt is written. It will be bad. A bad eval set that exists beats a good one that does not, and the first time you change the prompt you will have something to run.
The trust boundary. One sentence naming which text in the prompt is yours and which arrived from somewhere else, and one more naming what the model is allowed to do while any of the second kind is present. If the answer is "it can do anything it can do normally", you have found a problem rather than answered the question.
The cost ceiling. A number of model calls one request may make, enforced by a counter that refuses rather than by a code review. Say out loud what the feature does when the counter trips, because that behaviour is the feature now.
The quality signal. One number, collected from production, that would fall if answers got worse. Thumbs, acceptance of a suggestion, edit distance between what you proposed and what the user shipped, escalation rate. It does not need to be a good number. It needs to exist before the six weeks start.
Reach for the old toolbox for everything in the table: a latency budget, a timeout and retry policy with a bound, a cost-per-request figure, an idempotency key on anything that can be delivered twice, and a cached prefix if the prompt repeats. None of that is novel and all of it is still how the feature stays up.
SQL injection and prompt injection share a diagnosis: data arriving on a channel that is also read as instruction. They do not share a cure.
The SQL cure is a parser with two inputs. A prepared statement keeps the query structure and the parameter in separate arguments all the way down, so no quantity of hostile text in the parameter can change the structure. The separation is enforced by a grammar.
There is no grammar here. The prompt is one sequence, instructions and retrieved text are the same kind of token, and there is no equivalent of a bound parameter that the model is architecturally unable to read as a directive. Measures exist — delimiting, labelling provenance, a second model as a check — and all of them reduce the rate rather than close the hole.
So the design has to assume the instruction sometimes lands, which moves the work to what the model is permitted to do rather than what it is permitted to read. That is the subject of the sixth lesson here, and it is why that lesson is about authority rather than about sanitising input.
Check
The sort is a tool for allocating attention at the start of a project, and it goes wrong in one specific way: an old problem can acquire a new failure mode when the thing it is applied to is a model.
Caching is the clearest case. Caching a prompt prefix is ordinary caching, and the invalidation question is the ordinary invalidation question. But a cache keyed on a prompt that embeds one user's retrieved documents and is read by another user is a data-leak mechanism rather than a staleness mechanism, and the old toolbox does not prompt you to ask that. Idempotency has a similar edge: a retried call that is served from a cache returns the same answer, which is what you wanted, while a retried call that is not returns a different one, which means your deduplication key and your answer are no longer in agreement.
So the honest version of the rule: put the problem in the old pile, use the old tool, and then ask once what the model being on the other end of it changes. Usually nothing. Occasionally it changes who can read the cache.