Part VI · Chapter 10 · Models in Production · lesson 3
The first token takes two seconds and users think it hung
14 min
Builds on Your endpoint takes 400ms and the budget was 200
Part VI · Chapter 10 · Models in Production · lesson 3
14 min
Builds on Your endpoint takes 400ms and the budget was 200
Your summariser works. A user pastes a long document, presses the button, and watches a spinner for two seconds before one character appears — then the answer arrives over the following five. Three support tickets say it hung. The fix is not a faster model and it is not a bigger instance: it is noticing that those two waits are two different workloads, and that only one of them is yours to shorten this week.
Predict
Prefill and decode
Hypothetical model, not a forecast
Splits one request into its two phases, says which one owns the time, and gives the prompt-to-output ratio at which that ownership changes hands.
It assumes
It ignores
Keep the answer at three hundred tokens and lengthen the prompt until reading it costs more than writing the answer. Read off the prompt length where that happens, and compare it to the document you are actually summarising.
| Prompt tokens | Reading the prompt | Writing the answer | Total |
|---|---|---|---|
| 100 | 13 ms | 5 s | 5.01 s |
| 200 | 25 ms | 5 s | 5.03 s |
| 500 | 63 ms | 5 s | 5.06 s |
| 1000 | 125 ms | 5 s | 5.13 s |
| 2000 | 250 ms | 5 s | 5.25 s |
| 5000 | 625 ms | 5 s | 5.63 s |
| 10000 | 1.25 s | 5 s | 6.25 s |
| 16000 | 2 s | 5 s | 7 s |
| 20000 | 2.5 s | 5 s | 7.5 s |
| 50000 | 6.25 s | 5 s | 11.25 s |
Not there yet — keep moving the controls.
Three readings matter in that summary and they answer three different questions.
Time to the first token. Two seconds, and it is almost entirely the prompt phase plus one generated token. This is the number the support ticket is about. Nothing about the answer length changes it.
Total time. Seven seconds, of which five is writing three hundred tokens. The answer owns the clock even though the prompt is over fifty times as many tokens.
The crossing. With these rates the phases cost the same at about one hundred and thirty-three prompt tokens per output token — a prompt of forty thousand tokens against this answer. Below that line, writing the answer dominates however long the prompt gets, which is why shortening a prompt to make a chat feature feel faster does almost nothing and shortening it to make a document summariser faster does quite a lot.
Streaming does not make the request finish sooner. Total time is unchanged: the same tokens come out at the same rate. What it changes is which of the two numbers above the user experiences.
Without streaming the user waits the total — seven seconds of nothing, then everything. With streaming the user waits the time to the first token — two seconds of nothing, then text that keeps arriving faster than they can read it. The perceived wait collapses from the total to the first phase, and the second phase disappears behind the reading speed of a human being.
That is a much larger product change than any latency work available to you this week, and it comes with three consequences people meet in production rather than in design.
The first phase is now the entire visible latency, so it is the only thing worth optimising for feel. Shorter prompts, a cached prefix, retrieving four chunks instead of twelve. Work on the generation rate improves the stream's pace, which users notice far less once it is above reading speed.
Streaming breaks every piece of infrastructure that buffers. A gateway that waits for a complete response, a serverless function that returns one payload, a response-body log, a schema validator that needs the whole object. Each of those converts a stream back into a seven-second wait, and they do it silently.
A streamed answer cannot be checked before the user sees it. Anything you wanted to validate, filter or refuse happens after the first words are on screen. That is a real tradeoff rather than an implementation detail, and it is why a feature with a hard output contract often cannot stream at all — the subject of the fourth lesson here.
This model treats reading the prompt as linear in its length: double the tokens, double the time. That is the useful approximation and it is wrong in a direction worth knowing.
The attention computation in the prompt phase compares every token with every other token, so its cost grows with the square of the prompt length rather than with the length. At short prompts the other, linear work dominates and the straight line is a good fit. At long prompts the quadratic term takes over and the real curve bends upward away from the model's line.
The second approximation leans the other way during generation. Each generated token reads all the state accumulated so far, so the state grows as the answer grows and each token takes slightly longer than the one before it. A constant rate therefore understates the time of a long answer.
Both errors push the same way: this model is optimistic about long prompts and long answers, and accurate enough in the middle, which is where most features live. The crossing it reports is the number to trust least if your prompt is enormous — measure that one rather than computing it.
Stream, unless an output contract forbids it. It is the single highest-value change available on a generation endpoint, it costs a day on most stacks, and its benefit is proportional to how much of your time is in the second phase — which, from the model above, is most of it.
Measure the two numbers separately and alert on the first one. Time to first token and total time, as two different series with two different budgets. A single latency figure averages them together and then tells you nothing, because they move for different reasons and respond to different fixes.
Spend prompt-shortening effort only where the crossing says it pays. Put your own prompt length and answer length into the model, read the ratio against the crossing, and act on the answer. A retrieval-augmented question-answering feature with a long context and a two-sentence answer is prompt-bound and worth trimming. A chat reply is not, and the afternoon spent compressing its system instruction buys a few tens of milliseconds.
Cap the answer length from the eval set, not from the provider maximum. The cap bounds the term that owns the clock, and the longest acceptable answer you have actually seen is a better bound than the largest the provider permits.
Rule out the queue before believing any of this. Everything above assumes the request started immediately. If requests are waiting to be grouped with others before either phase begins, that wait is in front of the first token and is invisible in both of these numbers. The queueing arithmetic for it is the arithmetic from the performance track, unchanged.
Check
It stops once prompts grow longer than the answers they produce — past the crossing the model reports, which with these rates is well over a hundred prompt tokens for every output token.
Past that line the advice inverts. Streaming stops being the big win, because the time to the first token is now most of the total and the user is back to staring at a spinner for the majority of the request. Generation rate stops mattering much. The levers that work are the ones that attack the prompt phase: caching a stable prefix so most of it is never read again, retrieving less, splitting the document and summarising the pieces in parallel, or moving the whole thing off the request path and telling the user you will email them.
Three kinds of feature live past the crossing routinely — whole-document summarisation, classification of long text into a single label, and anything that reads a large codebase or transcript to answer one question. If yours is one of those, measure the prompt phase directly rather than trusting the linear model, because that is also exactly the region where the quadratic term in the Deeper section takes over and the model's estimate turns optimistic.