Part V · Chapter 8 · Running It · lesson 7

Ship a number you measured, and say what would reverse it

16 min

Builds on Say what would change your mind, before you look at options, How many requests can be inside your service at once and You raised the limit and the bottleneck moved somewhere else

Put WORKERS back to one. You now know this service completes about forty-seven requests a second, that its latency is the queue behind a single full worker slot, and that above capacity it works hardest while delivering least. The last knob is LIMIT: the depth at which the door refuses a connection without reading it.

Pick a number, defend it with a measurement, and write down what would make you change it.

Predict

You set an admission limit that holds the ninety-ninth percentile at about a quarter of a second, then offer the service twice the traffic it can serve. What happens to the number of answers callers actually receive?

It roughly halves, because half the arrivals are refusedIt holds at the ceiling, about forty-nine a second, at every offered rate above capacityIt falls to near zero, because the limit is reached immediately and almost everything is refusedIt rises above the ceiling, because refusing work frees capacity

Deriving the number rather than guessing it

The criterion comes first, because the number is a function of it. Here it is, in one sentence: the ninety-ninth percentile of answers received stays under a quarter of a second at any offered rate, and delivered answers never fall below the measured ceiling.

A quarter of a second is a product decision and not an engineering one. It came from somewhere outside this lab — a mobile client's own budget, a caller's timeout, a page that has to paint. The engineering starts once it is written down.

Little's law

Hypothetical model, not a forecast

Ties arrival rate, latency and concurrency together, solving for whichever of the three is left implied.

It assumes

The service drains at the measured ceiling of about forty-eight a second.
The latency figure is the whole time in the system, queueing included, not service alone.
The target latency is set a little under the stated budget to leave room for the tail.
Every admitted request occupies exactly one slot, in service or waiting.
The system is in steady state with the queue held at its bound, which is the regime the limit creates.

It ignores

Warm-up and shutdown periods, where the law simply does not apply
The cost of concurrency itself — memory, context switches, lock contention
Whether the implied concurrency is physically achievable on the hardware
Variance: this is an identity about averages and says nothing about tails

Hold the drain rate where it is and lower the target latency from two hundred and thirty milliseconds to one hundred. Read off how many requests may be in flight, and notice that the admission limit is that figure minus the one being served.

Arrivals48/s

Requests entering the system each second.

Latency230 ms

Average time one request spends in the system, entry to exit.

Implied concurrency: 11.04
Which quantity is implied: 0 concurrency, 1 arrivals, 2 latency: 0
Arrivals: 48/s
Latency: 230 ms
Requests in flight: 11.04
Arrival rate this concurrency sustains: 48/s

Not there yet — keep moving the controls.

Little's law, read backwards. Requests in flight equals the rate they leave at times the time each one spends inside. Set the time to the budget and the rate to the measured ceiling, and the identity hands you the number of requests allowed inside: a little over eleven.

One of those eleven is in service. The other ten are the queue. So the admission limit is ten, and the general form — the thing worth carrying off this page — is:

limit = workers × (latency budget − service time) / service time

One worker, a two-hundred-and-thirty-millisecond budget, a twenty-one millisecond service time: one times two hundred and nine over twenty-one, which is nine and nine tenths. Ten.

Shipping it

    environment:
      WORKERS: "1"
      SPIN_MS: "2"
      WAIT_MS: "18"
      LIMIT: "10"

docker compose up -d --force-recreate service
docker compose run --rm load --check
docker compose run --rm load --rate 40,48,60,100 --seconds 20

The second line of the acceptance output should now read limit 10 rather than limit off.

With the limit:

 offered    sent     ok  refused  gaveup  error  ok_per_sec   p50_ms   p99_ms    done    lost  queued
      40     797    795        2       0      0        39.8     61.5    225.1     795       0      10
      48     931    918       13       0      0        45.9    101.8    225.7     918       0      10
      60    1178    979      199       0      0        49.0    203.7    226.4     979       0      10
     100    1987    985     1002       0      0        49.2    225.3    226.8     985       0      10

The same two rates with the limit off, from lesson three:

 offered    sent     ok  refused  gaveup  error  ok_per_sec   p50_ms   p99_ms    done    lost  queued
      60    1178    503        0     675      0        25.1   1106.6   1969.0    1068       0     207
     100    1987    174        0    1813      0         8.7    940.6   1985.9    1074       0    1011

Four results, and they are the record.

The tail stopped moving. Two hundred and twenty-five, two hundred and twenty-six, two hundred and twenty-six, two hundred and twenty-seven milliseconds, across offered rates from forty a second to a hundred. The criterion asked for under a quarter of a second and the measurement is pinned just below it — not because anything is clamping the output, but because the deepest the queue may ever be is ten, and ten requests at twenty-one milliseconds each is the longest wait that can exist.

Delivered answers held at the ceiling instead of collapsing. At sixty offered: forty-nine a second with the limit, twenty-five without. At a hundred offered: forty-nine with, nine without. The limit multiplied delivered answers by a factor of five and a half at double overload, and the mechanism is not speed — it is that the worker stopped spending itself on answers nobody was waiting for.

Wasted work went to zero. Compare the done and ok columns. Without the limit at a hundred offered, the service completed 1074 requests and 174 of those answers were received: nine hundred requests' worth of work thrown away. With the limit, 985 completed and 985 received.

The refusals are now the visible cost. A thousand refusals at a hundred offered — half the arrivals told no, immediately, with a status a client can act on. That is the trade, stated in the one column that did not exist before.

The record

This is the rung-five deliverable, and it is four fields. Written for this lab so the shape is clear; write your own from your own sweep.

Setting. Admission limit of ten queued connections, one worker, refusing with a 503 and a retry hint.

Criterion. The ninety-ninth percentile of answers received stays under a quarter of a second at any offered rate; delivered answers never fall below the measured drain rate. Latency over completeness, deliberately: the caller here is a mobile client that abandons a request after two seconds, so an answer delivered late is an answer thrown away twice.

Measurement. Sweep at forty, forty-eight, sixty and a hundred arrivals a second, twenty-second windows, open-loop Poisson arrivals from a fixed seed, on one laptop with one worker. With the limit, p99 held between two hundred and twenty-five and two hundred and twenty-seven milliseconds and delivered answers held at about forty-nine a second at every rate. Without it, p99 ran to about two seconds and delivered answers fell to twenty-five a second at sixty offered and nine a second at a hundred.

Number. Ten, from workers times the budget minus the service time, over the service time: one times two hundred and nine over twenty-one.

What would reverse it. A measured service time materially different from twenty-one milliseconds, because the limit is derived from it and moves inversely: a service time of ten milliseconds raises the limit to twenty-two, and one of forty lowers it to five. A change in the worker count, which scales it linearly. A caller whose budget is not a quarter of a second. And a traffic mix in which the expensive requests are the ones worth keeping, because this limit is indiscriminate and refuses by arrival order alone.

What makes this a record rather than a note is the last field. A number with no reversal condition is a number nobody can audit, including you in six months — and the most common way a good setting becomes a bad one is that the thing it was derived from moved and nothing was written down to notice.

What people actually reach for instead of a fixed depth

The number above is correct for a machine whose service time is known and stable. Three mechanisms that survive a service time that moves, in the order I would adopt them.

Bound by time in queue rather than by depth. The thing the caller cares about is how long their request has been waiting, and depth is a proxy for it that has to be re-derived whenever the service time changes. The mechanism to copy is CoDel adapted for request queues: track the minimum time a request has spent queued over a sliding window, and start refusing when that minimum stays above a small target. A standing queue is detected; a short burst is absorbed. Meta's fbthrift implementation is the reference.

Derive the limit from observed latency, continuously. Netflix's concurrency-limits does the derivation in this lesson on a loop, using gradient algorithms to raise the limit while latency is flat and lower it when latency rises. It is Little's law run backwards once a second rather than once in a lesson, and the honest reason to prefer it is that it survives a deploy that changes the handler's cost.

Use what the edge already has, for the crude version. nginx's limit_conn caps concurrency per key and limit_req is a leaky bucket with a burst allowance and nodelay to refuse rather than delay. Envoy's circuit breakers expose max_pending_requests per cluster, which is exactly a bounded queue with fast rejection. None of these knows your service time, so each needs the number from this lesson typed into it — which is fine, as long as the record says where the number came from.

And refuse legibly, whichever you use. A 503 or a 429 with a Retry-After is a contract the client can act on. A connection reset is not, and a two-second hang is the thing you are trying to prevent. The lab's door answers with a 503 and a retry hint before closing, and the one line in its implementation worth reading twice is the single recv before the refusal: a socket closed with unread bytes still in its buffer is reset rather than closed, and a reset takes the refusal away with it.

Check

You deploy the limit of ten to production, where the service runs with four workers on faster hardware with a measured service time of twelve milliseconds. What is the effect, and what is the right limit?

The limit is far too tight — it refuses work the service could have served inside the budget. Four workers at twelve milliseconds against the same budget gives about seventy-threeThe limit is about right, because a faster machine finishing sooner needs less queueThe limit is far too loose, because four workers means four times as many requests in flightIt makes no difference, because the limit only applies under overload and production is not overloaded

Where this stops holding

The limit is ten because of two measurements taken on one laptop: a service time of twenty-one milliseconds and a worker count of one. Both appear in the formula and the formula is the whole derivation.

The crossing to watch for: once the machine you deploy to differs from the one you measured on in either term — a different core count, a different container quota, a different worker configuration, a handler whose cost changed in a release — the number is wrong by the ratio and wrong in a direction that is not obvious. Too tight refuses work the service could have served, which appears as intermittent errors under ordinary bursts. Too loose lets the queue grow past the budget, which appears as the original problem.

What to do about it, in increasing order of effort: re-derive the number per environment and put it in configuration rather than in code; have the service measure its own service time at startup and compute its own limit; or adopt one of the adaptive mechanisms above and stop shipping a constant at all. The record's last field exists so that whoever reads it knows which of those three you did, and what to re-measure if the answer stops holding.

Worth remembering

The criterion comes before the number, because the number is a function of it: a latency budget, measured drain rate, and a decision about latency against completeness.
Little's law read backwards gives the requests allowed in flight — drain rate times budget — and the admission limit is that figure minus the workers already serving.
The portable form is workers times the budget minus the service time, over the service time. One worker, a two-hundred-and-thirty-millisecond budget and a twenty-one-millisecond service time gives ten.
Measured with the limit on, the ninety-ninth percentile held between two hundred and twenty-five and two hundred and twenty-seven milliseconds across offered rates from forty a second to a hundred.
Delivered answers held at about forty-nine a second at every rate, against twenty-five and nine a second with the limit off — a factor of five and a half at double overload.
Wasted work went from nine hundred requests' worth in a twenty-second window to zero, which is the mechanism: the limit does not make the service faster, it stops it serving callers who have left.
The cost of the limit is now visible in a column: half the arrivals refused at double overload, immediately and with a status the client can act on.
A limit derived from a measurement is valid only on the machine measured. The same criterion on four workers at twelve milliseconds gives about seventy-three, not ten.
A record is four fields — setting, criterion, measurement, number — and it is the fifth, what would reverse it, that makes it auditable.