Part V · Chapter 8 · Running It · lesson 5

You raised the limit and the bottleneck moved somewhere else

14 min

Builds on How many requests can be inside your service at once and Raise the rate until the completions stop rising

One worker, full from two thirds of capacity upwards, with the latency standing behind it as queue. The fix is one line of compose.yaml: change WORKERS from one to four. Little's law says the ceiling should go from about fifty a second to about two hundred.

It does. Then you set it to sixteen, Little's law says eight hundred, and the service refuses to go past five.

Predict

With four workers the service completes about two hundred a second, as the arithmetic says. You raise WORKERS to sixteen and sweep again. Where does the completions column plateau?

About eight hundred a second — sixteen in flight divided by twenty-one milliseconds, the same division that worked at fourAbout five hundred a second — one divided by the two milliseconds of processor time each request costs, which no number of workers dividesStill about two hundred, because the four-worker result was already the machine's limitIt falls below two hundred, because sixteen threads contending for one interpreter is worse than four

The arithmetic that is about to stop applying

Little's law

Hypothetical model, not a forecast

Ties arrival rate, latency and concurrency together, solving for whichever of the three is left implied.

It assumes

Each worker thread serves one connection at a time from start to finish.
Time in flight is the full twenty-one milliseconds measured at rest, spin and sleep together.
Every worker is available to take work, with no shared lock serialising them.
The service is in steady state, with arrivals and completions in balance.
The machine can supply whatever processor time the implied arrival rate demands.

It ignores

Warm-up and shutdown periods, where the law simply does not apply
The cost of concurrency itself — memory, context switches, lock contention
Whether the implied concurrency is physically achievable on the hardware
Variance: this is an identity about averages and says nothing about tails

Raise the requests in flight from four to sixteen and read the arrival rate the identity now allows. Then hold that figure in mind while you read the measured sweep below, which stops well short of it.

Latency21 ms

Average time one request spends in the system, entry to exit.

Requests in flight4

Implied arrival rate: 190/s
Which quantity is implied: 0 concurrency, 1 arrivals, 2 latency: 1
Arrivals: 190/s
Latency: 21 ms
Requests in flight: 4
Arrival rate this concurrency sustains: 190/s

Not there yet — keep moving the controls.

At four in flight the identity allows a little over a hundred and ninety a second. At sixteen it allows a little over seven hundred and sixty. The first figure is about to be confirmed to within five per cent and the second is about to be missed by a factor of one and a half, and the model's fifth assumption is the one that breaks.

Four workers

One edit, then the same sweep over a higher range:

    environment:
      WORKERS: "4"
      SPIN_MS: "2"
      WAIT_MS: "18"
      LIMIT: "0"

docker compose up -d --force-recreate service
docker compose run --rm load --check
docker compose run --rm load --rate 48,80,120,150,170,190,200,220 --seconds 20

The acceptance check should now print 4 worker(s) on its second line. If it still says one, the container was not recreated.

 offered    sent     ok  refused  gaveup  error  ok_per_sec   p50_ms   p99_ms    done    lost  queued
      48     931    931        0       0      0        46.5     21.9     33.4     931       0       1
      80    1569   1569        0       0      0        78.5     23.3     44.6    1569       0       5
     120    2388   2388        0       0      0       119.4     25.1     61.7    2388       0      10
     150    2972   2972        0       0      0       148.6     26.4     79.9    2972       0      14
     170    3397   3397        0       0      0       169.8     40.5    103.4    3397       0      20
     190    3775   3775        0       0      0       188.8     63.2    173.1    3775       0      35
     200    3985   3985        0       0      0       199.2    327.1    533.2    3985       0     103
     220    4381   3506        0     875      0       175.3    983.4   1987.0    4291       0     479

Two things changed and one did not.

The ceiling moved by almost exactly the factor the arithmetic promised. About forty-seven a second became about a hundred and ninety-nine. Four times the workers, four times the throughput, and the identity's prediction of a hundred and ninety came in about five per cent low.

The row that used to be the knee is now unremarkable. Forty-eight offered produced a median of a hundred and twenty-two milliseconds with one worker. It produces twenty-two with four — the service time, with nothing queueing. The same arrival rate, the same handler, and the latency problem has gone.

The shape did not change at all. Flat until about three quarters of the new ceiling, bending at a hundred and seventy, vertical at two hundred, collapsing at two hundred and twenty. Every lesson from track one still applies; only the axis moved.

Sixteen workers

docker compose run --rm load --rate 200,300,400,450,500 --seconds 20

 offered    sent     ok  refused  gaveup  error  ok_per_sec   p50_ms   p99_ms    done    lost  queued
     200    3985   3985        0       0      0       199.2     27.0     43.6    3985       0       2
     300    5975   5975        0       0      0       298.8     29.7     49.1    5975       0       5
     400    7982   7982        0       0      0       399.1     36.1     66.8    7982       0       9
     450    8981   8981        0       0      0       449.1     52.7    108.0    8981       0      13
     500   10006   9620        0     386      0       481.0    404.9   1962.2    9662       0      26

Four hundred and eighty-one a second, against an arithmetic promise of seven hundred and sixty. The prediction is out by a third, in the direction that matters.

Then the confirming experiment, which costs one more edit and two minutes. Set WORKERS to sixty-four and sweep the same top end:

 offered    sent     ok  refused  gaveup  error  ok_per_sec   p50_ms   p99_ms    done    lost  queued
     450    8981   8981        0       0      0       449.1     61.0    127.4    8981       0       3
     500   10006   9527        0     479      0       476.4    397.3   1969.2    9588       0       5
     550   10969   9523        0    1446      0       476.1    421.0   1980.2    9704       0       5

Sixteen times the workers of the four-worker run, and four times the workers of the sixteen-worker run, for a ceiling that moved by less than one per cent. The worker count has stopped being a lever.

One divided by two milliseconds is five hundred a second. Measured: four hundred and seventy-six to four hundred and eighty-one, across worker counts differing by a factor of four. That is the processor ceiling from lesson two, within five per cent, and it is now the binding one.

The reading that goes quiet, and why that is the dangerous part

Look at the queued column on the sixty-four-worker run. Five. At an offered rate half again above capacity, with a median of four hundred milliseconds and a p99 near two seconds, the deepest the queue ever got was five.

With one worker, overload showed up as a queue two hundred deep. With sixty-four, overload shows up as sixty-four requests all in service simultaneously, each taking far longer than it should because they are sharing one processor. The queue is empty because every request was handed straight to a worker.

Which means the admission limit you are about to build in lesson seven — the one keyed on queue depth — would never fire at sixty-four workers. The mechanism that bounds latency perfectly at one worker is blind at sixty-four. Raising the worker count did not only move the ceiling; it moved the place where the evidence lives.

What people actually reach for

Separate the two kinds of concurrency, because they buy different things. Threads inside one process buy you waiting capacity: more requests parked in a sleep, a socket read or a database round trip. Processes buy you processor capacity, because each has its own interpreter and its own lock. In Python the distinction is load-bearing and visible: gunicorn's --workers are processes and its --threads are threads within each, and the handler's own ratio of waiting to computing tells you which number to raise.

For this handler — two milliseconds of computing to eighteen of waiting — the ratio says ten threads per process is where the processor becomes the constraint, and past that you need more processes. That is the arithmetic version of a rule of thumb usually given as "threads for I/O, processes for CPU", and the ratio is how you turn it into a number.

What I would actually set, for a service shaped like this one. Processes equal to the available cores, threads per process equal to the ratio of total request time to processor time per request, and then measure rather than believe it. For this handler on a four-core box that is four processes of ten threads: a worker ceiling near two thousand a second against a processor ceiling near two thousand, which is the balanced point. The reason to compute it rather than copy a default is that the default in every framework is tuned for a different ratio than yours.

Where this goes wrong, named. Raising threads on a handler that is mostly computing buys nothing and costs memory and context switches. Raising processes on a handler that holds a per-process cache multiplies the memory and lowers the hit rate. And raising either against a shared downstream — one database with a connection limit, one Redis instance, one external API with a rate quota — moves the ceiling onto something you do not control, which is strictly worse than a ceiling you can see. Before raising a worker count, find out what the next thing along is rated for.

The general shape, which is track one's lesson six measured. The part of the work that refuses to divide sets the ceiling. Here it is two milliseconds of ten, a serial fraction of one tenth, and the most any amount of parallelism can buy is ten times the single-worker figure: fifty a second becomes five hundred and stops. Measured, forty-eight became four hundred and eighty.

Check

You are at the processor ceiling of about five hundred a second with sixteen workers and you need a thousand. Which change actually moves it, and which look like they should?

Halve the processor cost of the handler, or run a second process on a second core — both double the ceiling. Raising the worker count, the queue size or the client timeout move nothingRaise the worker count again — sixty-four was not enough, so try two hundred and fifty-sixRaise the queue bound so fewer requests are refused under loadRaise the client timeout so requests that are queued long enough still get their answers

Where this stops holding

Everything above rests on the two ceilings being far apart, so that one of them is obviously the answer at any given worker count. At one worker they differ by a factor of ten and the diagnosis is trivial.

The crossing to watch for: once the processor ceiling sits within about twice the worker ceiling, neither one is the answer and both bind at once. The symptom is a sweep whose plateau arrives earlier than either arithmetic predicts and whose latency rises before the plateau, because requests are now queueing for a worker and contending for the processor. Four processes of ten threads on four cores — the configuration recommended above — is deliberately placed at that crossing, which is the balanced point and also the point at which neither single number describes the service any more.

What to do there: stop reasoning about ceilings and measure the pair. Sweep the arrival rate at two worker counts, one on each side of the configuration you are considering, and take the plateau you measure rather than either figure you can compute. The arithmetic is for finding which lever to pull. Once both levers are live, only the sweep is honest.

As of 2026-10-31

Dated as of this page: the docker compose up -d --force-recreate spelling; gunicorn's --workers and --threads flags and the fact that the first are processes and the second threads within each; and the claim that threads inside one CPython process serialise on a single interpreter lock, which is true of the default build on this date but is exactly what the free-threaded builds introduced from Python 3.13 onwards are changing — on such a build the processor ceiling in this lesson moves with the core count and the measured plateau would be different. The framework defaults for worker and thread counts have all moved at least once; compute the ratio for your own handler rather than copying a number. The arithmetic — the smaller of the two ceilings binds, and the serial fraction caps the speedup — does not expire.

Worth remembering

Raising the worker count from one to four moved the ceiling from about forty-seven a second to about a hundred and ninety-nine, within five per cent of what Little's law promised.
Raising it from four to sixteen moved the ceiling to about four hundred and eighty-one against an arithmetic promise of seven hundred and sixty — the identity stopped being the binding constraint somewhere in between.
Raising it again from sixteen to sixty-four moved the ceiling by less than one per cent, which is the experiment that proves the lever is dead.
The new ceiling is one divided by the processor cost of a single request: two milliseconds of serial work per request caps the process at about five hundred a second, measured at four hundred and seventy-six to four hundred and eighty-one.
The shape of the curve never changed. Only the axis moved, so every conclusion from the earlier tracks still applies at the new scale.
Raising the worker count also moved where the evidence lives: at sixty-four workers an overloaded service shows a queue depth of five, so an admission limit keyed on queue depth is blind exactly when it is needed.
Threads buy waiting capacity and processes buy processor capacity; the handler's ratio of total time to processor time per request is the number of threads at which the processor takes over.
Once the processor ceiling sits within about twice the worker ceiling, neither arithmetic describes the service and only a measured sweep is honest.