Part IV · Chapter 7 · When It Breaks · lesson 4

Under overload you refuse work, or you lose all of it

14 min

Builds on Why it gets slow at 70% busy, not at 100% and The queue has 400,000 messages and is growing

A promotion goes out at 10:00 and arrivals roughly double. The checkout service accepts every request, as it always has. By 10:06 its p99 is over thirty seconds, every worker is busy, CPU is pinned, and the queue holds about nine thousand requests.

Successful checkouts in that minute: eleven. The mobile client gives up after ten seconds, so almost everything the service produces is a response to somebody who left.

The service is at full utilisation and delivering nothing.

Predict

Arrivals are twice what the service can handle, it queues everything, and clients time out after ten seconds. What does goodput look like after the first few minutes?

About half — it serves what it can and the rest failNear zero, because the wait grows past the client timeout and stays thereIt degrades smoothly — everybody gets a slow responseRoughly the same as before, with higher latency

An unbounded queue does not absorb overload. It converts a fast failure into a slow one and charges you for the difference.

Below capacity, a queue is a buffer: the wait settles at some finite value and bursts get smoothed. Above capacity, arrivals exceed departures by a fixed amount every second and the backlog grows without limit. The wait grows with it.

Past the point where the wait exceeds the caller's timeout, every unit of work the system completes is delivered to a caller who has gone. Capacity is fully consumed and goodput is zero. The queue also holds memory, so a long enough overload ends in an out-of-memory kill rather than a recovery.

Refusing work is not a degraded mode. It is the only mode in which an overloaded system produces anything at all.

Where the threshold comes from

The threshold is not a round number somebody picked. It falls out of the wait you are willing to tolerate and the time one request takes.

Queueing delay

Hypothetical model, not a forecast

How long work waits in a single-server queue as utilisation rises, and the utilisation at which that wait breaks a stated budget.

It assumes

Arrivals are random and independent, and service times vary exponentially around their mean.
One server handles one request at a time, in order, with no preemption.
The queue has unlimited room and nothing waiting ever gives up and leaves.
The wait budget covers queueing only — the twenty milliseconds of service sit outside it.
Utilisation is clamped just below one, because the formula has no finite answer at exactly one.

It ignores

Load shedding, admission control, backpressure and timeouts — a real system refuses work rather than queueing forever, and the infinity at utilisation 1 is never actually reached
Multiple servers, which change the formula and soften the knee considerably
Service times more regular than exponential, which roughly halve the wait
Bursty, correlated arrivals, which make the wait worse at every utilisation
Retries, which turn an overloaded queue into a faster overloaded queue

Leave the service time where it is. Raise the arrival rate until the model stops reporting a longer wait and starts reporting that the offered load is at or above capacity — the state where the number on the screen is a clamp rather than a forecast.

Arrivals45/s

Requests reaching the server each second.

Wait budget100 ms

The queueing wait that counts as too long. Service time is not part of it.

Utilisation modelled: 0.9
Utilisation offered, before clamping: 0.9
Time waiting before service starts: 180 ms
Time in the system, wait plus service: 200 ms
Requests waiting: 8.1
Response time as a multiple of service time: 10
Utilisation at which the wait breaks the budget: 83.333%
Offered load is at or above capacity: 0

Waiting before service starts
Total time in the system

Two curves almost flat to seventy per cent utilisation, bending upward through eighty-five and going near-vertical in the last few points before one.
Utilisation, per cent	Waiting before service starts	Total time in the system
1	202 us	20 ms
2	408 us	20 ms
5	1.1 ms	21 ms
10	2.2 ms	22 ms
20	5 ms	25 ms
30	8.6 ms	29 ms
40	13 ms	33 ms
50	20 ms	40 ms
60	30 ms	50 ms
70	47 ms	67 ms
80	80 ms	100 ms
85	113 ms	133 ms
90	180 ms	200 ms
92	230 ms	250 ms
94	313 ms	333 ms
95	380 ms	400 ms
96	480 ms	500 ms
97	647 ms	667 ms
98	980 ms	1,000 ms
99	1.98 s	2 s

Not there yet — keep moving the controls.

Three numbers to take away from that.

At forty-five arrivals a second against a twenty-millisecond service time, utilisation is 90 per cent and the queueing wait is about 180 milliseconds — nine times the service time, for a server that is idle a tenth of the time. Utilisation of 90 per cent is not 10 per cent of headroom; it is a tenfold latency multiplier.

The wait budget of 100 milliseconds breaks at 83.33 per cent utilisation. That is the threshold, and it is a closed form: the budget divided by the budget plus the service time. In arrivals it is about forty-one a second, which is the number to put in the admission limit.

And above fifty arrivals a second the model flags overload instead of quoting a wait, because there is no wait to quote. Nothing settles. The figure on the screen is a clamp the model applies so the arithmetic stays finite, and the honest answer is that the queue grows for as long as the overload lasts.

What people actually reach for

Bound the queue by time, not by length. A length limit has to be re-tuned every time the service time changes, and it is the wrong unit: what you care about is whether a request has been waiting longer than its caller will wait. The mechanism to copy is CoDel, adapted for request queues in Meta's fbthrift — it tracks the minimum time a request has spent in the queue over a sliding interval and starts dropping when that minimum stays above a small target, which detects a standing queue while ignoring short bursts. Pair it with LIFO service under overload: when the queue is deep, the newest request is the one most likely to still have a caller attached.

Put a concurrency limit in front, and measure it rather than guessing. Netflix's concurrency-limits library derives the limit from observed latency using gradient-based algorithms, which is Little's law run backwards: the limit that keeps latency at your target is the target latency times the throughput you can sustain. A measured adaptive limit survives a change in service time; a hand-set number does not.

Use what your edge already has. nginx's limit_req is a leaky bucket with a burst allowance and nodelay to reject rather than delay; limit_conn caps concurrency per key. Envoy's circuit breakers expose max_connections, max_pending_requests and max_requests per cluster, and setting max_pending_requests low is exactly a bounded queue with fast rejection. Kubernetes' own API server ships API Priority and Fairness, which is the full pattern as a worked example: named priority levels with their own concurrency shares and queue sets, flow schemas assigning requests to them, and a 429 with a Retry-After when a level is full.

Declare criticality, and shed the bottom tier first. The scheme described in Google's SRE book — four levels from CRITICAL_PLUS down to SHEDDABLE — works because the tier travels with the request and every hop honours it, so shedding can happen at whichever layer notices first. The important half is that the tier is set by the caller and verified at the boundary, otherwise everything arrives labelled critical within a quarter.

If you are adding one thing to a service that currently has nothing: a concurrency semaphore sized from your own measured latency target, in front of the handler, returning 429 with Retry-After when it cannot acquire. It is perhaps twenty lines. Everything above is a better version of it.

What would you do

10:08. Arrivals are a little over double what the service can serve. The queue is unbounded and about nine thousand deep. Clients time out after ten seconds. Eleven checkouts completed in the last minute.

You have: a load balancer you can configure, an autoscaling group with a cold-start of around four minutes per instance, a feature flag that disables the recommendation panel, and the service's own queue settings behind a config reload.

Optimising for: Maximise successful checkouts over the next thirty minutes, with the promotion traffic already arriving and no time to deploy anything that needs review.
You cannot know: How long the promotion traffic lasts — the marketing email went to everybody at once, so it may decay in ten minutes or hold for an hour
What share of the arrivals are client retries of requests that already timed out
Whether the four-minute cold start is honest under this much load on the shared image registry
Whether the queue is holding enough memory to get the process killed before any of this takes effect

Scale the autoscaling group out by a factor of three and waitCap in-flight requests at the measured limit and return 429 with Retry-After to everything above itDrop the client timeout and the server's own queue timeout to two secondsTurn off the recommendation panel with the feature flag to free capacity

Check

Your queue is deep and you have to drop something. Which request is cheapest to drop and which is most expensive?

Drop the oldest requests first; serve the newestDrop the newest requests first, so the queue stays fair to whoever arrived earliestDrop the most expensive requests, since they consume the most capacityDrop requests at random, because any selection rule is itself work

Worth remembering

An unbounded queue does not absorb overload; it converts a fast failure into a slow one and then produces responses nobody is waiting for.
Above capacity nothing settles: arrivals exceed departures by a fixed amount each second and the wait grows for as long as the overload lasts.
Throughput can stay at its maximum while goodput goes to near zero, which is why utilisation graphs do not show this failure.
The admission threshold is the wait budget divided by the budget plus the service time — at a 100-millisecond budget and 20 milliseconds of service, 83.33 per cent utilisation, or about forty-one arrivals a second.
Ninety per cent utilisation is not ten per cent of headroom. At these figures it is a queueing wait of nine times the service time.
Bound queues by time in queue rather than by length, because length has to be re-tuned whenever service time changes and time is what the caller cares about.
Drop in this order: past their deadline, then retries, then lowest criticality tier, then oldest. Serve newest-first under deep overload, and never shed the health check.
Reject legibly — 429 with Retry-After is a contract the client can act on; a long hang is not.

Where this stops holding

Every mechanism above buys time. They assume the overload is a burst that an admission limit can hold the line through until it passes, or until capacity arrives.

If the overload is permanent — a tenfold traffic increase that is the new baseline, a dependency that has become four times more expensive to call, an abuse pattern that scales as fast as you shed it — then no queue bound and no concurrency limit changes the outcome. Shedding half of a doubled load serves your old traffic; shedding ninety per cent of a tenfold load serves your old traffic and refuses nine of every ten customers indefinitely, which is a business failure wearing an availability number.

The crossing to watch for is the duration of the overload against the time it takes you to add real capacity. While overload is shorter, admission control is the answer and this lesson holds. Once overload consistently outlasts your provisioning time, admission control has stopped being a fix and become a way of reporting a capacity decision nobody has made yet — and the work moves to the cost per request and the number of machines, which is a different track.