Why it gets slow at 70% busy, not at 100%

Your service runs as eight pods, one single-threaded process each, behind a round-robin load balancer. Each handles one request at a time and takes 20ms, so the arithmetic ceiling is 400 requests a second. At 100 a second nobody complains. At 280 a second the on-call channel opens. The CPU graph reads 70% and looks like headroom, because a graph with 30% of its height unused looks like headroom.

It is not headroom. The number that would have told you lives in a formula with one division in it.

Predict

One worker, 20ms of real work per request, arrivals at random. What is the average time a request spends inside it — waiting plus working — when that worker is 70% busy, and when it is 90% busy?

Choose one answer

About 25ms at 70%, about 30ms at 90% — queueing stays negligible until you are nearly fullAbout 67ms at 70%, about 200ms at 90%About 29ms at 70%, about 67ms at 90% — it curves, but gentlyAbout 67ms at 70%, about 2 seconds at 90%

Lock in answer

The curve, with your numbers in it

The chart below is one pod, with its own queue, taking an eighth of the traffic. Slide the arrival rate and watch where the line leaves the floor — it is much earlier than the shape of a CPU graph suggests.

Queueing delay

≈ Hypothetical model, not a forecast

How long work waits in a single-server queue as utilisation rises, and the utilisation at which that wait breaks a stated budget.

It assumes

• Requests arrive at random, independent times rather than on a fixed cadence.
• One worker handles one request at a time, in arrival order, without preemption.
• The queue is unbounded and nothing waiting ever times out or gives up.
• Service time varies the way an exponential distribution says it does.

It ignores

• Load shedding, admission control, backpressure and timeouts — a real system refuses work rather than queueing forever, and the infinity at utilisation 1 is never actually reached
• Multiple servers, which change the formula and soften the knee considerably
• Service times more regular than exponential, which roughly halve the wait
• Bursty, correlated arrivals, which make the wait worse at every utilisation
• Retries, which turn an overloaded queue into a faster overloaded queue

Raise the arrivals until one worker's average response time passes 100ms — five times the work it is actually doing. Read off the utilisation, then multiply the arrival rate by eight to get the whole service's figure.

Arrivals35/s

Requests reaching the server each second.

Service time20 ms

How long one request occupies the server when nothing is queued.

Utilisation modelled

0.7

Utilisation offered, before clamping

0.7

Time waiting before service starts

47 ms

Time in the system, wait plus service

67 ms

Requests waiting

1.63

Response time as a multiple of service time

3.33

Utilisation at which the wait breaks the budget

83.333%

Offered load is at or above capacity

0

• Time in the system
• Time spent waiting

◷ Not there yet — keep moving the controls.

It happens at 40 arrivals a second — 80% of one pod, 320 requests a second across all eight. Not 400. The hardware's ceiling is 400 and this arrangement's useful ceiling is around 320, and the gap is not waste or inefficiency. It is the price of arrivals that do not politely space themselves out, paid eight times over because each pod queues alone.

The model also reports the utilisation at which the wait alone breaks a stated budget. At a 100ms budget and 20ms of work that is 83.3%, solved rather than searched: a budget B is broken at B / (B + S).

What people reach for

Autoscale on CPU. The default in Kubernetes, and as of this writing the Horizontal Pod Autoscaler's averageUtilization target is what nearly everyone sets. It works, late. The scale-up decision needs a metrics window, then a pod schedule, then a container start, then a readiness probe to pass — and the queue has been building through all of it. Setting the target to 70 does not give you 30% of headroom; it gives you a trigger that fires when response time has already tripled.

What I would actually do: set the HPA target at 50–60% rather than 70, and treat it as a cost decision rather than a performance one. The honest reason to run at 60% is that the scaling loop is slow and the queueing curve is not.

Alert on the queue, not the CPU. The metric you want is requests waiting, or time-in-queue, and most stacks already expose it: nginx's $request_time minus $upstream_response_time is the time the request spent waiting to be handed on; Puma reports backlog through Puma.stats; a Go service can export the length of its own semaphore channel. One number, measured where the queue actually forms, beats three dashboards of utilisation.

Share one queue across the workers. The highest-value change available here, and it is free. The eight pods above each queue alone, so each behaves like a single server and pays the full M/M/1 penalty. Eight workers drawing from one queue at the same 70% utilisation wait about a twentieth as long, because a request only waits when all eight are busy rather than when the one it was assigned to is.

Concretely: gunicorn's worker processes share one listening socket and therefore one kernel accept queue, which is the good case. Eight pods behind a round-robin balancer is the bad case, and it is the more common deployment. If you cannot consolidate, a least-outstanding-requests policy at the balancer recovers most of the benefit — Envoy's LEAST_REQUEST, or nginx's least_conn — because it approximates one queue by always picking the shortest.

Bound the queue. The model above assumes an unbounded queue and nothing giving up, which is why it goes to infinity. A real service should refuse work instead: a concurrency limit with a short wait and a 503 past it. Track 5 is about choosing that threshold; what matters here is that an unbounded queue converts a latency problem into an outage with extra steps.

Check

You cut the per-request work from 20ms to 10ms and traffic stays at 280 requests a second. What happens to response time?

Choose one answer

It halves, to about 33ms — the work halved and the load did not changeIt falls to about 15ms, because utilisation drops to 35% as wellIt stays about the same, because the bottleneck is the queue rather than the workIt cannot be predicted without measuring

Check answer

Where this stops being true

The model assumes arrivals at random, independent times. That is the pessimistic end of a range, and your traffic may sit at either end of it.

If arrivals are evenly spaced, the curve is roughly half as bad — the constant-service-time, constant-arrival case has no variability to queue behind, and a perfectly regular queue has no wait at all until utilisation reaches 1. This is not a hypothetical: a batch consumer pulling from a queue at a fixed rate, or a load generator in closed-loop mode, behaves this way. Which is exactly why a load test can show a clean curve that production never reproduces.

If arrivals are bursty or correlated — retries, a cron job in every client firing on the minute, a mobile app that reconnects everyone after a network blip — the wait is worse than this at every utilisation, and the knee moves left.

So: use the formula to decide you cannot run at 90%. Do not use it to promise a particular millisecond figure at 70%. The shape is robust; the constant is not.

›Why more servers soften the kneeOptional

With one server the response time is S / (1 − ρ). With c servers sharing one queue it is an uglier expression (Erlang C), but the behaviour is easy to state: the knee moves right and gets sharper.

At 70% utilisation, one server waits 2.33 times its service time. Eight servers at 70% wait about 0.1 times it — a twentieth of the delay for the same utilisation. The reason is pooling: with eight servers, the chance that all of them are busy when a request arrives is much lower than the chance that one is.

The catch is at the other end. Those eight servers hold out to a much higher utilisation, and then fall over faster when they do go, because there is less and less statistical slack left to absorb a burst. A large pool run at 95% is a quieter, more dangerous place than a small pool run at 95%.

And all of it depends on one queue feeding all eight. Eight separate queues, which is what you get when a load balancer assigns requests to instances without knowing what each is doing, behave like eight single servers rather than one pool of eight.

Worth remembering

• Response time in a single queue is the service time divided by the idle share, so it is the idle share that decides the answer, not the busy share.
• At 70% busy, a request waits 3.3 times its own service time. At 90% it is 10 times, and at 99% it is 100 times.
• Utilisation measures slack already spent, not room remaining, which is why a 70% CPU graph looks far more comfortable than it is.
• Eight 20ms workers each with their own queue have an arithmetic ceiling of 400 requests a second and a useful ceiling around 320.
• Eight workers sharing one queue wait about a twentieth as long at the same utilisation, because a request only waits when all eight are busy.
• Halving the service time pays twice, because service time appears both as the work and inside the utilisation.
• Alert on time spent queueing rather than on utilisation; the queue is the thing that hurts and most stacks already expose it.
• Evenly spaced arrivals roughly halve the wait, which is why a closed-loop load test produces a curve production never reproduces.