How many requests can be inside your service at once

Somebody sized the worker pool at 16 because the box has 16 cores. The service handles each request in 50ms, of which 40 is a Postgres round trip. Traffic is now asking for 400 requests a second and the completions have flattened at a number well below that, with a growing queue in front of nginx.

The pool size and the throughput ceiling are the same fact stated twice. Here is the conversion.

Predict

A pool of 16 slots. Each request occupies one slot for 50ms, start to finish. What is the most requests a second this service can complete, assuming the slots are the only constraint?

Choose one answer

16 a second — one per slot per second320 a second800 a second — 16 cores at 50ms is a core-second per request, so 16 divided by 0.02It depends on the hardware and cannot be derived from these two numbers

Lock in answer

The ceiling, and what moves it

Below is the identity with the pool held at 16. The rising line is the concurrency the traffic demands; the flat line is the concurrency you have. They cross at the ceiling.

Little's law

≈ Hypothetical model, not a forecast

Ties arrival rate, latency and concurrency together, solving for whichever of the three is left implied.

It assumes

• The system is in steady state, with arrivals and completions in balance.
• Latency is the whole time a request holds a slot, not the time it spends computing.
• Every slot is interchangeable and none of them is reserved for anything else.
• The slots are the binding constraint, so nothing downstream saturates first.

It ignores

• Warm-up and shutdown periods, where the law simply does not apply
• The cost of concurrency itself — memory, context switches, lock contention
• Whether the implied concurrency is physically achievable on the hardware
• Variance: this is an identity about averages and says nothing about tails

Today's peak is 400 a second. Hold the pool at 16 and raise the latency until this pool cannot carry even 150 a second — then read off how much latency it took.

Latency50 ms

Average time one request spends in the system, entry to exit.

Requests in flight16

Implied arrival rate

320/s

Which quantity is implied: 0 concurrency, 1 arrivals, 2 latency

1

Arrivals

320/s

Latency

50 ms

Requests in flight

16

Arrival rate this concurrency sustains

320/s

• In flight, demanded
• In flight, available

◷ Not there yet — keep moving the controls.

It takes about 107ms. A slightly over two-fold rise in latency cuts the service's throughput ceiling by slightly over half, from 320 a second to under 150 — and the latency rise does not have to be yours. A Postgres primary with a fuller buffer cache, an index that stopped being used, a replica promoted onto a smaller instance: any of those lengthens the hold time, and the hold time is the divisor.

This is the mechanism behind "it worked at 100 and fell over at 400". The pool did not change. The hold time did, and the ceiling moved with it.

Sizing a pool without guessing

Three numbers and one multiplication:

1. Peak arrival rate, from the request-count metric at its highest minute over the last month. Not the average — the average is roughly a sixth of peak for anything with a daily rhythm.
2. Hold time at that rate, measured at p95 rather than at the median, because the pool has to hold the slow ones too. In practice this is the per-dependency client-side latency you exported in the previous lesson.
3. Multiply, then add margin. L = λ × W, and then at least 1.5 times that figure, because the identity gives you the average occupancy and arrivals do not arrive on average.

For 400 a second at a p95 hold of 60ms: 400 × 0.06 = 24 slots, times 1.5 is 36. So gunicorn --workers 4 --threads 12 across three pods, or 12 workers of 3 threads — the split matters for memory and for the GIL, not for this number.

And then the part people skip: the pool has to be the smallest of the pools. Thirty-six application slots each wanting a database connection, against pgbouncer's default_pool_size of 20, means the real concurrency limit is 20 and you have built a queue between two of your own components. Size inwards: the database's concurrency limit first, then the pool in front of it, then the worker slots in front of that.

What people reach for

A bigger pool. The first move and often correct, up to the point where the pool stops being the constraint. Past that point every added slot is a request admitted into a longer queue, which raises its hold time, which eats the capacity the extra slot was supposed to buy. HikariCP's own guidance has argued for years that small pools outperform large ones against a relational database, and the arithmetic reason is this one: a database asked to run more CPU-bound statements at once than it has cores does not finish them any sooner, so the extra sessions only lengthen W. The caveat is the workload — statements that spend most of their time waiting on storage do benefit from more concurrency than the core count, which is why the figure to size against is the database's own measured concurrency at saturation rather than its core count.

A connection-pool timeout. This is the setting I would not ship without. HikariCP's connectionTimeout, or SQLALCHEMY_POOL_TIMEOUT on the Python side, bounds how long a request waits for a slot before failing. Without it, a saturated pool is an unbounded queue, and an unbounded queue is the slow version of an outage. With it, you shed the overflow and the requests that do get through keep their latency. Set it well below the caller's own timeout — otherwise the caller gives up first and you do the work for nobody.

Async, to make the slots cheap. Switching from sync workers to asyncio or virtual threads genuinely removes the worker-slot ceiling: ten thousand coroutines waiting on sockets cost almost nothing. It does not remove the database pool's ceiling, and it makes the queue in front of that pool enormous rather than visible. Async without an explicit concurrency limit converts "we run out of workers at 320 a second" into "everything times out at 320 a second", which is harder to diagnose and worse for the user.

An explicit in-flight limit. The honest version of all of the above: pick a number from L = λ × W, enforce it at admission, and return 503 past it. Envoy does this through a circuit-breaker max_requests; a Go service does it with a buffered channel; nginx does a cruder version with limit_conn. The value of making it explicit is that the ceiling becomes a number in a file that somebody chose, instead of the product of two unrelated settings nobody connected.

Check

You double the pool from 16 to 32 slots. Postgres has 8 cores and your queries are CPU-bound on the database side. What happens to throughput?

Choose one answer

It roughly doubles, to about 640 a secondIt rises a little and latency rises a lotNothing changes, because the pool was never the constraintThroughput falls, because more concurrency always means more contention

Check answer

Where this stops being true

Little's law needs steady state: completions keeping pace with arrivals over the averaging window. During the thing you are most likely to be looking at, it is not in steady state at all.

In an overload, arrivals exceed completions and the backlog grows. In that regime L is not λ × W because W is not finite yet — it is still increasing, and any value you measure for it is a reading of a transient. The same applies in the first seconds after a deploy, during a cold-cache period, and while a queue drains.

The practical rule: average over a window long enough that arrivals and completions match within a few per cent, and treat any window where they do not as a measurement of the overload rather than of the service. If the backlog is growing, the number you want is not a pool size — it is a shedding threshold, which is a different lesson.

›Why peak is roughly six times averageOptional

The margin advice above leans on a figure worth justifying, because it decides the pool size.

A consumer product does most of a day's traffic in about four waking hours. 24 divided by 4 is 6, so peak-minute traffic lands near six times the flat daily average. An internal tool used in one timezone's working day is closer to three. A service driven by cron jobs on the hour is far worse than either, because its traffic is not spread over hours at all — it arrives in the first few seconds of each minute, and the relevant multiple is the ratio of a minute to those seconds.

Which means: derive the multiple from your own request-count metric at its finest available resolution, and be suspicious of a one-minute average if your clients share a clock. A pool sized against a one-minute average when the traffic arrives in five-second bursts is undersized by whatever that ratio is.

Worth remembering

• Requests in flight equals arrival rate times time in the system. It is an identity in steady state, not a model, and it assumes nothing about the arrival pattern.
• A pool of 16 slots held for 50ms each is a 320-a-second service, and that ceiling appears in no configuration file.
• Hold time is the divisor, so a two-fold latency rise roughly halves the throughput ceiling without anything in your configuration changing.
• Size a pool as peak arrival rate times p95 hold time, then multiply by at least 1.5 because arrivals do not arrive on average.
• Core count sizes compute; slot count sizes residency. They coincide only when requests are on-CPU for their whole life.
• Size inwards: the database's concurrency limit, then the connection pool, then the worker slots. The smallest pool is the real limit.
• A pool with no acquisition timeout is an unbounded queue, which is a slower outage. Set the timeout below the caller's own.
• The law needs completions to keep pace with arrivals. During an overload it does not apply, and the number you need then is a shedding threshold.