Part III · Chapter 3 · Under Load · lesson 7

Find the ceiling in a design you have never seen

15 min

Builds on Why it gets slow at 70% busy, not at 100% and How many requests can be inside your service at once

Here is a design you did not write, with the ratings its owners published. Your job this afternoon is one number — the requests a second at which it stops — and the name of the box that gives first.

That number appears nowhere in the table below. Four of the seven boxes are irrelevant to it, and the one that decides it is rated in the wrong unit.

The system

                                   ┌──────────────────┐
                                   │  auth-service    │
                                   │  4 pods x 32     │
                                   └────────▲─────────┘
                                            │ HTTP, p50 8ms
                                            │
  client ──▶ ALB ──▶ nginx ──▶ orders-api ──┼──▶ pgbouncer ──▶ Postgres
                     x2        12 pods      │     pool 20       primary
                                x8 workers  │                   8 vCPU
                                            │
                                            └──▶ Redis, 1 primary

Box	What its owners quote	Measured per request
ALB	"tens of thousands of new connections a second"	—
nginx, 2 instances	`worker_processes 4`, `worker_connections 1024`	—
orders-api	12 pods × 8 gunicorn sync workers = 96 slots	45ms end to end
auth-service	4 pods × 32 threads = 128 slots	8ms per call, one call
pgbouncer	`default_pool_size 20`, transaction pooling	—
Postgres primary	8 vCPU, `max_connections 200`	6ms of query time
Redis	single-threaded, 80,000 operations a second from `redis-benchmark`	3 operations

And the one thing the table does not say, which you got by reading the handler:

with db.transaction():                 # pgbouncer hands over a server connection here
    row = db.execute(SELECT_FOR_UPDATE, order_id)   #  3ms
    identity = auth_client.verify(token)            #  8ms  <- an HTTP call
    db.execute(UPDATE_ORDER, row.id)                #  3ms
                                       # COMMIT, WAL flush                6ms

The transaction is open for 20ms. Six of those milliseconds are queries.

Predict

Using Little's law on each box in turn — slots divided by hold time — which box gives first, and at roughly what load?

orders-api worker slots, at about 2,100 a secondthe Postgres primary, at about 1,300 a secondthe pgbouncer pool, at about 1,000 a secondnginx, at about 8,200 concurrent connections

Why the pool was rated in the wrong unit

Every box in that table is described either as a number of slots or as a rate, and only one of those two can be compared across boxes. Sort the ratings.

Not the same thing

A rated limit

A count of things permitted to exist at once. A configuration value, a licence term, or a hardware count.

Measured in: things, concurrently — connections, slots, sessions, cores
For example: pgbouncer's default_pool_size of 20, or Postgres max_connections of 200.
Chase it alone and: Compared directly against a traffic rate, as though 200 connections meant 200 requests a second. They differ by whatever the hold time is, which can be a factor of a thousand either way.

A throughput ceiling

Completions per unit of time. Either measured under load, or derived from a slot count by dividing by the hold time.

Measured in: completions per second
For example: 96 worker slots divided by a 45ms hold, giving 2,133 requests a second.
Chase it alone and: Quoted from a benchmark whose conditions differ from yours — one client, no TLS, pipelined, a warm cache — which inflates it by an amount nobody records.

Which is which? Put each one on a side.

default_pool_size 20
max_connections 200
worker_connections 1024
8 vCPU
96 worker slots at a 45ms hold
80,000 operations a second from redis-benchmark

The conversion between the two columns is always the same division, and the quantity you divide by — hold time — is the one quantity that appears in no vendor's documentation, because it is a property of your code rather than of their product.

The pool, as the only number that matters

Hold the pool at 20 and move the hold time. Everything about this design's ceiling is on this one chart.

Little's law

Hypothetical model, not a forecast

Ties arrival rate, latency and concurrency together, solving for whichever of the three is left implied.

It assumes

The pool is in steady state, with connections returned as fast as they are taken.
Hold time is the whole open transaction, not the time its queries take.
Every pooled connection is interchangeable and none is reserved for anything else.
Nothing downstream of the pool saturates before the pool itself does.

It ignores

Warm-up and shutdown periods, where the law simply does not apply
The cost of concurrency itself — memory, context switches, lock contention
Whether the implied concurrency is physically achievable on the hardware
Variance: this is an identity about averages and says nothing about tails

Today's peak is 400 a second. Keep the pool at 20 and raise the hold time until this design can no longer carry that peak. Write down the hold time you landed on — that is the number this architecture actually depends on.

Latency20 ms

Average time one request spends in the system, entry to exit.

Requests in flight20

Implied arrival rate: 1k/s
Which quantity is implied: 0 concurrency, 1 arrivals, 2 latency: 1
Arrivals: 1k/s
Latency: 20 ms
Requests in flight: 20
Arrival rate this concurrency sustains: 1k/s

Connections demanded
Connections in the pool

A rising straight line of demanded connections crossing a flat line at the pool size; left of the crossing the pool covers demand, right of it every request queues for a connection.
Arrivals a second	Connections demanded	Connections in the pool
100	2	20
200	4	20
300	6	20
400	8	20
500	10	20
600	12	20
700	14	20
800	16	20
900	18	20
1000	20	20
1100	22	20
1200	24	20
1300	26	20
1400	28	20
1500	30	20
1600	32	20
1700	34	20
1800	36	20
1900	38	20
2000	40	20

Not there yet — keep moving the controls.

50ms. The design carries today's peak as long as one transaction stays open for less than 50ms, and it is currently open for 20. The margin is 30ms, and 8 of the current 20 belong to a different team's service.

So the honest statement of this architecture's limit is not a request rate at all. It is: this design fails at today's peak the moment the auth service's latency reaches about 38ms — 50 minus the 12ms of database work inside the transaction. Not its p99. Its ordinary latency, because every single request pays it.

The number I would ship, and the haircut on it

1,000 a second is the arithmetic ceiling. It is not the number to put in a capacity plan, and the size of the haircut is worth deriving rather than guessing, because the usual rule of thumb is wrong here in the generous direction.

A pool of twenty pools reasonably well. Twenty servers sharing one queue at 85% utilisation make only 39% of requests wait at all, and the wait they add is about 13% of one hold time. At 95% it is 76% of requests and about three quarters of a hold time. So the queueing argument from the first lesson — which is brutal for a single server — would happily let you run this pool at 85%, or 850 a second.

The reason I would still plan at 700 is not queueing. It is that the 20ms hold is built out of medians, and one of its terms is another team's median. Auth's p95 is not 8ms; a pool is held for the actual duration of each request, not the typical one, and a hold-time distribution with a long right tail consumes connections at a rate the median figure understates.

So: plan for 700 a second, treat 1,000 as the cliff, and alert on pgbouncer's cl_waiting being non-zero for a minute rather than on the request rate. The rate is a proxy; the waiter count is the thing itself, and it will fire for causes — a slow auth deploy, a longer transaction — that no rate threshold anticipates.

Three changes, in the order I would make them:

Move auth_client.verify outside the transaction. Hold time goes from 20ms to 12ms, the ceiling from 1,000 to 1,667, and the dependency on another team's latency leaves the critical section entirely. One function call moved up four lines. Do this before touching any configuration.
Raise default_pool_size to about 24, and no further. With a 12ms hold that is 2,000 a second, which puts the pool just behind Postgres's own 1,333. Going past that moves the queue into the database, where pool metrics cannot see it.
Set a pool acquisition timeout below the caller's timeout, so a saturated pool sheds instead of queueing. Without it, step two has only moved the point at which an unbounded queue forms.

After all three, the binding constraint is the Postgres primary at around 1,333 a second, which is a box you can reason about and buy more of. That is what success looks like in this exercise: not a higher number, but a constraint that sits somewhere legible.

Check

The owners double the orders-api to 24 pods — 192 worker slots — and change nothing else. What happens to throughput?

It roughly doubles, since the worker slots doubledThroughput is unchanged and latency gets worseThroughput is unchanged and latency is unchangedThroughput falls, because more pods means more connection churn at pgbouncer

Where this stops being true

Every number above came from a rating somebody else published, and published ratings are measured by people who want them to be large.

redis-benchmark pipelines by default. The 80,000 figure is for batched commands on loopback. Un-pipelined, across a network, with TLS, it can be several times lower. The rating was never wrong; it was answering a different question.
The ALB's connection rate is for new connections, not for requests on established keep-alive connections, and the two differ by whatever your keep-alive ratio is.
max_connections 200 is not a concurrency rating. It bounds sessions so the server does not exhaust memory. The number of statements the box can actually execute at once is nearer its core count, which is where the 1,333 figure came from and which is itself an estimate.
The 45ms and 8ms figures are medians. A pool is held for the duration of the actual request, so pool sizing wants a high percentile, and the p95 hold time in a design like this is routinely twice the median.

Which means the correct output of a teardown is not a number but a number with its sensitivity attached: 1,000 a second, dominated by a 20ms transaction hold on a pool of 20; falls below today's peak if the hold reaches 50ms; the figure I am least sure of is the hold time, because it is a median and the pool cares about the tail.

Say that last clause out loud whenever you hand a capacity number to anybody. The number is a claim; the sentence after it is what makes the claim auditable.

Worth remembering

Convert every box to the same unit before comparing: slots divided by hold time gives completions a second, and a slot count alone gives nothing.
In the worked design the ceiling is 1,000 a second and pgbouncer's pool of 20 gives first, ahead of the Postgres primary at 1,333.
The pool was sized against 6ms of query time and is held for 20ms, because an HTTP call to another service sits inside the transaction.
A capacity limit is often not a request rate: this design fails at today's peak once the auth service's ordinary latency reaches about 38ms.
A pool of twenty tolerates 85% utilisation with only a 13% latency penalty; the reason to plan at 70% here is the hold-time tail, not the queueing.
Alert on the pool's waiter count rather than on the request rate: the rate is a proxy and the waiter count is the thing itself.
Move the dependency call out of the transaction before changing any configuration value; it is one line and worth a 67% ceiling increase here.
Raising a pool past the next constraint moves the queue somewhere your pool metrics cannot see it.
Adding capacity upstream of the binding constraint adds queue, not throughput — the same work finished with more work resident.
Published ratings answer the vendor's question: redis-benchmark pipelines, an ALB rate counts new connections, and max_connections protects memory rather than describing concurrency.
Hand over a capacity number with its dominant term and its sensitivity attached, or it cannot be audited by the next person.