Part V · Chapter 9 · On Call · lesson 3
Quick for eight people at once and hopeless for twelve
14 min
Builds on How many requests can be inside your service at once and One slow service took down four that do not depend on it
Part V · Chapter 9 · On Call · lesson 3
14 min
Builds on How many requests can be inside your service at once and One slow service took down four that do not depend on it
14:20. Every endpoint got slower at once, including the ones that read a single row and return it. The store behind all of them reports CPU at 31 per cent, normal query times, nothing queueing. Nothing was deployed today. The one clean measurement anybody has: on this build, eight concurrent requests are comfortable and twelve are hopeless.
Predict
"Comfortable at eight, hopeless at twelve" is not a severity report. It is the most informative sentence in the incident, because a response time that falls off a cliff at a particular concurrency is telling you there is something countable in the path and you have run out of it.
A resource that is proportional — CPU, bandwidth, memory throughput — degrades along a curve. You feel it at seventy per cent of capacity and you feel it more at eighty. A resource that is countable does nothing at all until it is gone, and then the wait is set by the queue rather than by the work. Below the width, waiting time is near zero; above it, waiting time grows with how long you have been above it rather than with how far above it you are, which is why the hour feels like the system is getting worse while nothing about the load changes.
The countable things in a request path, all of which produce this same cliff:
| What runs out | Where the time then appears |
|---|---|
| A connection pool | Before the query, inside whatever the client library calls acquire. |
| A request thread pool, or a worker pool | Before the handler runs at all — the request is accepted and parked. |
| An in-flight or concurrency limit | At admission, usually with a visible rejection rather than a wait. |
| Row or table locks at the store | Inside the query, with the holder visible on the store's own side. |
| File descriptors, ephemeral ports, a semaphore | At connection setup, often as an error rather than a delay. |
Every one of them looks identical from the outside, because waiting looks identical no matter what is being waited for. So the diagnosis is never "is something saturated" — the cliff already answered that — it is which queue, and the only reading that separates them is one that attributes the time to a layer.
Here is the structural reason this class of incident reliably gets misdiagnosed as a store problem: time spent waiting to enter a queue belongs to no component. The store has a dashboard. The handler has a dashboard. The gap between them, where a request sits holding nothing and doing nothing, is owned by a client library and is graphed by almost nobody. The component on the far side of the queue is the only thing in the neighbourhood with a chart, so it gets the blame by default.
Six queries against a shelf of seven readings that costs nine. Two of the panels on it eliminate the same pair of hypotheses as one of the expensive ones, which looks like a saving and is not.
Incident
You are the one who got paged
Query budget
0 of 6 spent
0 of 7 readings open. The shelf costs more than you have, so some of it stays shut.
Commit
Two things at once, and in this order for a reason: there is no list of causes to pick from yet. Say what you think happened in your own words, and predict one number you were never allowed to query. If the cause is right, the number follows from what you have already read.
What is the mean time a request spends waiting to acquire a connection from the pool?
Write the cause out first — a sentence you would say on the call.
The arithmetic matters as much as the eliminations, and on this shelf they are not sold together.
Two cheap readings from the store's own side kill two hypotheses for one query each. The sampled trace kills the same two for two. Measured in eliminations per unit of cost they are identical, and if that were all a reading did, the cheap pair would be the obvious buy.
It is not all a reading does. The trace is also where the number comes from: the held-back value is derivable from the trace plus two one-cost panels, and the two store dashboards contribute nothing to it. Take the cheap route and you need the trace anyway, which puts you two over budget — so the cheap route closes the incident and leaves the ruling unanswerable.
Rank readings by eliminations plus arithmetic per unit of cost, not by eliminations alone. A panel that kills one hypothesis and supplies a term in the sum is worth more than a panel that kills two and supplies nothing. That is the only reason the expensive reading was the economical one here.
Two further notes on the shape of this shelf, both general:
First, instrument the wait. Every mature pool exposes it and almost nobody alerts on it: HikariCP publishes connection acquisition timing, Go's database/sql exposes WaitCount and WaitDuration on DBStats, pgbouncer reports client waiting counts in SHOW POOLS, and most HTTP client libraries expose a queue depth for their connection manager. One panel per pool, graphing time spent waiting to get in, is the difference between this incident taking four minutes and taking four hours.
Second, bulkhead the expensive work. Give the long-running endpoint its own small pool so its queue is not the queue a one-row read stands in. This is the containment that actually holds, and it has a real cost: separate pools raise the total number of connections you hold against the store, and the store's connection count is itself a fixed resource with memory attached to each entry. A connection pooler in transaction mode — pgbouncer being the usual one for Postgres — lets you oversubscribe client-side pools against a smaller server-side one, at the price of losing session state, prepared statements across transactions, and some advisory-lock patterns.
Third, bound the hold time rather than the pool. An analytical query with no row limit and no statement timeout can hold a connection for as long as the planner feels like. A statement timeout on that endpoint's role converts an unbounded hold into a failed request, which is a much better failure: one endpoint returns an error instead of every endpoint getting slower.
What people reach for first, and should not, is the pool size. Doubling the width buys time proportional to the overshoot and moves the queue to the store, which has its own width and far less ability to shed. Widening a pool in an incident is a reasonable stabilisation and a terrible fix, and the giveaway that it was the wrong move is that it has to be done twice.
Check
All of it assumes the pool is narrower than peak occupancy. When the pool is larger than peak concurrency, the wait is near zero at every moment and this entire shape is unavailable: a slow endpoint's cost stays with the slow endpoint, every other endpoint is unaffected, and reading the symptom as a queueing problem will send you looking for a cliff that is not there.
The test takes a minute and is worth doing before the incident rather than during it. For each class of request, multiply peak arrival rate by mean hold time, sum the classes, and compare the total with the width. If the width exceeds the sum with real margin — a factor of two, not ten per cent — stop looking for a pool and go find the component that got slower, because the queue cannot be where the time went.
The reason "make the pool bigger than peak concurrency" is not a universal fix is that it relocates the width rather than removing it. Every client-side connection corresponds to a server-side one, and the store has a fixed ceiling with memory attached to each entry, no ability to shed politely, and a planner whose behaviour changes when hundreds of sessions are active at once. Wide pools are safe exactly to the extent that the far side can absorb them, which is the calculation a pooler in transaction mode exists to change.