Part IV · Chapter 6 · Several Services · lesson 5
One slow service took down four that do not depend on it
14 min
Builds on How many requests can be inside your service at once and Your median is 20ms and your p99 is 2 seconds
Part IV · Chapter 6 · Several Services · lesson 5
14 min
Builds on How many requests can be inside your service at once and Your median is 20ms and your p99 is 2 seconds
At 14:10 the search, profile, feed and settings endpoints all began timing out. None of the four calls the recommendation service. The recommendation service had slowed from 9ms to its two-second timeout twelve minutes earlier.
All of them call through the same HTTP client, configured once in a shared library three years ago, with one connection pool of fifty per instance.
Predict
The service tree here is three levels deep and each call makes about three more: the gateway calls three services, each of those calls three, each of those calls three. Two per cent of calls are retried.
Fan-out amplification
Hypothetical model, not a forecast
Traces one external request down a call tree of a given fan-out and depth, and reports the internal load it creates. A period is one level of the tree and period 0 is the request itself.
It assumes
It ignores
Set the retry rate to zero and keep the fan-out at three internal calls per call. Find the depth at which one external request first becomes more than a thousand internal calls.
Averaged over requests, so a fractional value is meaningful.
Level 1 is what the entry point calls; level 2 is what those call.
Applied at every level, so its effect compounds with depth.
| Level of the call tree | Calls at this level | Calls so far |
|---|---|---|
| 0 | 1 | 0 |
| 1 | 3.06 | 3.06 |
| 2 | 9.36 | 12.42 |
| 3 | 28.65 | 41.08 |
Not there yet — keep moving the controls.
One external request is 41 internal calls, and at two hundred requests a second that is 8,215 internal calls a second. The deepest level alone carries 5,731 of them.
Nobody holds that number in their head. The diagram has nine boxes on it and the call count feels like nine, because the fan-out is multiplicative and human estimates of trees are additive. The challenge makes the point harder than the default does: keep a modest three-way fan-out and go six levels deep, and one request becomes 1,092 internal calls. Six levels is not an exotic architecture. It is a gateway, a composition layer, four domain services, and the thing each of those calls.
Little's law, from track 1: work in flight is the arrival rate times the time each unit stays. Apply it to the pool.
Healthy. 8,215 internal calls a second, 9ms each, is about 74 calls in flight across the whole fleet. Ten instances with a pool of fifty is five hundred slots. Seven times the demand, and nobody has ever looked at this number.
One leaf slows. Say the recommendation service takes one in ten of the deepest-level calls, so 573 calls a second. It is now holding each slot for the full two-second timeout rather than 9ms. 573 times two seconds is 1,146 slots demanded — against five hundred that exist.
So the pool is not merely strained, it is permanently full, and it is full of calls to one service. The other nine-tenths of the deepest level — calls to services that are completely healthy — are queueing for a slot behind a dependency that will never answer. Each of them eventually reports a timeout against a healthy service, which is why the first forty minutes of this incident is spent investigating four services that have nothing wrong with them.
The multiplier is the one number to carry out of this: two seconds divided by 9ms is 222. One dependency getting 222 times slower multiplied its pool demand by 222, and the pool was shared with everybody.
People apply these in the order they met them, which is circuit breaker first (it has a library and a blog post), timeouts second, bulkheads never. That order is close to exactly backwards.
First: a bounded wait. Not because it is the best fix — because it is the precondition for the other two. A circuit breaker opens when it counts failures, and a call that never returns is never counted as a failure, so a breaker above an unbounded call never opens. A bulkhead caps how many slots a dependency may hold, and with no timeout each held slot is held forever, so the bulkhead fills and stays full. Set the timeout and the other two become possible; leave it unset and they are decoration. In this incident the timeout existed, at two seconds, which is what made every other number computable.
Second: a bulkhead. This is the fix for the symptom in the title, and it is
the only one of the three that stops failure crossing between dependencies. The
mechanism is a separate budget per destination: a pool, a semaphore, or a cap on
concurrent requests. In Envoy it is the circuit_breakers block on each
upstream cluster — max_connections, max_pending_requests and max_requests
are per cluster, so a saturated recommendation cluster cannot consume the
profile cluster's budget. In Go it is the least glamorous version and works just
as well: a separate http.Client with its own Transport per destination,
rather than one package-level client shared by everything. In the JVM,
Resilience4j's semaphore Bulkhead does it without a thread pool per
dependency.
The cost is real and is the reason people skip it: the sum of the per-dependency pools is larger than the single shared pool was, so you have moved the ceiling from the pool to the process — file descriptors, sockets, memory for the connection objects. Pick the per-dependency caps so their sum is a number you have checked against the process limit, rather than fifty each because fifty was the old default.
Third: a circuit breaker. Last, and it is still worth having. What it adds beyond the first two is speed of failure: with a bulkhead the calls to the sick dependency fail at their timeout, and with a breaker they fail immediately, which stops the sick service receiving load it cannot serve and gives callers their latency back. That is a genuine improvement and it is strictly smaller than the first two, because the blast radius was already contained.
It also has the worst failure mode of the three. A breaker tuned on an aggregate error rate will open on a dependency that is ninety per cent healthy and take the healthy ninety per cent with it, and the half-open probes from many instances at once are themselves a small retry storm. Tune it per endpoint, not per service, and give it a minimum request count before it is allowed to have an opinion.
Check
The dashboard during this incident showed processor utilisation in the low tens of per cent on every instance, which is the single most misleading reading in this failure class and the reason the first hypothesis is always wrong.
Waiting on a socket costs no processor. A thread or a goroutine blocked on a read consumes a slot in whatever bounded resource it occupies and almost nothing else. So a service can be completely unable to serve while every hardware metric reports comfort, and the resource that ran out does not appear on a standard infrastructure dashboard at all.
The metric that would have shown it is pool utilisation — slots in use over slots
configured — per destination. Almost no HTTP client exposes it by default, which
is the practical problem. In Go you can read http.Transport state only
indirectly, so the usual answer is to wrap the round-tripper and count. Envoy
publishes it as a matter of course: upstream_cx_active and
upstream_rq_pending_active per cluster, which is one of the better arguments
for having a proxy in the path even if you use none of its other features.
Track 1's lesson on CPU at forty per cent is the general version of this. Connections, file descriptors, threads and pool slots all saturate before the processor does, and all four are invisible unless somebody instrumented them on purpose.
The arithmetic above predicts a cross-service blast radius, and it predicts it only because the pool is shared. Give every destination its own isolated pool and the prediction stops being true: the slow dependency exhausts its own fifty slots, its own callers queue, and the search, profile, feed and settings endpoints never notice.
At that point this lesson has nothing left to tell you, and the problem has changed shape rather than gone away. What remains is local saturation — the callers of the slow dependency still queue, still time out, and still need somewhere for the excess work to go — and the answer to that is not another bulkhead. It is deciding which requests to refuse, which is track 5's subject and a different kind of decision: a bulkhead is arithmetic about resources, and load shedding is a judgement about which customers to disappoint.
Two things also move with the pools. The sum of the per-destination caps becomes the ceiling, as above. And the amplification number from the model stops being a blast-radius multiplier and becomes a straightforward capacity figure: 8,215 internal calls a second is how much internal traffic you are paying for, whatever the pools look like, and that number is the one the capacity track cares about.