Your median is 20ms and your p99 is 2 seconds

The search endpoint has a median of 20ms and a p99 of 2 seconds. Every one of the twenty shards it queries reports a healthy p99 of 100ms. Nobody can find the slow shard, because there isn't one: the shards take turns.

Two separate things are happening here, and the usual explanation covers only one of them. Commit to a number before you read on.

Predict

One request fans out to 20 shards and waits for all 20. Each shard independently has a p99 of 100ms. What is the user-visible p99?

Choose one answer

Still about 100ms — the slowest call is 100ms, so the request is 100msAbout 2 seconds — twenty calls times a 100ms tailAbout 165msAbout 500ms — somewhere between the single-call tail and the sum

Lock in answer

Both curves, on one chart

Slide the fan-out and watch the two lines. The latency line bends over. The other one does not.

Tail amplification

≈ Hypothetical model, not a forecast

What a fan-out does to a tail latency when one user request waits on many backend calls at once.

It assumes

• The request waits for every call, so the slowest one sets the total time.
• Calls are independent and drawn from one identical latency distribution.
• The latency tail is exponential, calibrated to pass through the stated percentile.
• No call is retried, hedged, cancelled early or answered from a local cache.

It ignores

• Hedged and speculative requests, which are the usual fix for exactly this
• Correlated slowness — one shared dependency makes calls slow together, which is worse than independence
• Partial results, where a page renders without its slowest widget
• The fan-out's own cost in connections, memory and scheduler pressure

Raise the fan-out until it is an even bet that some call in the request lands in its own slowest 1% — the point where half of all requests contain a slow call.

Backend calls per request20

Per-call tail latency100 ms

The latency the stated percentile of single calls comes in under.

Per-call tail latency

100 ms

Tail latency the user actually sees

165 ms

User-visible tail as a multiple of the per-call tail

1.65

Chance at least one call is slow

0.182

Slow calls per request, on average

0.2

Fan-out at which a slow call becomes an even bet

69

• Tail the user sees
• Tail of one call

◷ Not there yet — keep moving the controls.

69 calls. Which is a routine fan-out — a feed page assembling items, a permissions check per object, a dashboard with that many tiles. At that point the median request contains a slow call, and the per-shard p99 is still a perfectly honest 100ms.

So where did the 2 seconds come from?

Not from here. A hundredfold gap between median and p99 needs a mechanism this model explicitly leaves out, and there are two candidates worth checking in order.

Queueing, from the first lesson in this track. Response time is service time over the idle share, and at 99% utilisation that multiplier is 100. A gap of exactly the size in the opening paragraph is what a saturated queue produces, and it produces it on the requests unlucky enough to arrive while the queue is deep. Check utilisation and queue depth at the moment of the slow requests, not on average.

Correlated slowness, which is strictly worse than independence. The model assumes twenty independent draws. If all twenty shards share a connection pool, a node, a disk, or a garbage collector, then they are slow together — and the maximum of twenty correlated draws is just one draw, except it happens to every request at once. This is the failure mode where a per-shard p99 stays beautiful and the user-visible p99 collapses, because the slow periods line up.

The diagnostic that separates them: take the slow requests and ask whether their slow calls were spread across shards or concentrated in time. Spread across shards and spread in time is the independent case in this model. Concentrated in time is a shared resource, and the shared resource is what you fix.

The two patterns that bound it

Hedged requests. Send the call, and if it has not answered by roughly the p95, send a second copy to a different replica and take whichever returns first. This is the single most effective fix for an independent tail, because it converts "wait for the slowest" into "wait for the faster of two attempts", and it only costs extra load on the small fraction of calls that were going to be slow anyway. Envoy exposes it as hedge_on_per_try_timeout; gRPC has a hedgingPolicy in its service config.

The tradeoff is the thing that makes hedging dangerous: it adds load precisely when the system is already struggling, and a correlated slowdown makes every call eligible at once. So hedge only behind a budget — Envoy's retry_budget caps retries and hedges as a percentage of active requests, and gRPC's retry throttling does the same. Hedging without a cap is a retry storm with a respectable name.

Don't wait for all of them. The other bound, and the one that needs a product decision rather than a configuration change. Set a per-call deadline well inside the request's own budget, render what came back, and mark the rest as missing. A feed with nineteen of twenty items is a feed; a feed that waits is a spinner. Google's own write-up of this calls it returning "good enough" results, and the hard part is never the code — it is agreeing which parts of the response are optional.

What I would actually reach for, given one afternoon: per-call deadlines first. They are a one-line change per call site, they bound the damage immediately, and they cost no extra load. Hedging second, behind a budget, for the calls that are genuinely idempotent reads. And then the thing that beats both, which is reducing the fan-out — batching twenty per-object permission checks into one call takes the amplification from 1.65 to 1 and removes the frequency problem entirely.

Check

You cut the fan-out from 100 calls to 10 by batching. The per-call p99 stays at 100ms. What improves most?

Choose one answer

The user-visible p99, which falls by roughly a factor of tenThe share of requests containing a slow call, which falls from about 63% to about 10%Nothing much, because the batched call still does the same total workThe median, because fewer calls means less overhead per request

Check answer

Where this stops being true

Everything above assumes the request needs every answer. Flip that and the arithmetic flips with it.

If the request needs only the first answer — a read from any of three replicas, a lookup against any cache node that has it — then you are taking the minimum of n draws rather than the maximum, and more calls make the tail better. The chance that all three replicas are simultaneously slow is 0.01 cubed, which is a millionth. This is why quorum reads and replica fan-out improve tail latency while shard fan-out destroys it, and the two look identical on an architecture diagram.

The question to ask of every fan-out on your diagram is therefore not "how wide is it" but "how many of these answers does the request need". All of them is the case in this lesson. Any one of them is the opposite case, and it is free insurance. Some of them — a quorum — sits in between and is usually much closer to the good end.

›Why the tail grows like the logarithmOptional

Take a distribution whose tail falls off exponentially: the chance of exceeding t is exp(−t/θ). The chance that all n independent calls come in under L is then (1 − exp(−L/θ)) to the power n, and setting that equal to q and solving gives L = −θ × ln(1 − q^(1/n)).

For large n, q^(1/n) approaches 1, and the expression behaves like θ × ln(n) plus a constant. Which is the result: each doubling of the fan-out adds θ × ln 2 to the tail — a fixed amount, not a multiple.

Two consequences worth carrying. First, a fan-out of 1000 is not fifty times worse than a fan-out of 20; it is about twice as bad. Second, the only way to move the whole curve down is to change θ, which is the per-call distribution itself. Reducing the fan-out slides you leftward along a curve that is already flat; reducing per-call variance moves the curve.

That is the quiet argument for spending effort on the slowest shard's behaviour rather than on the fan-out's width — right up until you remember the frequency term, which does scale with n and does not care about θ at all.

Worth remembering

• A request that waits for all of n calls takes the maximum of n latencies, not their sum. Twenty calls at a 100ms per-call p99 give a user-visible p99 near 165ms.
• The tail figure grows like the logarithm of the fan-out: ten times the calls for about a third more latency.
• The frequency of slow requests grows almost linearly. At 20 calls, 18% of requests contain a call from the slowest 1%; at 69 calls it is an even bet.
• A hundredfold gap between median and p99 is not fan-out. It is queueing near saturation, or calls that are slow together because they share a resource.
• Correlated slowness is worse than independence: the maximum of n correlated draws is one draw that happens to every request at once.
• Hedged requests convert waiting for the slowest into waiting for the faster of two, and must sit behind a budget or they are a retry storm.
• Per-call deadlines plus partial results bound the damage with no extra load, and the hard part is agreeing which parts of the response are optional.
• Batching twenty calls into one removes the frequency problem outright, which is the larger of the two available gains.
• If the request needs only the first answer, more calls make the tail better — the minimum of n, not the maximum. The two cases look the same on a diagram.