The retries turned a blip into a two-hour outage

At 14:02 a database failover starts dropping about one request in twenty. It finishes at 14:03:30. At 14:04 the API is failing on a fifth of requests, and it stays there until 16:10, when somebody turns retries off in the mobile client.

Nothing was broken after 14:03:30 except the traffic the system was making about itself.

Three layers on this path retry: the mobile SDK, the gateway, the service. Three attempts each.

Predict

One tap in the app. Three attempts at each of three layers, nothing succeeding. How many requests reach the database?

Choose one answer

Three — only the innermost layer talks to the databaseNine, because three layers of three is three squaredTwenty-sevenNine, because the mobile SDK gives up once the gateway starts failing

Lock in answer

The part that makes it last two hours

Multiplication is only half of it. The other half is that the extra load raises the failure rate that produced it, which produces more retries.

The model below walks one tier of that loop. A thousand requests a second arrive, the system can serve twelve hundred attempts a second before it starts shedding, and some share of requests fails for its own reasons. Each round of retries is the previous round's failures coming back as new load, priced at the failure rate that all the accumulated load implies.

Watch the failure rate the system reports against the fault that caused it.

Retry amplification

≈ Hypothetical model, not a forecast

Follows retries round by round as failed requests come back as new load, raising the failure rate that caused them.

It assumes

• Every failed attempt is retried immediately, up to the configured attempt count.
• Capacity is a hard ceiling of attempts per second, and load above it is shed proportionally.
• A request fails either from the underlying fault or from being shed under overload.
• With backoff on, each retry round is spread over roughly twice the window of the previous one.
• The failure rate is the same for every request, with no hot key or bad replica.

It ignores

• Jitter, retry budgets, hedging and circuit breakers, all of which damp this
• Queueing delay, which turns shed load into slow load before it becomes errors
• That real capacity often falls as load rises, which makes this worse
• Clients that give up on their own, which is the only thing that ends it
• That the rounds here are attempts, not distinct requests from distinct users

Leave the attempt count alone — you are not allowed to ask three teams to change their configuration. Find the single setting that moves the underlying failure rate at which amplified load reaches capacity from 17.5 per cent up to 37 per cent.

Underlying failure rate20%

The share that fails for its own reasons, with the system not overloaded.

Retries per request2

Attempts after the first one. Zero means a failure is simply a failure.

Backoff on0

One means each retry round is spread over a longer window than the last.

Load the system actually sees

1.2k/s

Times more load than arrived

1.24

Failure rate once retries are counted

0.226

Share of attempts the system can take

0.968

Attempts served successfully

960/s

Capacity left over

-40/s

Amplified load exceeds capacity

1

Failure rate at which load reaches capacity

17.5%

A tipping point exists at all

1

• Load the system is seeing
• Capacity

◷ Not there yet — keep moving the controls.

Two figures out of that. A fault that fails one request in five arrives, and the system reports failing nearly 23 per cent — it manufactures two and a half percentage points of failure that no fault caused, because the retries pushed offered load to 1,240 attempts a second against a ceiling of 1,200. Turn the retries off entirely and the reported rate is exactly the 20 per cent that broke, with 200 attempts a second of headroom to spare.

And the tipping point. With these attempt counts and no backoff, amplified load reaches capacity once the underlying failure rate passes 17.5 per cent. Below that the retries are invisible and genuinely useful — they recover real work, and goodput with retries beats goodput without them. Above it they are the outage. There is no gradual region: the same configuration is a feature on one side of that number and the cause on the other.

Three fixes, and they do not do the same job

These three get named together and conflated constantly. They defend against different things, and two of them do not bound amplification at all.

Exponential backoff spreads the retries out in time. Round two lands after roughly twice the delay of round one, round three after twice that. What it prevents is a single client hammering a dependency while it is trying to recover. What it does not prevent is the total count: twenty-seven attempts spread over thirty seconds is still twenty-seven attempts. In the model, turning backoff on moves the tipping point from 17.5 per cent to 37 per cent — a real and large gain, and entirely about how many retries share a window rather than how many exist.

Jitter randomises each delay. What it prevents is synchronisation: ten thousand clients that all failed at the same instant, all backing off by the same multiplier, all returning in the same hundred-millisecond bucket, forever, because every round re-synchronises them. Jitter reduces the total number of retries by exactly zero. It converts a series of spikes into a flat elevated rate, which a system under recovery can survive and a spike cannot. Full jitter — sleep a uniform random time between zero and the backoff ceiling, rather than the ceiling plus a wobble — is the version worth copying, and it is what tenacity's wait_random_exponential, Polly's DecorrelatedJitterBackoffV2 and resilience4j's IntervalFunction.ofExponentialRandomBackoff all implement.

Retry budgets are the only one of the three that bounds amplification. A budget caps retries as a share of successful traffic: Envoy's retry_budget allows retries up to a percentage of active requests to that cluster, gRPC's retryThrottling is a per-server token bucket that a failed retry debits and a success refills, and the AWS SDKs' standard and adaptive retry modes keep a client-side retry quota that a retry debits and a timeout debits harder. When the bucket empties, the retry is not attempted at all and the error goes straight back to the caller. That is the behaviour you want during an outage and it is the one people disable first, because during normal operation it does nothing visible.

If you are adding one thing, add the budget. Backoff and jitter make a recovering dependency recoverable; only a budget stops the retry traffic from being the thing it has to recover from.

Check

Your service calls a payment provider. You have exponential backoff with full jitter, three attempts, and no budget. The provider has a thirty-minute degradation at a 40 per cent error rate. What happens?

Choose one answer

Backoff and jitter hold the retry load flat enough that the provider recovers normallyYour retry traffic stays roughly proportional to the provider's failure rate for the whole thirty minutesThe provider's own rate limiter protects them, so the amplification is their problemThree attempts is low enough that the amplification does not matter

Check answer

›Where to put the retry, if you are only allowed oneOptional

The cheapest fix for a twenty-seven is not better retries at every layer. It is retrying at one layer.

Retry at the innermost layer and you retry fast, against a dependency you can see the health of, and you consume the caller's remaining deadline rather than starting a new one. Retry at the outermost layer and you retry a whole operation, which is more likely to succeed on a different path, and far more expensive.

The middle layers are the ones to strip. A proxy that retries is multiplying a decision somebody else already made, with no information either of them has. Envoy's x-envoy-max-retries and gRPC's per-method retryPolicy can both be set to one attempt at the hop, which is often the single highest-value change available in this whole lesson.

The other half is deadlines. If the caller passes a deadline down and every hop honours it, a retry that cannot finish inside the remaining time is not attempted. That bounds the time the twenty-seven can consume even when it does not bound the count — and a retry nobody is waiting for is pure amplification.

Worth remembering

• Retries compose by multiplication, with the number of retrying layers as the exponent: three attempts at each of three layers is twenty-seven requests, and no configuration file contains that number.
• An error returned upward is indistinguishable from a single failed request, so no layer can see or account for the retries below it.
• Amplified load raises the failure rate that caused it: a 20 per cent fault was reported as nearly 23 per cent once retries pushed offered load past capacity.
• The tipping point is sharp. With these settings, retries are useful below a 17.5 per cent underlying failure rate and are the outage above it.
• Exponential backoff prevents one client hammering a recovering dependency. It does not reduce the total number of retries.
• Jitter prevents synchronised retry waves and reduces the retry count by nothing at all.
• Only a retry budget bounds amplification, by capping retries as a share of successful traffic and refusing them when the bucket empties.
• Stripping retries from middle layers, and propagating deadlines so a hopeless retry is never attempted, are usually cheaper than tuning any of the three.

Where this stops holding

Everything above assumes each layer owns its own retry allowance. Once the allowance travels with the request — a retry count or a remaining-attempts token in the request metadata, decremented at every hop, with hops refusing to retry when it reaches zero — the product collapses to a single budget and the exponent disappears. The multiplier becomes whatever that one number says, three layers or thirty.

That is the design worth having, and it is also why this lesson's arithmetic stops describing a system that has it. If you can see a per-request retry budget in the request headers, stop reasoning about products and go read the budget.