Part V · Chapter 9 · On Call · lesson 4

It recovered, failed again, recovered — on a four-minute rhythm

14 min

Builds on The retries turned a blip into a two-hour outage

09:14, a network blip lasting thirty-one seconds. Pricing failed, the alert fired, the blip cleared, the alert cleared. Then it fired again. It is now 09:36 and pricing is shedding about one request in seven with a p99 near two seconds. Nothing about the blip is still happening. Three instances, all healthy, autoscaling off. The team is waiting for it to clear on its own, because it did once.

Predict

Twenty minutes after the trigger ended, the symptom has not. Which single number would tell you most about why, and is almost certainly not on a dashboard?

The pricing service's p99 and shed rateThe rate of requests arriving at pricing, next to the rate its caller believes it is sendingHealthy instance count and per-instance goodputThe latency of the service pricing itself depends on

Find the clock, including the one with no interval

A failure with a rhythm is a clock, and the interval is a fingerprint. Match the period against the things in your system that have one:

Period you measure	Clocks that have it
Seconds, very regular	A health-check interval, a token-bucket refill, a retry delay, a connection keepalive.
Tens of seconds to minutes	A lease or lock renewal, a leader-election timeout, a metrics push interval, a cache entry lifetime.
Exactly on the minute or hour	A scheduled job, a batch window, a rotation, a certificate or token refresh.
Irregular but load-correlated	A garbage collector, a compaction, a flush — not a clock at all, but easily mistaken for one.

Measure the period before you theorise, because the period alone eliminates most of that table. The method is mechanical: find the time between two recoveries, then grep your configuration for that number.

This incident is the degenerate case of the same mechanism and it is worth seeing in that light. A failure that re-arrives on a four-minute rhythm has a clock with a four-minute interval. A failure that re-arrives immediately has a clock with an interval of zero, so there is no oscillation to observe: the system never drains, never recovers, and the outage looks permanent rather than periodic. Same cause, same arithmetic, and the version with no gap is the one that will still be happening in an hour.

Why a system can fail to return on its own.

Capacity is usually drawn as a horizontal line: a box serves so many requests a second, and load either fits under it or does not. For most real services the line slopes down instead. As queues lengthen, work expires before it is delivered; callers give up and re-ask; cache hit rates fall; scheduling costs rise. Goodput — useful work completed — falls as offered load rises.

A sloping capacity line permits two stable states at the same traffic: one at low load with high goodput, and one at high load with low goodput. A brief trigger can move a system from the first to the second. Removing the trigger does not move it back, because the load holding it there is now generated by the failure itself. The exit is always external: shed load, break the amplifier, or drain.

Six queries against a shelf of nine readings that costs eleven — the tightest ratio in this track. Three pairs of those panels say the same thing as each other, and one of the three says it at two different prices.

Incident

You are the one who got paged

It has not come back on its own

The system: An order service calls a pricing service once per order. Three pricing instances sit behind one internal gateway; autoscaling is off. Both services are owned by the same team.
What you were told: A thirty-second network blip at 09:14 made pricing fail. The blip ended. Twenty minutes later pricing is still shedding about one request in seven and its p99 is sitting near two seconds. Nothing about the blip is still happening.

Query budget

0 of 6 spent

0 of 9 readings open. The shelf costs more than you have, so some of it stays shut.

Commit

Two things at once, and in this order for a reason: there is no list of causes to pick from yet. Say what you think happened in your own words, and predict one number you were never allowed to query. If the cause is right, the number follows from what you have already read.

The cause, in your own words

Your prediction

How many requests a second are arriving at the pricing service right now?

a number

Write the cause out first — a sentence you would say on the call.

Which query to spend first

Four hypotheses had to die and each one had an elimination available. Two of them had two, at the same price; one of them had two, at different prices. That last one is the whole budget.

The messenger story — our dependency broke — can be killed by the dependency's own dashboard for one, or by a sampled trace for two. Both work. The trace feels more rigorous, because it shows you where the time went rather than asserting that somewhere else is healthy, and that feeling costs exactly one query. The cheapest closing set here leaves nothing spare: one elimination from each pair, the decisive configuration read at two, and the shed-rate panel at one, which is the budget to the last unit. Buy the trace instead of the dashboard and the panel you drop is the shed rate — and without the shed rate you cannot get from what the service is serving to what is arriving, which is the only arithmetic that answers the question you are being graded on.

This is the opposite call from the previous incident, under the same rule. There the expensive trace was the economical purchase because it carried a term in the sum as well as two eliminations. Here it carries one elimination that a cheaper panel also carries, and nothing else. Price the reading against what it eliminates and what it contributes to the arithmetic, every time, rather than against how authoritative it looks.

Two more properties of this shelf worth naming:

The shed rate eliminates nothing and is indispensable. "Eliminates nothing" and "not worth buying" are different judgements, and conflating them is how a learner closes the incident and then cannot answer the ruling.
The decisive reading was a configuration file that had not changed in fourteen months. There is a strong bias during an incident towards readings that are recent — graphs, logs, deploys — and against readings that are static, because static things feel like they cannot be the cause of something that started at a specific time. A standing multiplier plus a trigger is one of the most common incident shapes there is, and the multiplier is always old.

What people reach for

The reflex is to raise capacity, and it does work: enough instances and the high-load state stops existing. It is also the expensive answer to a problem whose cheap answer is upstream, and it has the property that you cannot buy your way out of a multiplier — a client configured to make every failure four times larger will find the new ceiling too.

What is actually reached for, in rough order of leverage:

A retry budget rather than a retry count. Cap retries at a small share of total requests — a token bucket that refills from successes — so a broad failure cannot multiply load at all. A count-based policy multiplies hardest precisely when the most things are failing, which is the worst possible time. This exists off the shelf: gRPC's retryThrottling, Envoy's retry budgets, Finagle's budgets, and the adaptive retry mode in the AWS SDKs. The tradeoff is real and worth stating: a budget converts some recoverable failures into errors the caller sees, which is a deliberate choice to fail a few requests rather than all of them.

Exponential backoff with full jitter. Backoff spreads the retries in time; jitter breaks the synchronisation that makes them arrive as a wave. Neither bounds the total, which is why a budget is the first item on this list and not the second.

A circuit breaker on the caller. Stop asking after a threshold of failures, and probe occasionally. This turns the amplifier off at the source. The failure mode is a breaker that opens against a dependency that was healthy, and the tuning is genuinely hard — a breaker is a load-shedding decision made with less information than the server has.

Priority-aware shedding on the server. If the server must refuse work, it should refuse the cheapest-to-refuse work and should refuse it fast, because a rejection that takes two seconds is still a request you paid for. Deadline propagation helps here: work whose deadline has passed should be dropped rather than served.

Idempotency, before any of the above. A retry is only safe if the operation can be repeated, which for anything that writes means an idempotency key the server deduplicates on. Retries on a non-idempotent write do not cause a storm; they cause duplicate orders, and that is a worse incident.

Check

Callers submit 120 requests a second, one call each. The client attempts up to four times with no delay and stops on success. During a failure, each individual attempt fails with probability one half, independently. What arrival rate does the service see?

About 240 a second — half fail, so the load doublesAbout 225 a secondAbout 480 a second — four attempts means four times the loadAbout 180 a second — 120 plus half again

Worth remembering

A periodic failure is a clock. Measure the period first, then search your configuration for that interval, before theorising about mechanisms.
An immediate retry is a clock with an interval of zero, which is why the storm version never oscillates and never drains.
Goodput can fall as offered load rises, which permits two stable states at the same traffic — and a trigger that moves you to the bad one does not have to persist.
A system that will not recover after the trigger ended is generating its own load; the exit is external, by shedding, breaking the amplifier, or draining.
Served, arrived and submitted are three different numbers, and self-inflicted load is only visible in the gap between the last two.
When two readings eliminate the same cause at different prices, the price difference is the entire decision — and the expensive one usually looks more rigorous.
A reading that eliminates nothing can still be indispensable, because the ruling is a number and the number comes from somewhere.
Cap retries by budget rather than by count: a count multiplies hardest when the most things are failing.
A retry is only safe on an operation that can be repeated, so idempotency comes before any retry policy.

Where this stops holding

The fixed point needs the next wave to arrive before the system has drained. That is a relationship between two intervals, and reversing it changes the incident completely.

Once the retry interval exceeds the recovery time, the storm cannot sustain itself. Each wave lands, overloads the service, is partly shed, and the queue drains before the following wave arrives — so the system recovers between waves and the symptom becomes the rhythm in this lesson's title rather than a permanent state. That version is strictly better: it is self-limiting, it does not require anybody to intervene to end it, and it is diagnosed by measuring the period rather than by reasoning about goodput at all.

It also means the fix is different. For the oscillating version, attempt count and jitter are the levers, and shedding is a distraction because the system is already draining. For the sustained version, the attempt count is what created the load and shedding is what buys you the room to turn it off. The question that decides which one you are in is the first thing to ask about any self-inflicted overload: how long does this thing take to drain, and how long until the callers ask again? If the second number is smaller, you have a state and not a cycle, and waiting is not a strategy.