Part V · Chapter 9 · On Call · lesson 2

Everything got slower at once, and nothing was deployed

14 min

Builds on CPU is at 40% and it is still slow

08:10. The page has been open since midnight and the summary at the top of the channel still says everything went wrong at once. What is actually on the dashboard: forty batch workers, a share of them failing every request they attempt, immediately, on every endpoint equally. At 00:05 it was three workers. The count has climbed every hour since. Nothing has been deployed for six days, to either side.

Predict

The number of affected workers has climbed smoothly for eight hours and shows no sign of a ceiling. What does a slope like that tell you, before you open anything?

A rollout is working through the fleet in batchesLoad crossed a threshold and more workers tip over as traffic growsSomething shared broke at one instant, and each client finds out the next time it needs itSomething is leaking — memory, descriptors, connections — and has been since midnight

"Everything at once" is a claim, not an observation

The first sentence written in an incident channel is a compression of what somebody saw in their first thirty seconds, and it is usually wrong in a way that steers the next hour. Three blast-radius shapes get flattened into "everything at once", and they have different causes:

Shape over time	Reading
A step: all clients at the same instant	A change that took effect everywhere simultaneously — a config push, a routing change, a restart of the thing in the middle.
A ramp with no ceiling: more clients each hour	A shared precondition that went bad once, discovered at the rate clients renew whatever carried it.
A step down, then partial recovery	Something restarted, and the clients that reconnected got a working path while the rest kept a broken one — or the exact reverse.

The other word in that first sentence worth auditing is "everything". Here it is load-bearing in a useful way: the failure is uniform across endpoints, and nothing that discriminates by handler can be uniform across handlers. That places it below the layer that knows what an endpoint is — transport, name resolution, admission control, or the client library every call shares. It is one inference, it costs nothing, and it eliminates every hypothesis that lives in business logic.

You have five queries against a shelf of six readings that costs eight. Two of the hypotheses you are about to face can be killed by readings that cost one each, and there is a cheaper way to kill both. Finding it is most of the lesson.

Incident

You are the one who got paged

It is getting worse at a constant rate

The system: An internal API behind a load balancer. Forty batch workers connect to it over TLS and hold long-lived connections, reopening them when a pool entry is retired or a request fails.
What you were told: At 00:00 a few workers started failing every request they attempt. Eight hours later more of them are failing, and the count has climbed steadily all night. The service's own error rate and latency look completely normal.

Query budget

0 of 5 spent

0 of 6 readings open. The shelf costs more than you have, so some of it stays shut.

Commit

Two things at once, and in this order for a reason: there is no list of causes to pick from yet. Say what you think happened in your own words, and predict one number you were never allowed to query. If the cause is right, the number follows from what you have already read.

The cause, in your own words

Your prediction

Of the forty workers, how many are failing every request right now?

a number

Write the cause out first — a sentence you would say on the call.

Which query to spend first

The shelf had four eliminations on it and one of them was on sale.

Two of the panels are the obvious first clicks for any on-call engineer: the change log, to find out what shipped, and the resolution check, to find out whether the clients are even pointed at the right place. Each costs one and each kills one hypothesis. Buying both costs two and leaves three, which is not enough for the decisive configuration read and the panel the arithmetic needs.

The worker's own error lines cost the same one and kill both of those hypotheses together, because they name the layer the failure happens in. That is the general move worth taking away: a reading that identifies the layer eliminates every hypothesis that lives in a different layer, and it does it in one purchase. Component-shaped readings eliminate one thing at a time. Layer-shaped readings eliminate in bulk. Ask what layer before you ask what component, and your first query is worth three.

The remaining structure of the budget:

The decisive read costs two, and it has to be bought, because exactly one hypothesis survives the layer identification and nothing cheaper contradicts it. An incident where the last rival dies to a cheap panel is an incident with no teeth.
The panel the prediction is computed from also costs two. One plus two plus two is the budget exactly. Spend one query on reassurance and you can close the incident without being able to say what the number is, which this track scores as the guess it is.
The callee's own health dashboard eliminates nothing. It is clean because the failing requests never arrive, and a clean dashboard on the far side of a broken connection is not evidence of innocence. There was one honest number on it, and it was a rate that had quietly halved — the only trace this class of failure leaves on the serving side.

What people reach for

Nobody solves this class of problem by getting better at diagnosing it. They solve it by making the precondition's remaining life a monitored quantity.

The standard assembly in late 2026: automated issuance and renewal through an ACME client — cert-manager inside a cluster, Certbot or a built-in ACME client outside one — against a CA such as Let's Encrypt, with days-to-expiry scraped by a Prometheus blackbox exporter probe and alerted at thirty days and again at seven. The two-level alert is deliberate: thirty days is a ticket, seven days is a page, and the gap exists because the first alert is routinely acknowledged by somebody who intends to look at it later.

Three failure modes survive that assembly, and all three are common:

Renewal succeeded, the process never reloaded. The file on disk is current and the handshake is still serving the old chain, because nginx, HAProxy and most TLS terminators read the certificate at startup. The probe has to test the handshake from outside, not the file on disk.
The probe tests one path. A balancer serving several names, or several certificates behind name-based routing, needs a probe per name. Probing the default path proves the default path.
The expiring thing is not a certificate. The same shape comes from an expiring API token, a credential rotated by a schedule, a trust-store update that removed an intermediate, or a leased lock. Days-to-expiry is a property of all of them and is usually monitored for none of them.

The containment worth copying is narrower than full observability: treat a failed renewal as a page rather than a log line. A renewal pipeline that has not succeeded in a cycle is the only early warning this failure ever offers, and it is nearly always written down somewhere nobody reads.

Check

Forty clients hold connections that last twelve hours on average and are renewed at random. A shared handshake precondition breaks at midnight. Roughly how many clients are failing by eight in the morning?

All forty, as soon as their next request goes outAbout twenty — roughly half of themAbout three — only the clients that restartedNone, until something forces the clients to reconnect

Worth remembering

A smoothly climbing affected count is a discovery rate, not a spreading failure — the slope is the rate at which clients renew whatever carried the broken precondition.
A failure uniform across endpoints lives below the layer that knows what an endpoint is, which eliminates every hypothesis in business logic for free.
A reading that identifies the layer eliminates several hypotheses at once; a reading about a component eliminates one. Ask about the layer first.
A batched rollout produces the same staircase and is the rival worth keeping — it differs by having a ceiling.
A clean dashboard on the far side of a broken connection is not evidence of innocence, because the failed requests never arrived to be counted.
The one honest signal an expiry leaves on the serving side is a request rate that quietly halves.
Days-to-expiry belongs to tokens, leases, trust stores and locks as much as to certificates, and is usually monitored only for certificates.
Renewal automation succeeding is not the same as the serving process having reloaded — probe the handshake, not the file.

Where this stops holding

The cheapest elimination in this incident was a change log that said nothing had happened. That reading is worth one query here and worth nothing at all in a continuously deployed environment.

Once the change window contains more than one deployment, "a change landed near the start time" stops being evidence, because there is always a change near any timestamp. With dozens of deploys an hour, the change log tells you only that you work somewhere healthy. The triage order inverts: buy the layer-identifying reading first, never buy a change log as an elimination, and use it exclusively as a lookup once you know which layer and which component you are asking about.

What restores it is attribution rather than frequency. A deploy event stream that records which service and which layer each change touched lets you filter the window down to changes that could have produced the shape you are looking at, and that filtered list is evidence again. The test is the ratio: count the changes in your window, and if filtering by the affected layer does not get the list down to a handful, the change log is a lookup table and not a query worth spending on.