Part V · Chapter 9 · On Call · lesson 5
Broken for a tenth of people and perfect for everyone else
14 min
Builds on One table is 400GB and everything that touches it is slow
Part V · Chapter 9 · On Call · lesson 5
14 min
Builds on One table is 400GB and everything that touches it is slow
06:40. Support has four tickets saying messages will not send. Writes are fine on fifteen of sixteen shards. Shard seven's write latency is around ten times its peers and its CPU has been pinned since 06:00. One release went out at 23:40 last night. Everyone else's experience is perfect, which is why nothing paged until a human complained.
Predict
A minority failure has a boundary, and there are only a few things in a system shaped like a boundary. Each produces its share for a different reason, and each has a reading that identifies it:
| Boundary | Why it is that share | What identifies it |
|---|---|---|
| A hash partition or shard | One of N partitions, so the share is near a multiple of one over N — unless the keys are uneven, in which case it is whatever the keys say. | Group failures by partition id, or by the hash of the key. |
| A region or availability zone | Exactly that zone's traffic share, with a clean geographic cut. | Group by zone, or by client network. |
| One node of a fleet behind a balancer | One over N of the fleet, and the same user succeeds and fails alternately. | Group by which instance served the request. |
| A flag or canary cohort | Exactly the rollout percentage, and it is sticky per user. | Group by variant or build version. |
Two free discriminators separate those four before you buy anything.
Per-user or per-request? A partition and a cohort are sticky: an affected user is affected on every attempt, because the thing that put them in the broken set is a property of who they are. A bad node behind a balancer is not sticky: the same user fails, retries, and succeeds. Asking one affected person to try again costs nothing and splits the table in half.
Is the share a round number? Five, ten, twenty-five and fifty per cent are numbers a human typed into a rollout. A share like six per cent or fifty-six per cent is a number that came out of a key distribution. Rollouts produce suspicious round numbers; data produces awkward ones.
And one standing warning, because it is the trap in this incident shape: a resource pinned on the affected node is a consequence until something proves it is a cause. A node doing many times the work of its peers will show a saturated CPU, a deep queue and bad latency, and all three are what being overloaded looks like from the inside. The reading that turns that around is not a deeper look at the node — it is a comparison of how much work each node is being asked to do.
Five queries against a shelf of seven readings that costs nine. Two pairs eliminate the same hypothesis, and one reading identifies the boundary while eliminating nothing at all.
Incident
You are the one who got paged
Query budget
0 of 5 spent
0 of 7 readings open. The shelf costs more than you have, so some of it stays shut.
Commit
Two things at once, and in this order for a reason: there is no list of causes to pick from yet. Say what you think happened in your own words, and predict one number you were never allowed to query. If the cause is right, the number follows from what you have already read.
How many writes a second was shard seven taking at this hour yesterday?
Write the cause out first — a sentence you would say on the call.
This is the first shelf in the track with slack in it: the cheapest closing set costs four against a budget of five. Slack is where souvenirs get bought, so it is worth being precise about what the fifth query is for.
The structure was three eliminations, and two of them were available twice:
The elegant part of this shelf is that the panel which eliminates the hardware story is also the panel the arithmetic is computed from. One purchase, one elimination, and every term in the sum. When a reading describes the shape of an imbalance it usually also prices it, which is why "compare the work each node is doing" is the highest-yield single query in any incident with a minority boundary.
The reading that identifies the boundary eliminates nothing, and that is not a criticism of it. Direction and elimination are different services. A reading that tells you the failures all share an attribute does not rule out a single hypothesis — every story on the shelf can produce a concentrated boundary — but it tells you which comparison to make next, and in a real incident with no shelf to pick from, that is the difference between three candidates and thirty. Buy it for direction, and do not score it as progress.
A per-partition view, permanently. The aggregate hides this failure completely: sixteen shards averaged together look mildly busy while one of them is on fire. What teams reach for is a top-N-by-rate panel plus a single imbalance number — the busiest partition divided by the mean — alerted when it crosses something like two. Most managed stores expose the raw material: CloudWatch per-partition metrics and contributor insights for DynamoDB, per-topic partition lag and size for Kafka, per-shard statistics in Vitess and Citus, nodetool tablehistograms for Cassandra. The alert on the ratio is the part people skip.
Key design that bounds the grain. A hash spreads keys, not rows, so the smallest unit it can place is one key's worth of traffic. If the key is a tenant, the grain is a tenant. The standard remedies all shrink the grain by hand: add a salt or a bucket number to the key so one tenant spreads across a known number of partitions, or keep a compound key whose high-cardinality component does the spreading. Both cost you something on read — a lookup by the coarse key now has to fan out across the buckets and merge, which turns one request into several and puts you in the fan-out arithmetic from track four.
Replay before you ship. The containment that would have prevented this is narrow and cheap: take a day of real write keys, run them through the proposed partition function offline, and refuse the change if the busiest partition would take more than roughly twice the mean. A partition key is a traffic-distribution decision dressed as a data-modelling decision, and it deserves the review you would give a schema migration rather than the review you would give a rename.
What not to reach for: moving the hot partition to a bigger machine. It works for a while, it is the only option available in the first ten minutes, and it leaves a system whose load distribution is set by one customer's growth curve.
Check
Every move above assumes the broken set is bounded and stable: the same tenth of people, failing on every attempt, with fifteen partitions out of sixteen untouched.
When the affected share grows past a single partition, concentration by key stops being the model. Two hot partitions out of sixteen is not twice as much of this incident — it is evidence that the thing you are looking at is not the key at all, because a key distribution does not change twice in one morning. The candidates shift to something global: total write volume, a change in the size of each write, a replication or compaction backlog that happens to be visible first on the busiest partitions, or a client library that started doing more work per call.
And a boundary that is moving is a different shape entirely. A share that climbs hour by hour is a discovery rate, which is the second lesson in this track, and the per-partition comparison will send you in circles because today's boundary is not tomorrow's.
The test is to measure the boundary twice. Describe the affected set, wait twenty minutes, describe it again. A stable set is a structural fault and the structure is where the answer is. A growing set is a process, and processes are diagnosed by their rate rather than by their membership.