Part V · Chapter 9 · On Call · lesson 1
Saying the right words about a cause you have not understood
14 min
Builds on The thing that broke is downstream of the thing that caused it
Part V · Chapter 9 · On Call · lesson 1
14 min
Builds on The thing that broke is downstream of the thing that caused it
02:40. The page says profile read p99 is 905 milliseconds against a 200 millisecond budget. p50 is 41 and was 38. No deploy for nine days, no traffic change, no errors — every read succeeds, slowly. Someone types "the store got slow" into the channel. They are probably right. They also cannot tell you what a read that misses the cache costs, which means they have a label rather than a diagnosis.
Predict
"The store got slower" is a sentence that costs nothing to say and cannot be wrong in an interesting way. So is "the network", "garbage collection" and "it is the database". Each of them is compatible with half a dozen mechanisms that want completely different fixes, and each of them survives contact with any dashboard you care to open, because none of them forbids anything.
A label becomes a diagnosis at the moment it predicts a number you have not looked at yet. That is the entire discipline of this track, and it is why the block below will not let you type a cause without also committing to a value for one reading that is not on sale at any price. The cause is graded on the number. If the mechanism in your head is the real one, the number follows from panels you already bought. If you recognised a vibe, it does not.
A mean over a mixed population is a weighted average of the paths inside it. With two paths — a cheap one taken a known share of the time and an expensive one taken the rest — the relationship is fixed:
mean = share x cheap + (1 − share) x expensive
Three of those four quantities are usually already on a dashboard. The fourth falls out by rearranging, which means the cost of a path nobody instrumented is frequently computable from panels that exist.
This is the most useful arithmetic in incident response and almost nobody does it, because a mean looks like an answer rather than a sum. The habit: whenever a percentile and a median disagree, write down the two populations you are averaging and solve for the one you cannot see.
You have five queries. The shelf holds more than five queries' worth of readings, so the triage is forced: every panel you open is a panel you cannot open later, and two panels that rule out the same hypothesis are one reading and a souvenir.
Incident
You are the one who got paged
Query budget
0 of 5 spent
0 of 8 readings open. The shelf costs more than you have, so some of it stays shut.
Commit
Two things at once, and in this order for a reason: there is no list of causes to pick from yet. Say what you think happened in your own words, and predict one number you were never allowed to query. If the cause is right, the number follows from what you have already read.
What is the mean latency of a read that misses the cache?
Write the cause out first — a sentence you would say on the call.
Rank the shelf by hypotheses eliminated per unit of cost, not by how interesting a panel sounds.
On this shelf there were only three eliminations available at any price, and two of them were sold twice. The cache hit rate and the queries reaching the store rule out the same hypothesis; the host's resources and the host-by-host spread rule out the same other one. Buying both halves of either pair spends a query to learn something you already knew. Three of the eight panels — the headline latency split, the change log, the error log — eliminate nothing at all, which does not make them worthless; the headline split is where the arithmetic comes from, and that is a different kind of useful.
The budget was exactly tight enough to punish one specific mistake. The cheapest set that both closes the incident and leaves you able to derive the held-back number costs all five: one elimination from each pair, the trace, and the panel the arithmetic needs. The change log, with its maintenance window and its conveniently-timed backfill, is the most tempting reading on the shelf and it is the one that cannot be afforded. A coincidence in a change log is a coincidence until a reading connects it to a mechanism, and the connecting reading was going to cost two.
The order that works, in general:
| Spend on | Because |
|---|---|
| A reading that eliminates, over a reading that confirms | Confirmations are compatible with every hypothesis that was going to produce this symptom, which is all of them. |
| The cheapest reading that eliminates a given cause | Two panels with the same elimination are one purchase. |
| A panel your arithmetic depends on, early | You cannot predict a number from a dashboard you decided to skip. |
| A trace, once two hypotheses disagree about where time is spent | It is the only reading that attributes time to a layer, and it is priced accordingly. |
The fix for the detection failure here is not a better alert threshold on the headline percentile — that number was behaving exactly as designed. It is to stop averaging two paths into one series.
In a Prometheus-and-Grafana stack that means a histogram with a label for the cache outcome, so the hit path and the miss path are separate series, with the alert on the miss path's own quantile. The cost is cardinality: every label value multiplies the series count, and a label with high cardinality — a user id, a key — is how a monitoring bill and a query latency both quietly double. Cache outcome is two values, which is the cheapest useful label in existence.
The second reach is exemplars: OpenTelemetry and Prometheus both support attaching a trace id to a histogram bucket, so clicking the slow bucket lands you on a trace of a request that was in it. That is the difference between paying two queries for a sampled trace and already having one. It needs tracing deployed on the path and a sampler that keeps slow requests, which is a head-based sampler with a tail rule, or a tail sampler if you are willing to run the collector that does it.
What not to reach for: raising the cache's entry lifetime to push the hit rate up. It works, in the sense that it moves the percentile. It also widens the window in which every reader serves a stale profile, and it leaves the dependency's regression in place and invisible, waiting for the next hit-rate dip to expose it.
Check
Everything above assumes the two populations were both being recorded, even if only as part of a mixture. Split-the-mean works because the hit path's cost and the hit rate were on a dashboard; solving for the third quantity is arithmetic, not instrumentation.
When the metric was never recorded at all, the method changes completely. There is nothing to rearrange, no prediction anybody can referee, and the honest first move during the incident is to add the instrument — a counter and a histogram on the miss path, shipped mid-incident behind a flag — and wait for data. That is minutes you are spending to find out where to look, and it is usually the right trade, because the alternative is restarting components in a sequence chosen by seniority.
The test for which situation you are in is concrete and takes one minute: for each hypothesis on your list, write down a reading that would be different if it were true. If a hypothesis has no such reading on any dashboard you own, it is unfalsifiable today, and you are not debugging — you are voting.