Part V · Chapter 9 · On Call · lesson 6
The graph came back and the complaints did not stop
15 min
Builds on Your median is 20ms and your p99 is 2 seconds
Part V · Chapter 9 · On Call · lesson 6
15 min
Builds on Your median is 20ms and your p99 is 2 seconds
No page. A message at 16:10 from an analyst: the approval rate has drifted from 31 per cent to 25 over five weeks, is anything wrong? Nothing alerted. No run failed, no feed arrived late, no errors. The nightly job has written the same number of rows every night for a year and the scoring service has not changed in fourteen months. Five weeks, and a human eye found it.
Predict
The headline that recovers while the complaints continue is the loud version of this failure. A service has an incident, the error rate returns to its usual line, the dashboard goes green, and the support queue keeps filling — because the error rate is an average over a population, and one slice of that population is still broken while the rest dilutes it back into the noise.
This incident is the quiet version. There is no recovery to be fooled by, no page, and no complaint queue; the only thing wrong is that a number is smaller than it used to be. But the arithmetic is identical, and so is the only move that gets anywhere.
A headline over two groups obeys one equation:
headline = share x rate in that group + remaining share x rate in the other
Which means a six-point fall in the headline has an enormous space of explanations, and they are not close to each other. A success rate that slips from 90 per cent to 85 might be every user losing five points, or one user in ten losing fifty while everybody else has a perfect day. Those two situations look identical on the chart, have nothing in common as causes, and want opposite responses. The chart cannot tell them apart, and will never be able to, because the information was destroyed when the average was taken.
Rearranged the other way, the same equation is a prediction engine. If you have the headline, the headline before, and the size of a group you suspect, the group's own rate is determined — you can compute it before you measure it. That is the strongest form of evidence available in a data incident, because it is a number you committed to while you still could have been wrong.
Three readings of a time series, in increasing order of what they settle.
A level is two endpoints: then and now. It is the cheapest reading on any shelf, and it distinguishes nothing, because every hypothesis predicts the same two endpoints.
A slope is accumulation. A population drifting, a backlog growing, a cache warming, an index degrading — something is happening continuously and has been for a while.
A step is an act. It happened once, at a time you can name, and it has not moved since. Populations do not step. People, deploys, config pushes and upstream releases do.
So: a level tells you something is different, a slope tells you something is accumulating, and a step tells you something was changed. Only the last two are evidence about a cause, and they are the expensive ones.
Seven queries against a shelf of nine readings that costs eleven — the most generous budget in this track. That makes a different mistake available, and two of the readings on this shelf exist mainly to offer it to you.
Incident
You are the one who got paged
Query budget
0 of 7 spent
0 of 9 readings open. The shelf costs more than you have, so some of it stays shut.
Commit
Two things at once, and in this order for a reason: there is no list of causes to pick from yet. Say what you think happened in your own words, and predict one number you were never allowed to query. If the cause is right, the number follows from what you have already read.
Among this week's rows whose months-at-address is EXACTLY zero, what is the approval rate?
Write the cause out first — a sentence you would say on the call.
The field's two endpoints — then and now — cost one query. The same field night by night costs two. The difference between those two readings is the entire incident, and the cheap one is written to be as satisfying as possible: it names a field, it quantifies the change, and it is compatible with every hypothesis on the shelf, including the one where nothing is broken and the applicants genuinely changed.
That is the general rule worth leaving with. Pay for shape. A level is two measurements and is nearly always available for free somewhere; the nights in between are what cost money, and they are what tells you whether you are looking at a population or at an act.
The rest of the shelf is the budget trap that comes with slack rather than the one that comes with scarcity:
And the reason the prediction is a referee rather than a second quiz question is visible once you have the panels: the arithmetic rests on a clause the data does not supply. Confining the damage to the suspect group is the hypothesis. Three different mechanisms are consistent with every purchasable reading on this shelf and imply three different numbers from the same panels — one where the group is ordinary and merely weaker, one where the whole table was shifted uniformly, and one where the group is destroyed. You cannot get the number right by arithmetic alone. You have to have the mechanism right first, which is the only property that makes a number a ruling.
Assert distributions, not presence. Every guardrail in this pipeline was built for absence: null checks, row counts, rejected-row counters, job exit codes. An empty value that parses defeats all of them at once, because the field is present, the type is right, the row survives, and the run succeeds. What catches it is an assertion about the distribution of each input field — the share of rows holding any single value, night over night, with a step in that share failing the run before anything downstream reads the table.
The tools people use for this in late 2026 are Great Expectations, dbt's test layer, Soda, and Deequ on the Spark side, plus the anomaly-detection products that sell themselves as data observability. They all support value-distribution expectations and they are all most commonly deployed with not-null and unique only, which is the configuration that would have missed this.
Know where your empty values come from. This failure class is usually a serialisation boundary:
| Boundary | How "unknown" becomes a value |
|---|---|
| proto3 without field presence | A scalar that is unset and a scalar set to zero are indistinguishable on the wire; marking the field optional restores presence. |
| JSON | Three distinct states — absent, null, and empty string — and consumers routinely coerce all three to a zero. |
| Avro | A field is nullable only if its type is a union with null; otherwise a missing value takes the schema default. |
| CSV and flat extracts | An empty cell has no type at all, and whichever reader you use has an opinion about it. |
A feed change marked non-breaking at the schema level can therefore be catastrophic at the semantic level, and the review that would catch it asks a different question from "does it still parse".
Monitor the slices, not only the headline. The detection fix is to publish the metric cut by the handful of dimensions that matter — segment, channel, client version, region — and alert on divergence between slices rather than on the aggregate's level. The cost is honest: slice alerts are noisier, and a team that gets paged for a small slice moving will turn them off. The usual compromise is to alert on the aggregate and to make the slice view the first thing the runbook opens.
The tradeoff nobody escapes is threshold sensitivity. A distribution assertion tight enough to catch a quarter of the rows changing will also fire on ordinary seasonal movement unless it is written as a step test rather than a level test. Assert on the change between consecutive nights, not on the absolute share, and the noise mostly disappears — which is the same distinction between level and shape that solved the incident.
Check
The split arithmetic has one assumption that is easy to miss because it is almost always true: that the metric and the group share are counted in the same unit. Every row is one decision, every decision is weighted once, and the group's share of rows is also its share of the thing being averaged.
When the metric is sampled per user rather than per request, that stops holding and the prediction comes out wrong in a direction you cannot see. Suppose the damaged group submits three applications each and everybody else submits one. A per-user sample puts the group at one in ten of the sample while it is three in ten of the decisions, so the share you multiply by is wrong by a factor of three — and the rate you solve for is wrong by about the same factor, with no hint on the panel that anything is off.
The discipline is to compute the split in the unit the harm occurs in, and to say out loud which one you are using. Request-level harm — a failed checkout, a declined application, a dropped write — is counted per request, and a per-user metric understates it for heavy users and overstates it for light ones. User-level harm — how many people had a bad day — is counted per user, and a per-request metric is dominated by whoever is most active. Both questions are legitimate and teams need both. What is not legitimate is a weighted average whose weights come from one unit and whose rates come from the other, which is also the most common way a correct-looking data investigation reaches a confident wrong number.