Part IV · Chapter 7 · When It Breaks · lesson 6

The postmortem that changes something, and the one that does not

14 min

Builds on The thing that broke is downstream of the thing that caused it

Two postmortems, six weeks apart, same team. Both blameless, both signed off.

The first has eleven action items. Nine are still open, four have no owner, and three are sentences beginning "be more careful about".

The second has two. Both shipped inside a fortnight, and one is a check in the deploy pipeline that now refuses the change that caused the outage.

The cause recurred once in that period. Only one document had anything to do with that.

Predict

What most reliably separates a postmortem that changes something from one that does not?

How thoroughly it establishes the root causeWhether it was blamelessWhether each action item would have changed a number in this outage's timelineHow quickly it was written after the incident

The worked example

The outage: a payments consumer stopped writing settlement rows at 02:10 because a migration ran after the rollout that depended on it. A broad exception handler swallowed the failure, auto-commit acknowledged the messages anyway, and nothing alerted. A human noticed at 08:30. Replay finished at 14:20.

Here is the document, in five sections, and then the thing it is for.

1. The timeline, with detection on its own line


Change deployed	01:50
Failure began	02:10
Detected	08:30
Mitigated	08:52
Fully recovered	14:20

Detection was six hours and twenty minutes. Mitigation was twenty-two minutes. Recovery was five and a half hours.

Writing those three as separate numbers is the highest-value line in the document, because it decides where the work goes and it contradicts everybody's instinct. The incident felt like twelve hours of heroics. It was twenty-two minutes of fixing, six hours of nobody knowing, and five hours of replay.

2. The cause, the amplifier, the symptom — named separately

Cause: the pipeline applied the migration after the application rollout, so new code ran against an old schema for the length of the deploy.

Amplifier: the broad exception handler plus auto-commit. Without it the consumer would have crash-looped at 01:52 and paged somebody inside two minutes. The amplifier here did not make the failure bigger; it made it quieter, which is the same thing measured in hours.

Symptom: missing rows in a finance dashboard, observed by a human.

Detection gap: nothing measured the output. Five monitors, all of them counting events, none of them able to see an absence.

Separating these three is what stops the document producing a single action item about migration ordering and calling it done.

3. The counterfactual column

Every candidate action item gets one sentence: what would have been different.

Candidate	What changes in this timeline
Pipeline runs migrations before rollout	The failure does not happen. This trigger only.
Freshness alert on the settlement table	Detected at about 02:25 instead of 08:30
Narrow the exception handler, commit after success	Consumer crash-loops at 01:52, paged in minutes
Bank file job fails rather than writing an empty file	No empty file delivered; no regulatory notification
"Review migrations more carefully in code review"	Nothing in the timeline changes
"Add a dashboard for consumer health"	Nothing — there were five, all green

The last two are the ones that would have survived the meeting unchallenged.

4. The classification

Four kinds, and only three of them are work.

Remove the trigger. The migration ordering fix. Cheapest, most satisfying, and narrowest: it prevents this exact change and nothing else. Worth doing and never worth being the only item.

Bound the amplifier. Narrowing the handler and committing after success. This is the item with the widest reach, because it changes the system's behaviour for every future failure of that consumer regardless of cause — including the causes nobody has thought of. Amplifier items are almost always the highest-value ones and almost always the ones that get deferred, because they are refactors rather than fixes.

Shorten detection. The freshness alert. Attacks the dominant term in the timeline, costs very little, and like the amplifier item it is cause-agnostic.

Promise to be careful. Review more thoroughly, communicate earlier, monitor closely, be mindful of ordering. These have no completion condition anybody other than the author can check, no owner who can finish them, and no effect while everybody is asleep — which is when the next one will happen.

5. The items that shipped

Two. The freshness alert, and a check in the deploy pipeline that fails the build when a migration adding a non-null column ships in the same release as code that writes to it.

The second one is the shape worth noticing. It is an automated check rather than a practice, so it keeps working after everybody who remembers the outage has left. An action item that becomes a test, a lint, a limit or a default is the only kind that does not decay.

The trigger is what happened this time. The cause is the property of the system that let it matter.

A migration ran in the wrong order: trigger. A consumer that cannot distinguish "this message failed" from "every message is failing": cause. Fix the trigger and the cause returns in a month behind a different trigger — a credential expiry, a downstream schema change, a bad deploy of the thing it writes to — and the new postmortem will look unrelated to the old one.

This is also the standing objection to walking five whys down to a single root. It produces a chain of counterfactuals about one specific path and ends on the most specific, least transferable link. Name the amplifier and the detection gap alongside it or the document is about an event rather than about a system.

What would you do

The review meeting has produced six candidate action items for the outage above. Everybody agrees all six are good ideas. Your team's observed completion rate on postmortem items last quarter was two in nine, and there is no reason to think this quarter is different.

You are choosing which two to write down, knowing that writing down six is the same as choosing two at random.

Optimising for: Minimise the total duration of the next incident caused by this class of problem, given that you will get two items shipped and the rest will sit open.
You cannot know: Whether the next incident in this class arrives through the same trigger or a different one
How much of the five-hour replay was avoidable, which nobody measured
Whether the broad exception handler is load-bearing for a malformed-message case somebody hit two years ago
Whether the finance team would act on an alert at 02:25 or wait until morning anyway

Migration ordering in the pipeline, and a code-review checklist item for schema changesFreshness alert on the settlement table, and narrow the exception handler so a systemic failure crash-loopsFreshness alert on the settlement table, and migration ordering in the pipelineMake the bank file job fail on zero rows, and add an alert on consumer error-rate with a lower threshold

Check

Which of these is real work rather than a promise to be careful?

Add alerting for this failure modeAdd a CI check that fails the build when a migration adding a non-null column is in the same release as code writing to itDocument the deploy ordering requirements in the runbookHold a brown-bag session on migration safety for the team

Now write one

Pick an outage you have actually had — one you were present for, not one you read about. It does not have to be large. A broken deploy, a stuck job, a day of corrupted data, an alert that fired at 04:00 for nothing.

Write these seven lines. Not a document; seven lines.

The timeline, with detection on its own line. Change, failure, detection, mitigation, full recovery. Five timestamps, and the three durations between them.
Which of the three durations was the largest. Write the number down even if you have to estimate it, because this is the line that decides the rest.
The cause, the amplifier and the symptom, as three separate sentences. If you cannot separate them, the document is about an event.
What was already true before the change. The property of the system that made this change dangerous — and which is still there.
One sentence of luck. What went right that you did not control. There is always one, and a postmortem without it is describing a different, luckier system than the one you operate.
Two action items, each with its counterfactual. What number in your own timeline would read differently if this had already been done. If you cannot answer for one of them, replace it.
The thing you are deliberately not fixing, and why. Written down, so the next person finds a decision rather than an oversight.

The seventh line is the one that takes the longest and it is the one that makes the document honest. Every real postmortem leaves something unfixed. A document that pretends otherwise is the eleven-item version, and you already know what happens to it.

The number that tells you whether any of this is workingOptional

One figure, measured once a quarter: of the action items from last quarter's postmortems, what share shipped?

If it is above two thirds, your lists are the right length and you can afford to be ambitious. If it is below a third, your lists are fiction and every meeting is producing a document whose main output is the feeling of having responded. The fix is not more discipline. It is shorter lists — two items chosen on the counterfactual test beat eleven chosen on consensus, because the eleven ship in the order of how easy they are.

The incident tooling in common use — incident.io, FireHydrant, Rootly, and the Jira or Linear projects most teams actually use instead — will all report this if you let them, and almost nobody looks, because the number is embarrassing in a specific way: it is a measurement of the gap between what the team decided and what the team did.

Two pieces worth reading in full rather than summarised: John Allspaw's "Blameless PostMortems and a Just Culture", which is where the argument against counterfactual root-cause chains comes from, and the Howie guide to incident analysis, which is the most practical treatment of running the interview that produces an honest timeline.

Worth remembering

Split the timeline into time-to-fail, time-to-detect and time-to-mitigate, and act on the largest. It is usually detection, and it is usually the one nobody measures.
Name the cause, the amplifier and the symptom separately, or the document produces one action item about the trigger and calls it finished.
The trigger is what happened this time; the cause is the property of the system that let it matter. Fixing only the trigger means the cause returns behind a new one.
Give every candidate action item one sentence of counterfactual: which number in this timeline would read differently. No answer means it is not a mitigation.
Three kinds of item are work — remove the trigger, bound the amplifier, shorten detection — and the amplifier and detection items are cause-agnostic, so they pay off against failures nobody has imagined.
The three tests: can an outsider tell it is done, can one person finish it without permission, does it do anything while everybody is asleep.
An item that becomes a test, a lint, a limit or a default is the only kind that survives the people who remember the outage.
Your action-item completion rate is the binding constraint, so a list longer than the rate can absorb is a lottery in which the easy items win.
Write down what you are deliberately not fixing, so the next person finds a decision rather than an oversight.

Where this stops holding

The counterfactual test is anchored to one timeline, and that is its limit as well as its strength.

An item that would have changed this outage's numbers is not necessarily an item that bounds the next one. The failure case is specific and common: the same underlying cause returns behind a different trigger, and because the action items were scored against the old timeline, none of them fire. The consumer that could not distinguish one bad message from total failure gets a freshness alert on its output table, and then the failure next time is that it writes the wrong rows rather than none, which is fresh and wrong and silent again.

So the signal that this method has run out is a second incident whose postmortem reads as unrelated to the first and whose amplifier is the same. When that happens, the unit of analysis has to move up: not this outage, but the five documents from the last two quarters read together, looking for the amplifier or the detection gap that appears in three of them under three different names. That reading is a different exercise, it needs a backlog of honest documents to work on, and the only way to have one is to have been writing the seven lines all along.