It was broken for six hours and no alert fired

A finance analyst opens a dashboard at 08:30 and asks why yesterday's settlements are missing. The reconciliation consumer stopped writing rows at 02:10, six hours and twenty minutes earlier.

In that window: zero alerts, zero errors logged above DEBUG, consumer lag flat at zero, throughput normal, every health check green, CPU and memory unremarkable. The service was polling messages the whole time and acknowledging them.

It just was not producing anything.

Predict

A consumer processes messages, commits its offsets, reports zero lag and zero errors, and writes nothing downstream for six hours. Which monitor would have caught it?

Choose one answer

Consumer group lag — a stuck consumer is exactly what lag is forError rate, with a low enough thresholdA check on the output — how old the newest settlement row isA liveness probe with a deeper health check inside it

Lock in answer

Case file

◇ Composite, not a real person

A payments team of nine. A Kafka consumer reads capture events and writes one settlement row per event into Postgres, feeding a finance dashboard and a nightly bank file.

A release at 01:50 adds a new required column to the settlement insert. The migration that adds the column is in the same pull request and runs as a separate step, which the deploy pipeline executes after the application rollout rather than before.

Hours broken before anybody noticed

6.3

Alerts fired

0

Monitors in place and healthy

5

Hours to full recovery after detection

5.8

1. 1The message handler had been wrapped in a broad try/except years earlier, to stop one malformed event killing the consumer. The handler logged the exception at DEBUG and returned normally.
2. 2Auto-commit was left on, so every message was acknowledged whether or not the handler had done anything with it.
3. 3Monitoring on this service was consumer lag, error rate, CPU, memory and a liveness probe that returned 200 whenever the process was running. All five were healthy for the whole window.
4. 4The nightly bank file job ran at 03:00, found zero new settlements, wrote a valid empty file, and exited zero.
5. 5The on-call engineer was paged at 08:41, after a human opened a dashboard. The migration was applied at 08:52 and rows began appearing again.
6. 6Replaying the six hours took until 14:20, because the offsets had been committed and the events had to be re-read from the topic's retained log by resetting a new consumer group.

No data was lost, because the topic's retention was seven days and the events were still there. The bank file for that day was empty and had already been delivered, which was discovered by the counterparty and required a manual resubmission and a regulatory notification. Total elapsed time from failure to full recovery was twelve hours and ten minutes, of which six hours and twenty was detection.

What luck did

Topic retention was seven days rather than the twenty-four hours it had been eighteen months earlier, so every missed event was still available to replay. Somebody had raised it for an unrelated capacity reason and nobody had revisited it, which is the only reason this was a delay rather than permanent data loss.

Who else tried this

Broad exception handlers around message handlers are close to universal, and almost all of them never cause an incident, because almost all of the exceptions they swallow genuinely are malformed single messages. The same pattern is running in most event consumers in the industry right now and producing nothing but quiet reliability, which is exactly why nobody removes it and why the one time it matters is a surprise.

Why it is not a template

This is one composite assembled to make a detection gap visible. It does not say that auto-commit is wrong, that broad handlers should be banned, or that migrations always belong before a rollout. The transferable part is narrow: when a failure produces no event, every monitor that counts events reports health, and the only thing that can see it is a measurement of the expected output.

Three genres, one property

The three failures that eat a working day have nothing in common mechanically and everything in common from the monitoring's point of view.

The stuck or ineffective worker. A consumer that polls and discards, a thread pool whose workers are all blocked on a lock, a loop whose body is inside a catch-all. The process is alive, the metrics it emits are the metrics of a healthy process, and the work is not getting done. Lag catches the version where it stops polling; nothing that counts errors catches the version where it polls and throws away.

The silently empty response. An upstream returns 200 with an empty array. A credential expires and the identity provider returns a token with no scopes rather than an error. A schema change makes a field null and the parser treats null as absent. A filter gets an extra condition and matches nothing. Every status code is a success, every latency is normal, and the payloads are empty. This is the one that reaches customers fastest, because an empty list usually renders as an empty page rather than an error.

The job that stopped being scheduled. A CronJob suspended during an incident and never resumed. A schedule that moved an hour when the cluster's timezone handling changed. A workflow paused in the orchestrator's UI by somebody debugging. A runner whose credential expired, so the job is scheduled and never starts. There is no failing job to alert on — there is no job.

The shared property: the failure emits nothing. Alerting built on counting bad events is structurally blind to all three, and the amount of monitoring you have makes no difference, because every monitor is counting something that is not happening.

What to alert on instead

Four patterns, in rough order of how much they pay for the effort.

Freshness of the output. The age of the newest record the system is supposed to produce, compared with how often it is supposed to produce one. now() - max(created_at) on the settlement table, alerting when it exceeds a few times the expected interval. This single check would have caught the case file at about 02:25, and it would equally have caught a migration failure, a credential expiry, a suspended job, a stuck consumer and a bug nobody has thought of yet — because it does not describe a failure mode, it describes success and notices its absence. If you add one monitor after this lesson, add this one. dbt's source freshness is this check as a first-class concept, and the data-quality tools built around it — Great Expectations, Soda, the commercial observability products — are mostly elaborations of the same idea.

Volume floors and the no-data trap. A counter that stops increasing is the signature of all three genres, and almost every alerting system defaults to not firing when a series disappears. Prometheus needs absent() or absent_over_time(metric[1h]) written explicitly, because a rule over a missing series produces no result rather than a breach. CloudWatch alarms need TreatMissingData: breaching, and the default is missing, which keeps the alarm in its previous state forever. Grafana's unified alerting has a separate No Data state that has to be mapped to Alerting on purpose. This is the single most common misconfiguration in this whole area: the one class of failure where the metric vanishes is the class your alerting is configured to ignore.

Invariant ratios between two sides of a pipeline. Settlement rows written divided by capture events consumed should sit near one. Orders created divided by payments captured, emails sent divided by notifications queued, documents indexed divided by documents uploaded. A ratio breaks when either side stops, and neither side has to produce an error for the ratio to move. It also catches partial failures that a freshness check misses — the consumer that writes a tenth of what it should has fresh output and a broken ratio.

Heartbeats, watched from outside. For anything scheduled, invert the direction: the job reports completion to an external service, and that service alerts on silence. Dead Man's Snitch, healthchecks.io and Cronitor all exist to be the thing that notices nothing arrived; Prometheus ships a permanently-firing Watchdog alert for exactly this purpose, routed out of Alertmanager to one of them. The reason it has to be external is that a monitoring pipeline cannot detect its own silence, and a failure that takes out your scheduler often takes out the thing that was going to alert on your scheduler.

Check

You have one of these to add to a nightly ETL job and you have to pick first. Which?

Choose one answer

An alert on the job's exit code being non-zeroAn alert when the newest row in the output table is older than the schedule allowsA heartbeat from the job to an external dead man's switchA row-count comparison against the same day last week

Check answer

›Why the detection time is the number in the postmortemOptional

Incident duration splits into detection, diagnosis and mitigation, and teams measure the last two obsessively because those are the parts with people in them.

In the case file, detection was six hours and twenty minutes and mitigation was eleven minutes. No amount of improvement to the eleven minutes matters. A team that halves its mitigation time saves five minutes; a team that adds one freshness check saves six hours, every time any of a dozen unenumerated causes occurs.

This is also why detection time is the honest denominator for monitoring work. "Do we have a dashboard for this" is unanswerable. "If this broke right now, how long before something told us, and which signal would it be" is answerable, and the answer is often a shrug for exactly the systems that produce something important once a day.

Run it as an exercise rather than an audit: pick three things your system is supposed to produce, and for each one name the signal that would notice its absence and how long that would take. Where the answer is a human opening a dashboard, you have found your next monitor.

Worth remembering

• Error-rate alerting can only fire on events that exist, and the defining property of a silent failure is that it emits none.
• Three genres account for most of them: a worker that runs without doing the work, an upstream that returns success with nothing in it, and a job that stopped being scheduled at all.
• Consumer lag catches a consumer that stopped polling, not one that polls, throws inside a swallowed exception, and commits the offset anyway.
• Freshness of the expected output — the age of the newest record against the expected interval — catches every upstream cause at once, including the ones nobody enumerated.
• Most alerting systems default to not firing when a series disappears, which makes them structurally blind to exactly this class: Prometheus needs absent_over_time, CloudWatch needs TreatMissingData breaching, Grafana needs No Data mapped to Alerting.
• An invariant ratio between two sides of a pipeline catches partial failures that a freshness check passes.
• Anything scheduled needs a heartbeat watched by something outside your monitoring stack, because a pipeline cannot detect its own silence.
• Detection time is usually the dominant term in incident duration and the one nobody measures.

Where this stops holding

All of this is a workaround for a failure that produces no event. The better outcome, wherever you can reach it, is to make the failure produce one.

Remove the catch-all around the handler, or narrow it to the exception class it was written for and let everything else kill the poll loop. Turn off auto-commit and commit after the work succeeds, so a failure stops the stream instead of quietly draining it. Assert the invariant in the code — this insert must affect exactly one row — and throw when it does not. Make the empty upstream response a failure at the boundary rather than a value that propagates. Each of those converts a silent failure into a loud one.

Once the failure throws and something counts the throw, error-rate alerting is strictly better than everything in this lesson: it fires in a scrape interval rather than a freshness window, it points at a stack trace rather than at a missing row, and it has a far lower false-positive rate than any threshold on a seasonal volume. The freshness check stays as a backstop for the causes you did not anticipate, and it stops being your primary detector.

So the test is whether the failure can be made to throw. Where it can, do that and the arithmetic of this lesson no longer describes your system. Where it cannot — a job that was never scheduled has no code in which to throw — these four patterns are all you have.