The thing that broke is downstream of the thing that caused it

09:14. The page says pgbouncer: no more connections available. Every checkout is timing out. The database dashboard is solid red: connections pinned at the pool ceiling, waiting clients climbing, query latency up tenfold.

Four engineers spend forty minutes on the database. They raise the pool size twice; both times it is full again within a minute.

The database was fine. It is still fine. Whatever caused this had already happened before the first graph moved.

Predict

The loudest signal in a cascade is almost never the cause. Which graph should you look at first to find one?

Choose one answer

The one with the largest deviation from baselineThe one that departed from baseline earliest, however slightlyThe one belonging to the service that owns the alertThe deploy timeline, because it is nearly always a deploy

Lock in answer

Four shapes, and what each one rules out

Before you know which service is involved, the shape of the first graph to move tells you what category of thing happened. These are the four worth being able to name on sight.

Shape	Reading	What it rules out
Latency rising, completions flat or falling	Saturation. Work arrives faster than it leaves and the difference is sitting in a queue.	A pure demand increase. If demand were the story, completions would be rising too.
Latency rising, completions rising with it	Demand. More work arrived and the system is doing more work.	A fault. Go and find out what changed about the traffic before touching the service.
Errors rising, latency flat or falling	Fast rejection. Something stopped doing work: a breaker opened, a pool is refusing, a limiter is shedding, a dependency is returning an error quickly.	Saturation. A saturated component gets slower before it gets wrong; a rejecting one gets faster.
A step at a timestamp, no ramp	A discrete event. A deploy, a config push, a scheduled job, a leader election, an expiry.	Anything load-driven. Load moves in ramps and cycles; it does not move in a single vertical line.

The first one always means saturation. It is not a heuristic. Latency rising while completed work stays flat is the definition of a queue growing: arrivals exceed departures, and the gap accumulates as waiting time. Something in the path has run out of a resource and the rest of the system cannot tell which, because waiting looks identical from the outside no matter what is being waited for.

Two consequences follow, and both are commonly missed.

The saturated resource is often not CPU, so CPU utilisation can sit at a third of the box while the service is fully saturated on something else — a connection pool with every connection checked out, a thread pool with every worker blocked, a semaphore, a file descriptor limit, a lock. "Utilisation is low, so we have headroom" is wrong whenever the thing that ran out is countable rather than proportional.

And the saturated component is not necessarily the component whose latency rose. Waiting propagates upward. Every caller of a saturated thing shows the same shape, which is why the shape tells you a queue exists but not where it is.

Case file

◇ Composite, not a real person

A retail platform, roughly forty engineers, eight services behind one gateway. Checkout reads a product-pricing cache in Redis and falls through to Postgres on a miss, through PgBouncer in transaction pooling mode.

A routine deploy at 08:58 adds a currency field to the pricing cache key, so every key written by the new code misses every key written by the old. The rollout is gradual, two instances at a time, over about twelve minutes.

Minutes of customer impact

52

Minutes spent on the wrong component

39

Minutes from sorting by time to rollback

6

Pool size changes that helped

0

1. 1Paged at 09:14 on PgBouncer connection exhaustion. Treated the database as the subject of the incident, because that is what the alert named.
2. 2Raised the PgBouncer pool size from 100 to 200 at 09:21. Filled again within a minute, and query latency got worse rather than better.
3. 3Raised it again to 400 at 09:29. Postgres itself now began to degrade, because four hundred concurrent transactions is a different workload from a hundred.
4. 4Checked the deploy timeline at 09:33, found the 08:58 deploy, and dismissed it because the graphs had not moved until 09:12 and the deploy was described as a one-line change.
5. 5At 09:41 somebody sorted every affected panel in Grafana by the time it left baseline, rather than by how red it was.
6. 6The cache hit ratio was the earliest departure, falling from about 96 per cent at 09:02 and crossing 10 per cent around 09:11. Rolled the deploy back at 09:47.

Hit ratio recovered as the old keys were repopulated; connection pressure cleared about four minutes after the rollback, before anybody touched the pool settings again. Total customer impact was fifty-two minutes, of which the first thirteen were the cache warming down and the remaining thirty-nine were spent on a database that was never faulty. The pool was left at 400 for two weeks and caused a separate, smaller incident.

What luck did

The deploy was gradual rather than all at once, which stretched the cache miss ramp over roughly ten minutes and left a visible slope for somebody to find. The same change shipped to every instance simultaneously would have produced a vertical step in all the graphs at once, destroying the time ordering that eventually solved it.

Who else tried this

Most teams who raise a pool limit during an incident report that it helped, because most of the time the incident was ending anyway and the change is credited with the recovery. The same four decisions are made every week across the industry and produce a successful-looking incident log, which is exactly why raising the limit is still the first reflex and why this team had done it twice before without learning anything.

Why it is not a template

This is one composite built to make an ordering visible. It does not say that a database is usually innocent, that pool limits should never be raised, or that cache keys are a common cause of anything. The transferable part is narrow: when several signals degrade together, the earliest and smallest departure is the candidate cause and the loudest one is the symptom, and sorting by time costs about a minute.

Reading it backwards

The path was: cache key change, then cache misses, then every request falling through to the database, then connection demand multiplying, then the pool exhausting, then checkout timing out. Six stages. The alert fired on the fifth.

Walking it backwards is mechanical if you ask one question at each stage: what would have to be true upstream for this to look like this? A pool exhausts because demand for connections rose or because connections stopped being returned. Demand for connections rose because queries rose. Queries rose because something that was absorbing them stopped. That is three questions and it lands on the cache without knowing anything about the deploy.

The shape would have said the same thing. Checkout latency rose while completed checkouts stayed flat — saturation, which rules out a demand story, which means the traffic was unchanged and something internal had run out. Nobody looked, because the alert had already named a component.

Check

A service's error rate jumps from near zero to 30 per cent in one scrape interval, and its p99 latency drops at the same moment. What happened?

Choose one answer

It saturated — the errors are timeouts under loadSomething started refusing work quickly — a breaker opened, a limiter engaged, or a dependency began failing fastA dependency got slower, and the errors are the knock-onA bad deploy introduced an exception on a hot path

Check answer

›Three panels, in this order, on every service dashboardOptional

Most dashboards are organised by component because that is how the exporters are organised. The ordering that actually helps during an incident is by question.

Panel one: arrivals and completions on the same axes. Requests started per second and requests finished per second. When they separate, you have the saturation shape, and you have it before any latency percentile has moved far enough to alert. This single panel distinguishes the first two shapes in the table and almost nobody has it.

Panel two: latency percentiles and error rate on the same time axis. Not the same axes — the same axis of time, side by side, so the direction they move relative to each other is readable at a glance. That distinguishes saturation from rejection.

Panel three: a deploy and config-change annotation overlay. Steps need a changelog next to them, and an annotation stream from the deploy pipeline turns "something changed at 09:58" into a link to a diff. Grafana annotations and the equivalent event overlays in Datadog and in OpenTelemetry-backed stacks all do this, and the work is a webhook from the deploy pipeline.

The methods behind this are worth reading in full: the four golden signals (latency, traffic, errors, saturation) from Google's SRE book, Brendan Gregg's USE method for resources, and Tom Wilkie's RED method for request-driven services. They disagree about vocabulary and agree about the panels.

Worth remembering

• A cascade has a cause, an amplifier and a symptom, and they are usually in three different services. The alert names the symptom.
• Sort every affected signal by the time it left baseline rather than by how far it moved — the earliest departure is the candidate cause and it is usually small.
• Latency rising while completions stay flat always means saturation, because that is the definition of a queue growing.
• The saturated resource is often countable rather than proportional — a pool, a thread, a semaphore — so low CPU utilisation does not mean there is headroom.
• Waiting propagates upward, so every caller of a saturated component shows the saturation shape and the shape cannot tell you which one ran out.
• Latency rising with completions means demand, not a fault. Go and look at what changed about the traffic.
• Errors rising while latency falls means something is refusing work quickly, which is the opposite diagnosis from saturation.
• A step with no ramp is a discrete event and needs a changelog, not a capacity investigation.

Where this stops holding

The whole method is time ordering, so it dies the moment the resolution of your metrics is coarser than the cascade.

A cascade that completes in forty seconds, read off a dashboard scraping every fifteen seconds and graphed as a rate over a five-minute window, shows every signal turning at once. The averaging window smears the ordering out of existence and the earliest departure is no longer visible, so the only thing left to sort by is magnitude — which puts you back on the symptom.

The test is concrete: compare the duration of the incident with the width of the window your panels average over. If the incident is not several times longer than the window, the ordering on the screen is an artefact and you should go to raw event data — request logs with timestamps, the deploy pipeline's own event stream, distributed traces with their span start times — instead of to the graphs. Traces in particular keep per-request ordering that no aggregated metric can reconstruct, and they are the right tool for a cascade that was over before the second scrape.