Part IV · Chapter 7 · When It Breaks · lesson 2

Which link in your chain owns your downtime

13 min

Builds on You promised three nines. That is nine hours a year and Your diagram has four dependencies. You have nineteen

Your status page promises 99.9 per cent. Every dependency on the request path carries a 99.9 per cent SLA of its own, and there are six of them: the managed database, the object store, the identity provider, DNS, the internal session service, and the payment gateway.

Six suppliers, all hitting their number. Nobody is in breach of anything.

Predict

Six dependencies in series, each at 99.9 per cent, each one needed for the request to succeed. What downtime per month does that imply for you?

About forty-three minutes — the same as any one of themAround two hundred and sixty minutes, over four hoursCloser to eighty-six minutes, because outages usually overlapIt cannot be worked out without knowing how often each one fails

Series availability multiplies, and multiplication of fractions goes the wrong way from intuition.

Six dependencies at 99.9 per cent is 0.999 to the sixth power. Each one contributes its own unavailability to the total, so the unavailabilities add up to roughly six times one component's — and the composite lands below every single figure that went into it.

This is why the first serious reliability conversation in a growing system is never about making a component better. It is about how many components a request has to pass through, because that count sets the ceiling before any individual number is negotiated.

The arithmetic, and the one thing that beats it

Availability chain

Hypothetical model, not a forecast

Composite availability of a chain of dependencies, each of which may be backed by replicas, and the downtime that composite implies.

It assumes

Every dependency in the chain carries the same availability figure, which real chains never do.
Dependencies fail independently, so two outages coinciding is vanishingly unlikely.
A request needs every dependency in the series to be up at the same moment.
A replica group counts as up whenever at least one of its replicas is up.
Downtime is quoted against a flat thirty-day month, with no maintenance windows excluded.

It ignores

Correlated failure: shared power, shared network, shared deploy, shared bug
Partial degradation, where a dependency is up but too slow to be useful
Failover time, during which the replicas are notionally up and actually not
Caches, retries and fallbacks, all of which hide some dependency failures
That adding a replica adds a dependency on whatever routes between them

You cannot renegotiate anybody's SLA, so leave the per-dependency availability where it is. Find the change that buys back more than half a percentage point of composite availability — and satisfy yourself that shortening the chain will not get you there.

Dependencies in series6

Every one of them has to be up for the request to succeed.

Replicas of each dependency1

One means no redundancy at all.

Composite availability: 0.994
Composite availability in percent: 99.401%
Share of the month unavailable: 0.006
Downtime, minutes per month: 259
Downtime per month: 4.3 h
Availability of one replica group: 0.999
Composite with no replicas at all: 0.994
Availability the replicas bought: 0
Nines of composite availability: 2.22

Downtime, minutes per month

A line rising almost straight from zero, adding roughly forty-three minutes of monthly downtime for every dependency added to the path.
Dependencies in series	Downtime, minutes per month
0	0
1	43.2
2	86.36
3	129
4	173
5	216
6	259

Not there yet — keep moving the controls.

Two replicas of each dependency takes the composite from 99.4015 per cent to 99.9994 per cent — from four hours and nineteen minutes a month to under twenty seconds. The second replica is worth more than the fifth by an enormous margin, because the gain is the unavailability raised to the replica count, and the first squaring does nearly all the work.

Which is the only honest way to beat a long chain. You cannot shorten it to one dependency and you cannot talk any supplier into another nine. You can stop any single instance being load-bearing.

Now find the one that is actually costing you

The model assumes every dependency carries the same number, which no real system does. Once the figures differ, the question stops being arithmetic and becomes a ranking problem — and the ranking surprises people, because the component that contributes the most downtime is rarely the least reliable one.

The contribution of a dependency is three things multiplied: how often it is unavailable, what share of requests touch it, and whether a request that touches it fails when it is down. That third factor is usually zero or one, and it is the one nobody writes down.

Here is a real-shaped path, ranked by contribution rather than by reliability.

Dependency	Availability	Share of requests	Fails the request?	Contribution
Managed relational database, multi-zone	99.95%	100%	yes	0.050 pp
Identity provider	99.99%	100%	yes	0.010 pp
DNS and certificate validation	99.99%	100%	yes	0.010 pp
Managed object store	99.9%	12%	yes, for those	0.012 pp
Payment gateway	99.99%	3%	yes, for those	0.0003 pp
Internal recommendation service	99.0%	100%	no — renders empty	0 pp

The recommendation service is a hundred times less available than the identity provider and contributes nothing. The database is the most reliable thing on the list after the identity provider and DNS, and it owns more of your downtime than everything else combined.

The reason is not a coincidence, and it generalises.

A component known to be unreliable gets a fallback, because somebody got paged about it and built one. A component known to be reliable gets depended on absolutely, because building a fallback for a thing that never fails is impossible to justify in a planning meeting.

So reliability and hardness are inversely correlated by construction. The unreliable things are soft and cost you nothing; the reliable things are hard and own your entire error budget. Ranking dependencies by their own availability reliably produces the wrong list.

Not the same thing

A hard dependency

One whose unavailability fails the request. There is no path through the code that produces a useful answer without it.

Measured in: Whether a request that touches it fails when it is down
For example: The primary database behind a write endpoint, or the identity provider in front of every authenticated route.
Chase it alone and: Its contribution is its full unavailability times its coverage, and nothing in its own metrics says so — it looks like the healthiest thing you own.

A soft dependency

One whose unavailability degrades the response. The request still completes, with something missing, stale or defaulted.

Measured in: Whether a request that touches it fails when it is down
For example: A recommendation carousel that renders empty, or a flag service whose SDK falls back to a value compiled into the binary.
Chase it alone and: Softness is a property of the code path, not of the component, and it rots. A timeout that is longer than the caller's budget turns a soft dependency hard.

Both sides accept
Both appear in the dependency list, both have dashboards, and both have owners. Nothing in the chain's diagram distinguishes them — only reading the error path does.

Which is which? Put each one on a side.

A feature-flag service whose SDK streams updates and evaluates against an in-memory cache, with defaults in the binary
The managed relational database behind every authenticated read
An object store holding user-uploaded images, served through a CDN with a long cache lifetime
The identity provider validating the session on every request
A search service, where a failure falls back to a filtered database query that is slower and less relevant
DNS resolution and certificate validation for the API hostname

Check

You have one engineering quarter and the ranked list above. Which piece of work buys the most composite availability?

Raise the recommendation service from 99.0 to 99.9 per cent — it is by far the worst number you ownMake the database path survive the primary being gone — a read replica serving reads, and a bounded write queueAdd a second identity provider, so authentication survives one of them failingNegotiate a stronger SLA with the object store provider

What a published SLA does and does not tell youOptional

The number on a provider's page is a commercial commitment with a refund schedule attached, and it is measured the provider's way.

Three things to read before using it as an input. What counts as down — most SLAs measure error rate per five-minute interval across a region, so a degradation that leaves you at a 30 per cent error rate for an hour may credit nothing. What is excluded — scheduled maintenance, anything attributed to your own configuration, and anything the provider classifies as a force-majeure event. And at what granularity — a regional figure says nothing about the zone or the instance you are actually on.

Observed availability is usually better than the SLA during normal quarters and much worse during the bad one, because the SLA is a floor the provider is prepared to pay out against rather than an estimate of behaviour. Use the published figure for arithmetic like this lesson's, and use your own measured dependency error rates for anything you are going to act on.

Worth remembering

Availabilities in series multiply, so a composite is always worse than the worst component: six dependencies at 99.9 per cent is 99.4 per cent, four hours and nineteen minutes a month.
Each dependency added to the path costs roughly one dependency's worth of downtime, so the count of hops sets the ceiling before any individual SLA is negotiated.
Replicas are the only lever that beats a long chain: two of each takes the same six-dependency chain to 99.9994 per cent, and the second replica is worth far more than the fifth.
A dependency's contribution is its unavailability times its share of requests times whether a request fails when it is down.
The third factor is usually missing from dependency lists, and it is the one that decides the ranking.
Unreliable components get fallbacks and contribute nothing; reliable components get depended on absolutely and own the error budget, so sorting by availability produces the wrong work list.
A published SLA is a refund schedule measured the provider's way, not an estimate of behaviour — use it for arithmetic, not for decisions.

Where this stops holding

The product assumes independence, and the whole calculation changes the moment you can name one thing that would take two dependencies down together.

A shared availability zone. A shared control plane, where the API you call to replace an instance is down for the same reason the instance is. A shared deployment pipeline that pushed the same bad configuration to both. A shared DNS zone, or a shared certificate authority. When any of those exists, the composite is better than the product predicts during ordinary months — the failures overlap instead of adding — and far worse in the one month that matters, because a correlated event takes out your replicas at the same instant as the thing they were replicating.

So the test for whether this lesson's arithmetic applies to you is a single question asked of every pair on the list: name a single event that would take both of these down. If you can name one, the multiplication is decoration and the real figure is the frequency of that event.