Part II · Chapter 2 · One Service · lesson 6
You promised three nines. That is nine hours a year
12 min
Part II · Chapter 2 · One Service · lesson 6
12 min
The contract draft says 99.9% availability, monthly. Sales has already said yes verbally. Legal wants a sign-off by Friday.
Nobody in the thread has converted the percentage into a duration, which is the only form of it anybody can be held to.
How much downtime a year is 99.9%? Commit to a figure before you read on.
Predict
Two pieces of arithmetic, and they point in opposite directions.
Serial dependencies multiply. A request needs the load balancer, the application, the database and the identity provider all working at once. Three things at 99.9% each is 0.999 cubed, which is 99.7% — worse than any single component, and worse the more components there are. A product of numbers below one falls below all of them.
Replicas multiply the failure instead. A group survives if any replica survives, so two replicas at 99.9% each give a group unavailability of 0.001 squared. The group reaches 99.9999%.
That asymmetry is the whole game. Adding a dependency costs you a nine; replicating one buys you several. And the second replica is worth enormously more than the fifth, because the first squaring is the one that matters.
Three dependencies in series, each one promising 99.9%, none of them replicated. That is an unremarkable description of an ordinary service.
Availability chain
Hypothetical model, not a forecast
Composite availability of a chain of dependencies, each of which may be backed by replicas, and the downtime that composite implies.
It assumes
It ignores
Keep three dependencies in series and keep each one at 99.9%. Find the smallest number of replicas per dependency at which the whole chain finally beats the 99.9% you were about to promise.
Every one of them has to be up for the request to succeed.
One means no redundancy at all.
| Dependencies in series | Downtime, minutes per month |
|---|---|
| 0 | 0 |
| 1 | 43.2 |
| 2 | 86.36 |
| 3 | 129 |
Not there yet — keep moving the controls.
One dependency at 99.9% is 43.2 minutes a month — about 8.6 hours a year on the model's flat thirty-day month, and 8.8 on a real calendar. That is the number from the prediction.
Three of them in series is 99.70%, or 129 minutes a month: a little over two hours, which annualises to roughly 26 hours. You were about to sign a promise for nine hours a year while running a chain that delivers twenty-six, and nothing in the contract negotiation would have surfaced that.
The challenge shows the other direction. Two replicas of each dependency takes the same chain to 99.9997% — about eight seconds a month. One structural change, three orders of magnitude.
Count the chain honestly, and count the things nobody draws. For an ordinary web service the serial path is: DNS, the TLS certificate, the load balancer, the application, the database, and the identity provider if you check tokens against it per request. That is six, not three. Six at 99.9% is 99.4%, which is about four hours a month.
Then shorten the chain, which is the cheapest available nine. Every dependency you can take out of the request path is worth more than any redundancy you can add to one. Cache the identity provider's signing keys and verify tokens locally and the provider leaves the serial path entirely — it becomes something you need at key-rotation time rather than per request. Serve the config from a file baked into the image instead of fetching it. Each removal is a multiplication you stop performing.
Replicate the one thing left that holds state. Managed Postgres with a standby in a second availability zone, with automatic failover. As of late 2026 an Amazon RDS multi-AZ failover typically completes in about 60 to 120 seconds, which means a database failure costs you roughly two minutes rather than the length of a restore — the difference between spending 5% of a monthly budget and spending all of it twice over.
Then spend the budget on deploys, because that is where it goes. Forty-three minutes a month is the whole allowance, and a single migration that takes an exclusive lock on a large table spends half of it in one afternoon. In practice the first nine is bought by connection draining, readiness probes and migrations written in expand-migrate-contract order — not by hardware. Hardware buys the third and fourth nines; process buys the first and second.
And write the promise with its exclusions. What I would actually sign: 99.9% monthly, measured as the share of requests returning a non-5xx response at the load balancer, excluding a declared maintenance window and excluding failures caused by the customer's own configuration, with a service credit rather than a refund. Measured at the load balancer because an uptime monitor polling a health endpoint every minute cannot see a failure that lasted forty seconds, and forty seconds is a meaningful fraction of your allowance.
Check
The condition under which every figure above is optimistic: when dependencies fail together rather than independently.
Multiplying availabilities assumes the three failures are unrelated events. The common real failures are not. A single availability zone takes your primary and your standby if both are in it. One expired certificate takes every service that presents it. One bad configuration push reaches every replica within seconds, because that is what a deployment system is for. A provider's regional control plane going down can leave instances running and nothing able to reach them.
When failures are correlated, two things happen. The composite is no better than the worst correlated component, so the arithmetic above overstates what you have. And replicas buy nothing at all, because the squaring assumed independence and there is none.
Which is why the practical ordering is: remove dependencies first, then spread the replicas you do have across failure domains that do not share power, network or deploy pipeline, and only then argue about the fourth nine. A second replica in the same rack is a rounding error dressed up as redundancy.
Because the allowance divides by ten and the causes do not.
Going from 99% to 99.9% means getting from 3.65 days a year to 8.8 hours, and most teams achieve that by removing one category of problem — usually unmanaged deploys. The next nine, 8.8 hours to 53 minutes, cannot be bought that way, because nothing is left that is one category. It needs automated failover that has been tested under load, a deploy process that cannot take the site down even when the migration is wrong, and alerting fast enough that a human is looking within the first few minutes.
The nine after that, 53 minutes to 5 minutes, means no human is in the recovery path at all, because 5 minutes is less time than it takes somebody to read a page and open a laptop.
So the cost curve is not about hardware. It is about progressively removing people from the loop, and every step there is a change to how the team works rather than to what it runs.