Part III · Chapter 6 · The Window · lesson 2
Twenty-five clean out of twelve hundred proves less than you think
16 min
Part III · Chapter 6 · The Window · lesson 2
16 min
Twelve hundred deploys went to production last year. Every one of them was supposed to have an approval from somebody other than the person who wrote the code.
Somebody picked twenty-five of them at random and checked. All twenty-five had the approval. Nothing was wrong with any of them.
Before you read another line: how many of the other eleven hundred and seventy-five could be missing their approval, and this test would have come back looking exactly the same?
Predict
A clean sample supports two separate sentences, and they are not the same sentence.
The first is the estimate. Twenty-five examined, nothing found, so the measured deviation rate is zero. That is what the sample says.
The second is the bound. Of all the possible populations, which ones would have given a clean draw of twenty-five often enough that we cannot dismiss them? The worst of those is the bound: the largest number of broken items the result does not rule out.
The estimate is zero. The bound is in three figures. Both come from the same twenty-five items, and a report that quotes one as though it were the other is the most common error in this field.
Sampling a real population happens without replacement: once you have examined a deploy, you do not examine it again, and the population you are drawing from gets smaller. That makes the exact answer a finite-population one.
The model carries both. The headline bound is the exact finite-population figure, and alongside it sits the approximation every published sampling table uses — the one that pretends the population is unlimited. They agree closely when the sample is a small slice. Push the sample towards the whole population and they part company spectacularly, which is what the challenge is for.
Sampling confidence
Hypothetical model, not a forecast
The largest deviation rate a sample does not rule out. A clean test estimates the rate at zero; this is the worst rate still consistent with that result, which is a different and much larger number.
It assumes
It ignores
Make the test a census: bring the population down until it is no larger than the number of items examined. Find the state where the exact bound reaches zero and the approximation is still reporting more than one item in ten.
Every item the conclusion is meant to cover, not just the ones that got listed.
| Items examined | Exact, finite population | The approximation in the tables |
|---|---|---|
| 1 | 95% | 95% |
| 2 | 77.6% | 77.6% |
| 3 | 63.1% | 63.2% |
| 5 | 44.9% | 45.1% |
| 8 | 31.1% | 31.2% |
| 12 | 21.9% | 22.1% |
| 20 | 13.8% | 13.9% |
| 25 | 11.2% | 11.3% |
| 30 | 9.33% | 9.5% |
| 50 | 5.67% | 5.82% |
| 80 | 3.5% | 3.68% |
| 120 | 2.33% | 2.47% |
| 200 | 1.33% | 1.49% |
| 300 | 0.83% | 0.99% |
| 500 | 0.42% | 0.6% |
| 800 | 0.17% | 0.37% |
| 1200 | 0% | 0.25% |
Not there yet — keep moving the controls.
One hundred and thirty-four of the twelve hundred. That is the figure, and it is a count of items rather than a rate, which is the form worth carrying: a hundred and thirty-four deploys could have gone to production unapproved last year and this test would look identical.
Read the two curves where they meet the right-hand edge. The exact one falls to the floor — you examined everything, you found nothing, so nothing is broken. The approximation does not fall to the floor, because it was never told the population was finite; at a census of twenty-five items it is still reporting a rate in double figures for a population every member of which was checked and found clean.
That is not a rounding problem. It is the approximation answering a question about an unlimited population when you asked about a bounded one, and it is the figure almost every conclusion in this field is built on.
Check
Twenty-five is the number people take because it appears in a table and looks like enough. If the conclusion you actually want is that nothing above one item in twenty is in play, move the sample size control and watch where the exact curve crosses. It lands around sixty items rather than twenty-five, and getting the bound down to one in a hundred needs several hundred.
Working that direction deliberately — picking the rate you want excluded first and reading off the sample it costs — is the next model in this track.
The question the bound answers is mechanical. Suppose the population really did hold exactly D broken deploys. What is the chance a random draw of twenty-five misses every one of them?
For D of one, the draw has to avoid a single item out of twelve hundred, which is very likely. For D of six hundred, the draw has to avoid half the population twenty-five times running, which essentially never happens. Somewhere between those the probability falls through the threshold the confidence level sets.
The model walks D upwards and stops at the last value where a clean draw is still likely enough not to dismiss. One item either side of the answer, the probability moves by about a tenth of a percentage point — so the boundary is a genuine integer rather than an artefact of rounding, and the honest answer to how many is a whole number of deploys.
Notice what this does and does not assume. It assumes the draw was random. It assumes nothing about why a deploy might be missing an approval, which is why the arithmetic cannot tell an isolated slip from a systematic one.