Part III · Chapter 6 · The Window · lesson 4

Work it the other way: how many to rule out one in twenty?

14 min

Builds on Twenty-five clean out of twelve hundred proves less than you think

Four hundred and twenty change tickets went through the period. Your reviewer will test some of them, and somebody has to say how many.

The previous lesson worked forwards: here is the sample, here is what it fails to rule out. This one works backwards, which is the direction anybody planning the work actually needs. Pick the rate you want excluded. Pick how sure you want to be. The sample size falls out.

Predict

Four hundred and twenty tickets. You want to come out of this able to say that nothing above one ticket in twenty is in play. How many do you have to examine, finding nothing, to get there?

Twenty-five — the number that appears in the sampling tablesAround sixtyAround a hundred and fiftyAll four hundred and twenty — sampling cannot support a claim like that

Inverting the bound takes one line of arithmetic.

A clean test of n items leaves a bound that falls roughly as one over n. So for any rate you want to exclude, there is a smallest n that gets the bound below it, and it depends on only two things: the rate, and the confidence.

Not the population. That is the surprising part — the required sample is essentially the same whether you are drawing from four hundred items or forty thousand, which is why the published tables can be printed without a population column at all.

What the population decides is whether the answer is reachable. When the sample a target needs is larger than the population holds, sampling cannot get you there, and the only remaining route is examining every item.

Sampling confidence

Hypothetical model, not a forecast

The largest deviation rate a sample does not rule out. A clean test estimates the rate at zero; this is the worst rate still consistent with that result, which is a different and much larger number.

It assumes

The sample size solved for assumes the larger sample also comes back clean.
The inversion is the with-replacement one, which is what the published tables use.
Each item independently either passes or fails the test, at one constant rate.
Deviations occur independently of each other and of when in the period they fall.
The bound is one-sided, and the confidence level describes the procedure rather than this sample.
Nothing here says the sample was drawn at random; the arithmetic assumes it and cannot check it.

It ignores

Whether the sample was random at all — a convenience sample, the first twenty rows, the ones an owner could find, or anything picked for being easy carries no confidence bound whatsoever, and this is the assumption that fails most often in practice
Any reading of the bound as an estimate. The estimate from a clean test is zero; the bound is the largest rate that result fails to rule out, and the two are not the same claim. Saying the rate could be 11% is right; saying it is around 11% is not
Whether the tests across controls really are independent, which Sidak assumes and which is false whenever controls share an owner, a system or a weekend — the chance that at least one bound is wrong is then higher than the figure reported here, not lower
Stratification and risk-weighted selection, which change what the bound is a bound on
Severity — one deviation that lost the keys and one that misspelled a name count the same here
Whether a deviation found was an isolated slip or the visible edge of a systematic failure, which no sample size can tell apart
Which period the sample was drawn from, which is what evidence-coverage is for
That the binomial figure reported for comparison is the one in every published sampling table, and is the one most conclusions in the field are actually built on
That the sample size reported for a target rate comes from the with-replacement inversion rather than from the exact arithmetic behind the headline, so the two disagree — at a population of 420 and a target of one in twenty it asks for 59 items where the exact crossing is 53. The gap widens as the sample becomes a large share of the population, which is exactly where this subject's examples live, and the conservative direction is the honest one to be wrong in for a sample size

Four hundred and twenty tickets is all you have. Tighten the rate you want excluded, or raise the confidence you want it excluded at, until a clean test cannot get you there from this population at all — the sample needed is larger than the population holds.

Confidence level95%

How often a bound computed this way is right. It is never 100.

Deviation rate to rule out5%

The rate the conclusion wants its upper bound to sit below.

Largest deviation rate this result does not rule out: 0.11
The same rate in percentage points: 10.952%
Items that could be deviations without changing this result: 46
Deviation rate seen in the sample, which is the estimate: 0
The same rate under the with-replacement approximation: 0.113
Items examined, after clamping to the population: 25
Deviations counted, after clamping to the sample: 0
Share of the population examined: 0.06
Items nobody looked at: 395
Items a clean test needs to push the bound below the stated rate: 59
The stated rate is unreachable by sampling this population: 0
Chance at least one bound in the engagement is wrong: 0.05
Per-control confidence needed to hold the whole set to the stated level: 95%
Items a clean test needs once the whole set is held to that level: 59

Bound a clean test of this size supports
The rate you want excluded

A falling curve crossing a flat line: the sample size where the bound a clean test supports finally drops under the rate the conclusion wanted excluded.
Items examined	Bound a clean test of this size supports	The rate you want excluded
1	95%	5%
2	77.4%	5%
3	62.9%	5%
5	44.8%	5%
8	31%	5%
12	21.7%	5%
20	13.6%	5%
25	11%	5%
30	9.05%	5%
50	5.24%	5%
80	3.1%	5%
120	1.9%	5%
200	0.95%	5%
300	0.48%	5%

Not there yet — keep moving the controls.

Read the sample size the model solves for as you move the two controls.

At one item in twenty, with the usual confidence, it is fifty-nine. Tighten the target to one in a hundred and it is two hundred and ninety-nine — five times the sample for a fifth of the rate, which is the ratio worth carrying in your head. Raise the confidence instead and keep the target at one in a hundred, and it passes four hundred and fifty, which is more tickets than the period contained.

That last state is the one worth sitting with. The conclusion was not expensive; it was unavailable. No sampling plan reaches it from four hundred and twenty items, and the only honest routes are to examine every ticket, to accept a looser target, or to accept less confidence in it.

Check

Your population is forty thousand deploys rather than four hundred. How does that change the sample needed to rule out anything above one in twenty?

It scales with the population, so roughly a hundred times as manyBarely at all — the required sample is almost the sameIt falls, because a larger population makes each item more representativeIt cannot be computed without knowing the population

The approximation is doing you a small favour here

The number the model solves for is the with-replacement one, because that is the formula the published tables are built from. It is slightly conservative: at this population it asks for fifty-nine items where the exact finite-population arithmetic reaches the same bound at fifty-three.

Six items of extra work for a figure that is easier to defend because everybody else computes it the same way. That is usually a trade worth taking, and it is worth knowing which direction the error runs — towards more sample, not less.

Where this stops holding

The whole inversion assumes deviations occur independently: that whether ticket number two hundred has its approval tells you nothing about ticket two hundred and one.

Real deviations cluster. A rule gets misconfigured on a Tuesday and every change that week goes through unapproved; somebody covers for a colleague for a fortnight and does it differently. The failures arrive in runs, and a random sample of fifty-nine spread across the period is poorly placed to see a run: it will probably catch one item from the bad fortnight, which looks like an isolated slip rather than the two weeks it came from.

That is not an argument for a bigger sample. It is an argument for looking at when the deviations you find fall, which is what the evidence-coverage model in track 4 is for, and for treating one deviation as a question about a period rather than as a count of one.

Worth remembering

Inverting the bound needs only two inputs: the rate you want excluded and the confidence you want it excluded at. The population does not enter.
Ruling out anything above one in twenty takes around sixty clean items; one in a hundred takes around three hundred.
Halving the rate you want excluded roughly doubles the sample, and raising the confidence multiplies it again.
The population decides reachability: when the sample a target needs exceeds the population, sampling cannot support that claim and only a census can.
The figure the tables give is the with-replacement one, which asks for a few more items than the exact finite-population arithmetic needs.
The inversion assumes deviations occur independently; real ones cluster in time, and a spread sample reads a bad fortnight as one isolated slip.