Part III · Chapter 6 · The Window · lesson 5

Ten controls, each at ninety-five per cent, and the trap in the middle

15 min

Builds on Twenty-five clean out of twelve hundred proves less than you think

Ten controls. Each one sampled properly, each one coming back clean, each one carrying a bound computed at the usual confidence level. Ten sound conclusions, in one report, about one organisation.

Every individual test in that paragraph was done correctly. The set of them is in much worse shape than any one of them, and nothing in the process anywhere mentions it.

Predict

Ten controls, each tested to a bound at ninety-five per cent confidence. What is the chance that at least one of those ten bounds is wrong?

One in twenty — that is what ninety-five per cent meansAbout one in ten, because errors partly cancelRoughly two in fiveIt cannot be stated without knowing how the controls relate to each other

Confidence levels do not add and they do not average. They compound.

A bound at ninety-five per cent is wrong one time in twenty. Two independent bounds are both right nineteen times in twenty, times nineteen in twenty. Ten of them, and the chance that all ten are right has fallen a long way.

So the quantity the report is implicitly claiming — that none of its conclusions is wrong — is a different and much weaker claim than the one each page carries.

Holding the whole set to one-in-twenty instead means tightening each individual bound, and the sample size needed rises with it. That is the trade: either the set is weaker than it looks, or every sample in the engagement gets bigger.

Sampling confidence

Hypothetical model, not a forecast

The largest deviation rate a sample does not rule out. A clean test estimates the rate at zero; this is the worst rate still consistent with that result, which is a different and much larger number.

It assumes

Each control is tested separately, with its own sample and its own bound.
Composition across controls assumes the tests are independent of one another.
The correction reported is Sidak, which is exact under independence and conservative without it.
Every control in the set is held to the same confidence level.
The sample size reported for the set assumes each of those larger samples also comes back clean.
Nothing here changes the bound on any individual control; it changes how much of the set to believe.

It ignores

Whether the sample was random at all — a convenience sample, the first twenty rows, the ones an owner could find, or anything picked for being easy carries no confidence bound whatsoever, and this is the assumption that fails most often in practice
Any reading of the bound as an estimate. The estimate from a clean test is zero; the bound is the largest rate that result fails to rule out, and the two are not the same claim. Saying the rate could be 11% is right; saying it is around 11% is not
Whether the tests across controls really are independent, which Sidak assumes and which is false whenever controls share an owner, a system or a weekend — the chance that at least one bound is wrong is then higher than the figure reported here, not lower
Stratification and risk-weighted selection, which change what the bound is a bound on
Severity — one deviation that lost the keys and one that misspelled a name count the same here
Whether a deviation found was an isolated slip or the visible edge of a systematic failure, which no sample size can tell apart
Which period the sample was drawn from, which is what evidence-coverage is for
That the binomial figure reported for comparison is the one in every published sampling table, and is the one most conclusions in the field are actually built on
That the sample size reported for a target rate comes from the with-replacement inversion rather than from the exact arithmetic behind the headline, so the two disagree — at a population of 420 and a target of one in twenty it asks for 59 items where the exact crossing is 53. The gap widens as the sample becomes a large share of the population, which is exactly where this subject's examples live, and the conservative direction is the honest one to be wrong in for a sample size

Start at one control and raise the number of controls tested this way. Find the point at which the chance that at least one bound in the engagement is wrong passes one in two — a report more likely than not to contain a wrong conclusion.

Confidence level95%

How often a bound computed this way is right. It is never 100.

Controls tested this way1

How many separate controls carry a bound of their own in the same engagement.

Largest deviation rate this result does not rule out: 0.112
The same rate in percentage points: 11.167%
Items that could be deviations without changing this result: 134
Deviation rate seen in the sample, which is the estimate: 0
The same rate under the with-replacement approximation: 0.113
Items examined, after clamping to the population: 25
Deviations counted, after clamping to the sample: 0
Share of the population examined: 0.021
Items nobody looked at: 1,175
Items a clean test needs to push the bound below the stated rate: 59
The stated rate is unreachable by sampling this population: 0
Chance at least one bound in the engagement is wrong: 0.05
Per-control confidence needed to hold the whole set to the stated level: 95%
Items a clean test needs once the whole set is held to that level: 59

Not there yet — keep moving the controls.

Fourteen controls. At fourteen, a report in which every single test was performed correctly is more likely than not to contain at least one bound that is wrong.

Fourteen is not an unusual number of controls to test. A small register tests more than that.

Keep going to forty and the chance climbs to around seven in eight. Nothing has gone wrong at any point: forty samples were drawn properly, forty bounds were computed properly, and the set they form is a near-certainty to contain a mistake somewhere in it.

What it costs to fix

Read the two figures the model reports once the number of controls is up: the per-control confidence needed to hold the whole set to the stated level, and the clean sample that level then costs.

At ten controls, the per-control level required is well above ninety-nine per cent, and the clean sample for the same target climbs from fifty-nine items to a hundred and three. Nearly double the work on every control in the engagement, for exactly the same conclusion about exactly the same controls.

This is the honest price of a report whose conclusions hold as a set. Almost nobody pays it, and the alternative is not dishonest either — it is to understand that each conclusion stands on its own and the collection was never tested as a collection.

Check

Your reviewer tests twelve controls and reports twelve clean results. What is the defensible thing to say about the set?

All twelve controls operated without exception at ninety-five per cent confidenceEach result stands on its own at the stated level; the twelve together carry no such level, and the chance one of them is wrong is much higherThe results are unreliable, because twelve tests at that level cannot be trustedThe twelve bounds should be averaged into one figure for the register

Where this stops holding

The correction the model reports assumes the tests are independent: that one control's bound being wrong says nothing about another's.

They are not independent, and the ways they fail together are obvious once named. Two controls with the same owner fail in the same fortnight that owner was away. Three controls evidenced out of the same system share that system's retention and its clock. Controls that run on a release cadence all skip the same frozen December.

The uncomfortable part is which direction that pushes the answer. Correlated tests mean the chance that at least one bound is wrong is higher than the independent arithmetic says, not lower — and how much higher depends on structure nobody has measured. So the figure in the model is the optimistic one, and the true figure for a real engagement is unknown.

Which leaves a sentence rather than a number: the conclusions in a report hold one at a time, they were never tested as a set, and the controls most likely to be wrong together are the ones that share a person or a system.

Why nobody does thisOptional

Three reasons, and only one of them is bad.

The first is that the reader of the report asks about controls one at a time. A buyer worried about access review reads the access review page, and for that reader the individual bound is the right object.

The second is that the correction is expensive in a very visible way. Nearly doubling every sample is a line item somebody has to approve, and the benefit is a conclusion that already felt established.

The third is that nobody has ever been asked for it. The composition arithmetic is not in the sampling tables, not in the planning templates, and not in the questions buyers send. An omission that nobody is ever challenged on stays omitted, which is the one reason of the three that does not survive being said out loud.

Worth remembering

A confidence level attaches to one bound. Ten bounds at that level give roughly a two-in-five chance that at least one of them is wrong.
At fourteen controls a report in which every test was done correctly is more likely than not to contain a wrong conclusion.
Holding the whole set to the stated level means tightening every individual bound, which nearly doubles each sample in the engagement.
Nothing about the composition changes any individual control's bound; it changes how much of the collection to believe.
The correction assumes independent tests, so the reported figure is a floor — correlated controls make it worse, by an amount nobody has measured.
The controls most likely to be wrong together are the ones sharing an owner, a system or a release calendar.