Part III · Chapter 6 · The Window · lesson 11
Run the bound on your own control, and say the sentence correctly
14 min
Builds on Twenty-five clean out of twelve hundred proves less than you think
Part III · Chapter 6 · The Window · lesson 11
14 min
Builds on Twenty-five clean out of twelve hundred proves less than you think
Pick one control in your own register that you marked as operating. Preferably the one with the most instances, because that is the one somebody will sample.
You need three numbers about it. How many times it happened across your period — the whole population, not the ones that got listed. How many of those somebody would examine. And how many of those came back with something wrong.
The third number can be zero. Zero is the interesting case and it is the one this whole track has been about.
Sampling confidence
Hypothetical model, not a forecast
The largest deviation rate a sample does not rule out. A clean test estimates the rate at zero; this is the worst rate still consistent with that result, which is a different and much larger number.
It assumes
It ignores
Every item the conclusion is meant to cover, not just the ones that got listed.
Items in the sample that failed the test. Zero is the clean-test case.
How often a bound computed this way is right. It is never 100.
The rate the conclusion wants its upper bound to sit below.
How many separate controls carry a bound of their own in the same engagement.
| Items examined | Exact, finite population | The approximation in the tables | The rate you want excluded |
|---|---|---|---|
| 1 | 95% | 95% | 5% |
| 2 | 77.6% | 77.6% | 5% |
| 3 | 63.1% | 63.2% | 5% |
| 5 | 44.9% | 45.1% | 5% |
| 8 | 31.1% | 31.2% | 5% |
| 12 | 21.9% | 22.1% | 5% |
| 20 | 13.8% | 13.9% | 5% |
| 25 | 11.2% | 11.3% | 5% |
| 30 | 9.33% | 9.5% | 5% |
| 50 | 5.67% | 5.82% | 5% |
| 80 | 3.5% | 3.68% | 5% |
| 120 | 2.33% | 2.47% | 5% |
| 200 | 1.33% | 1.49% | 5% |
| 300 | 0.83% | 0.99% | 5% |
| 500 | 0.42% | 0.6% | 5% |
| 800 | 0.17% | 0.37% | 5% |
| 1200 | 0% | 0.25% | 5% |
One sentence, in your own words, about your own control. It has to contain two things: what the test found, and what it leaves open. Write it before you read the next section.
Most people produce something close to this shape on the first attempt:
Of the N instances in the period, M were examined and K had an exception. The estimate from that is K over M; a sample of that size leaves up to J of the N instances that could have failed without this result looking any different.
Clumsier than what goes in a report, and impossible to misquote, which is the trade.
These are the three from earlier in this track. Hold your own sentence against each one and see whether you have drifted into it.
The bound in the estimate's grammar. If your sentence says the rate is anything other than what you measured, it has turned a statement about what was not excluded into a measurement nobody took. The repair is two words: could be.
The confidence level attached to the control. If your sentence says you are some per cent sure the control works, the level has come off the procedure it describes and landed on the conclusion. The level attaches to how the bound was computed, not to whether the control is any good.
The bound dropped on the floor. If your sentence stops after the clean result, it is the estimate promoted to a verdict. This is the one almost everybody writes, because it is nearly the sentence the report exists to produce.
Check
Now put the control and its evidence into your register, with the period you declared in the previous lesson, and let the referee read the whole thing back rather than just this one control.
Your own register
Nothing here is graded, and nothing here leaves your browser unless you are signed in. A filled-in register is an inventory of your known weaknesses, so keep the real one where your auditor can see it — this is somewhere to think.
One or two sentences, in your own words. The test is whether you would read it out to a customer.
Six facts about you
These decide which controls are even yours to argue about. The honest answer usually removes more than it adds.
The period you are claiming
Everything in here is measured against these two dates. Typed as year-month-day, so the same thing is stored whatever your browser thinks the date format is.
two dates, the second after the first
What could go wrong
A sentence with a victim in it, not a category. There is no score field here and there is nowhere to put one: the two bands are never multiplied.
Once a year or so
A customer notices
What will you do about it?
What you could hand over
Only what already exists, with a date on it. A policy is real evidence of design and no evidence at all of operation, which is why that is the second question.
What does it show?
What does not line up
Nothing yet, because there is nothing in here yet. An empty workspace has no contradictions and tells you nothing.
These are contradictions between things you wrote. Nothing here can tell you whether any of it is enough — that is a judgement, and it moves with the auditor, the boundary and the year.
What appears there is not a verdict on your sampling. It is the other half of the problem: a bound is about the instances you did not examine, and the referee is about the rows that disagree with each other. A control can carry a perfectly honest bound and still contradict your period, your risks or your own notes about who owns it.
Every bound in this track assumes the sample was drawn at random from the whole population. Look at where your population number actually came from.
If it is a count from a system — every deploy the pipeline recorded, every ticket in the tracker — the assumption is roughly sound and the arithmetic means what it says.
If somebody assembled the list by hand, it is not a population. It is a list of the instances that person could find, which systematically excludes the ones with no artefact attached — and those are precisely the instances most likely to be deviations. A clean sample of twenty-five drawn from a hand-built list of the well-evidenced ones carries no bound on anything, and the arithmetic has no way to know it was handed a curated list.
This is why a reviewer asks how the population was built before asking for the sample, and it is the single most useful question in this whole track. The honest version of your sentence may have to start a step earlier: here is the population, here is how it was derived, and here is what it would not have contained.