Finding nothing is not an estimate of nothing

The whole of this lesson is one sentence, and the sentence is boring until you have heard somebody get it wrong out loud in a meeting.

Finding nothing is not an estimate of nothing, and it is not an estimate of anything else either.

The previous lesson produced two numbers from one clean test of twenty-five deploys: a measured rate of zero, and a hundred and thirty-four items that the test could not rule out. People carry exactly one of those two numbers out of the room, and which one they carry decides whether the sentence they say next is true.

Not the same thing

The estimate

What the sample measured. Twenty-five examined, zero deviations, so the best estimate of the deviation rate from this test is zero.

Measured in

Deviations found, divided by items examined. It is a division, and the answer is the answer

For example

Zero of twenty-five. Also zero of five, also zero of six hundred — the estimate is the same in all three and they are wildly different tests

Rests on

Arithmetic on what was actually seen. Nothing probabilistic, nothing assumed

Chase it alone and

On its own it is nearly content-free, because a tiny clean sample and an enormous clean sample produce the identical figure and are not the identical evidence

The bound

The worst rate the result does not rule out. The largest number of broken items the population could hold and still have produced a clean draw often enough not to be dismissed.

Measured in

A count of items, or that count as a share of the population, always with the confidence level attached

For example

A hundred and thirty-four of twelve hundred, at the usual confidence, from the clean test of twenty-five

Rests on

That the sample was drawn at random, and that the confidence level describes the procedure rather than this one conclusion

Chase it alone and

It is routinely quoted as though it were the estimate, which converts a statement about what was not excluded into a claim that something was measured

Both sides accept
Both come out of the same twenty-five items and neither is more honest than the other. A conclusion needs both: what was seen, and what the seeing could not exclude. The error is never using one of them — it is using one of them in the grammar of the other.

Which is which? Put each one on a side.

• Zero deviations in twenty-five items

  The estimateThe bound

• A hundred and thirty-four items could be deviations without changing this result

  The estimateThe bound

• The failure rate is around one deploy in nine

  The estimateThe bound

• If the real rate were one in nine, a clean draw of twenty-five would happen about one time in twenty

  The estimateThe bound

• Nothing in the sample failed the test

  The estimateThe bound

The three ways this gets said wrongly

These are worth memorising as a set, because you will hear all three and the third one is the one you will catch yourself saying.

“The failure rate is about one in nine.” The bound in the estimate's grammar. It asserts a measurement nobody took; the measurement was zero. The repair is one word: could be, not is.

“We are ninety-five per cent sure the control works.” The confidence level detached from the procedure it describes and stapled to the conclusion. A bound computed this way is right nineteen times in twenty across many samples — it is a property of the method, not a probability sitting on this particular answer. The repair is to say what the level attaches to, which is the bound and not the control.

“We found nothing, so the control operated effectively.” The estimate promoted to a verdict, with the bound dropped on the floor. This one is the hardest to resist because it is nearly the sentence the report is for, and because it is said by people who did the work properly. The repair is to keep the second half: we found nothing, and a sample of this size leaves this much unexcluded.

Something people say

“The sample came back clean, so we know the control works. If it did not work, we would have found something in twenty-five items.”

Why it spreads

It is how evidence feels. A clean test is genuinely good news, the work was done properly, and the person saying it has not made an arithmetic mistake — they have made a grammatical one, which nothing in the process catches. It is also the sentence a buyer wants, which makes it the sentence that gets drafted.

The part that is true

A clean result really is evidence, and more of it really is stronger. Twenty-five clean items rules out the catastrophic cases: a control failing half the time would almost certainly have shown up, and after this test you can say that it is not failing half the time. That is a genuine conclusion and it was genuinely earned.

Somebody says: the test was clean, so the deviation rate is under one in a hundred. What specifically is wrong with that?

Nothing — a clean test of twenty-five is consistent with a rate under one in a hundredThe test rules out far less than one in a hundred, so the sentence claims an exclusion the sample size cannot supportThe rate should be quoted as a count of items rather than as a rateNo conclusion can be drawn from twenty-five items at all

The practical version of all of this is a habit rather than a formula. When somebody hands you a clean sample result, ask two questions in this order: how many did you look at, and out of how many. Until both numbers are on the table, there is no sentence to check.

Worth remembering

• Finding nothing estimates the rate at zero; it does not estimate it at the bound, and it does not estimate it at anything in between.
• The bound is the worst rate the result does not rule out, and it belongs in the grammar of could be rather than is.
• A confidence level is a property of the procedure across many samples, not a probability attached to one conclusion.
• The three standard misquotes: the bound stated as the rate, the confidence stated as certainty about the control, and the clean result promoted to a verdict with the bound dropped.
• Stating the bound as a count of items rather than a rate makes it much harder to misquote.
• When handed a clean sample result, ask how many were examined and out of how many, before evaluating any sentence about it.