Track 11 · Noise · lesson 2

The sample you never see

12 min

During a war, a research group is asked where to add armour to aircraft. Armour is heavy, so it can only go in a few places. They examine the planes coming back and map the bullet holes: wings, fuselage, tail.

The obvious answer is to armour where the holes are. The answer that turned out to be right was the opposite — armour the places with no holes, because the planes with holes there did not come back to be measured.

The sample you can see is the sample that survived whatever filter stands between you and it. Everything the filter removed is still relevant to the question, and it is now invisible.

This is not a subtle statistical point. It is the ordinary condition of almost every piece of evidence anyone encounters about money, because the filter — who writes books, who gets interviewed, who is still trading, who is willing to tell you — removes failure far more efficiently than it removes success.

The shape of the problem

Three separate filters are usually operating at once, and they compound.

The outcome filter. Businesses that closed do not have websites. Funds that did badly get merged into other funds and their record goes with them. Traders who blew up have no track record to show. Whatever list you are looking at was assembled from things that still exist.

The willingness filter. People who did well will tell you about it in detail. People who did badly, on the same strategy, will change the subject. So even among the survivors, the visible accounts are skewed.

The interest filter. Nobody publishes the story of the person who followed sensible advice and ended up somewhere unremarkable, because it is not a story. The distribution of published outcomes has both tails removed and the middle removed too.

Predict

A list of the ten most successful people in a field is examined, and eight of them share an unusual habit. What has been established?

The habit is strongly associated with success in that fieldNothing yet — the same habit may be just as common among those who failedThe habit causes success in that fieldThe habit is worth adopting, since the cost is low and the upside is large

A visible winner

Case file

Composite, not a real person

A founder who sold a logistics software business eleven years after starting it

Left a salaried engineering job at thirty-one with about eighteen months of savings and no dependants.

Years to exit: 11
Outside capital: None for 4 years
Proceeds: ~25 years of spending
Similar founders: Unknown, and large

1Picked a dull, unglamorous problem in an industry they had worked in for six years.
2Took no outside capital for the first four years and stayed close to break-even.
3Hired slowly, and kept fixed costs low enough to survive a bad quarter.
4Turned down an early acquisition offer that would have been a comfortable outcome.
5Sold in year eleven, to a buyer that approached them.

A sale producing roughly twenty-five years of the household's spending, and a plausible-sounding story about patience, focus and domain knowledge.

What luck did: The buyer existed and was acquisitive in that particular year. Two years earlier or two years later the same business would have found no buyer at that price, and the founder had no influence over the buyer's strategy or timing.
Who else tried this: A large number of founders did approximately this — dull problem, own money, slow hiring, patient — and are still running businesses that pay a decent salary and will never be sold, or closed them after six years having earned less than the job they left.
Why it is not a template: The decisions here are indistinguishable from the decisions of the founders who did not get the outcome. That is the point of the case: it is not a template, because the part that separated it from the others was a buyer's timing.

Read that case file with the last three fields covered up and it is a lesson about patience and domain knowledge. Read it with them showing and it is a lesson about a buyer's acquisition strategy in a particular year.

Both readings use exactly the same decisions. That is the whole difficulty: survivorship bias is not corrected by looking harder at the survivor.

Where the missing sample is hiding

The correction is procedural rather than attitudinal. Given any claim built on successful examples, three questions recover most of what is missing.

Who else did this? Not "did it work for them" — who attempted the same thing. If nobody knows, the number is not small, it is unmeasured.

Where would the failures be if they existed? Sometimes there is an actual record: closure registers, discontinued products, funds that no longer report. Often there is not, and knowing that the record does not exist is itself informative.

What does the filter select for? Every visible sample got there somehow. The mechanism that made these examples visible usually selects for exactly the property being claimed as a cause.

Not the same thing

The visible sample

The examples that reached you, having passed whatever filter stands between you and the full set.

Measured in: Number of cases you can name, and how compelling each one is
For example: Ten founders on a conference stage.
Chase it alone and: It is selected by outcome, so every property common to the group looks like a cause of the outcome.

The full set

Everyone who attempted the same thing, however it went and whether or not they are still around.

Measured in: Base rate — what share of attempts produced each result
For example: Everyone who started a business of that type in that year, including those who closed in year two.
Chase it alone and: It is usually impossible to assemble, so the honest answer is often an unsatisfying “nobody knows”.

Which is which? Put each one on a side.

The strategies described in books written by people who used them
The list of every company registered in a sector in one year, followed forward
A fund's ten-year performance figures, as currently published

Check

What would actually make the founder case file above into evidence about strategy?

More detail about the decisions, so the strategy can be reconstructed preciselyNine more case files of founders who did the same thing and also succeededA record of everyone who started that kind of business that year, and how each endedThe founder's own account of what they think made the difference

Worth remembering

The visible sample is whatever survived the filter between you and the full set, and the filter removes failure much more efficiently than success.
Outcome, willingness and editorial interest all filter in the same direction, and they compound.
A property shared by successful examples means nothing without knowing how common it is among the unsuccessful.
Survivorship bias is not corrected by studying the survivor more carefully — the missing information is not in the survivor.
The recovery questions are: who else attempted this, where would the failures be recorded, and what does the filter select for.