Part III · Chapter 5 · Evidence · lesson 11
When you buy a platform that promises the evidence
13 min
Builds on Automating it changes what you have to prove, not how much
Part III · Chapter 5 · Evidence · lesson 11
13 min
Builds on Automating it changes what you have to prove, not how much
There is a whole category of product now whose pitch is that the evidence problem is solved. You connect it to your cloud accounts, your identity provider, your repositories and your human resources system; it installs an agent on the laptops; it runs a few hundred checks on a schedule and draws a grid.
The grid is mostly green. That is not a criticism — most of what it checks really is fine, and it rechecks it every day, which is more often than anybody was checking before.
The question this lesson is about is narrower: for which of your controls is that green an artefact, and for which is it a picture of something adjacent?
Predict
Two different objects, and the grid draws them in the same colour.
A check. A question with a machine-readable answer, asked repeatedly. Is multi-factor enforced on this directory? Does this bucket allow public reads? Has this laptop got disk encryption on? Green here is a real, dated, independently produced artefact about a setting — stronger than a screenshot, for exactly the reasons the screenshot lesson gave.
A control. Something the organisation does, which may or may not have a machine-readable surface. Somebody with standing reviews a list and removes what they cannot justify. Somebody answers the alert. Somebody restores the backup and finds out whether it works.
Where a control reduces to a check, the platform's output is evidence and good evidence. Where it does not, the green is about the check, and the control is unevidenced in a window that looks fully covered.
Not a complete list, and these are the ones where the mismatch is sharpest.
The restore. A platform can see that a backup job reported success and that a retention setting is configured. It cannot restore anything, which means the one thing the control exists to establish — that data comes back out into a working system inside an agreed time — has no machine-readable surface at all. Green on backups and no restore test is the most common version of this in the whole category.
The alert somebody answered. It can see that a monitor exists and is enabled. Whether the alert reached a person who was awake and authorised, and what they did, lives in acknowledgement times and triage notes. Some platforms read those from the paging tool; most read whether the paging tool is connected.
The access review. It can read current group membership continuously, which is genuinely useful and is a different claim. The control is that a person who knows what the role grants looked at the list and removed what they could not justify. The artefact for that is the review record with a name on it, and a membership snapshot is compatible with no review ever having happened.
Who approved the change. It can read the branch protection settings, which is design evidence and is exactly where the policy and the configuration usually disagree. Whether any individual change was approved by somebody other than its author is in the merge history, which is a different query and is the one that finds the self-approvals.
Who accepted the risk. Nothing machine-readable exists. A person with the standing to carry a consequence said they would carry it. No check has an opinion about that.
What the supplier assessment concluded. It can confirm a questionnaire was completed and a document uploaded. Whether anybody read the answers, and what they decided about the ones that were bad, is not in the file's existence.
Something people say
“The platform says we are 94% compliant, so the rest is paperwork.”
The number counts checks and is read as counting controls. Those populations are not the same shape: the checks concentrate where settings are readable, and the remaining open items concentrate where a human has to do something and be seen to have done it. So the last stretch is not the same kind of work as the first — it is the only kind the tool could never do.
There is a second problem with a figure like that, which is what it does once it exists. It is a single number next to a word about readiness, which makes it the most screenshottable object in the whole programme. It gets pasted into a customer deck, and the sentence that travels with it loses the word checks somewhere in the second retelling.
None of the above is an argument against buying one, and a lesson that left it there would be useless.
The real value is in three places. Daily rechecking of configuration catches drift that any periodic manual process misses, and drift is the failure mode nobody has a good answer for. The artefacts it produces are dated by something other than you, which is the distinction from the dating lesson and it applies at scale. And the collection is a by-product of the tool running, which is the property the whole track has been arguing for.
Three questions worth asking before the trial ends, none of which appear on a feature list.
Check