Part III · Chapter 5 · Evidence · lesson 2
Two different things evidence can show, and only one is hard
12 min
Builds on Designed, or running? Decide for each one
Part III · Chapter 5 · Evidence · lesson 2
12 min
Builds on Designed, or running? Decide for each one
Somebody has asked you to show that changes to production get approved by a second person. You send three things, because you have three things.
All three are real. All three are relevant. Two of them answer a question nobody asked.
Predict
Evidence can show two things, and the registry carries the distinction as a field on every shape of evidence rather than as a note in the margin.
Design — the control is set up, somebody decided how, and the arrangement is what you say it is. A written rule, a diagram, a settings export, a signed approval of the approach.
Operation — it happened, on dates, through the period you are talking about. A completed review with a name on it. A ticket with an approver. A log of the job that ran.
Design evidence can be produced this afternoon for a control that has never once run. Operation evidence can only be gathered while the control is running. That is the whole difficulty, and it is a difficulty about time rather than about effort.
The registry names five things a piece of evidence physically is: a document stating intent, a record the control itself threw off, a system setting exported, something a tool generated on a schedule, and a person asserting something with their name attached.
Those five are useful for finding the artefact. They do not tell you what it shows, which is why they are a separate field. A report generated monthly can speak to operation. A report generated once, describing how things are meant to work, speaks to design. The same tool, the same file format, two different claims.
The pairing that catches people is the settings export. It feels like proof that something is happening, because the setting is the mechanism. What it actually says is that on the day you exported it, the mechanism was configured a particular way. To say anything about the period you need either the same export repeated through it, or records of the mechanism doing its job.
Something people say
“We have the policy, so we have the control.”
The policy says what is meant to happen. Nothing in it is a record of anything happening, and no amount of care in drafting changes that. Two organisations with identical policies, one of which performs the control every month and one of which has never performed it, produce identical design evidence.
Which means the interesting question about any policy is what it causes to exist. A rule that says access is reviewed quarterly, with no named reviewer, no system of record and no artefact at the end, has described a control whose operation is unevidenceable by construction. The design document is where the operation problem is created or avoided, and almost nobody reads it that way.
The consistency engine encodes exactly this. Mark a control as running, attach only things whose field says design, and it tells you that everything linked evidences design rather than operation, then asks what the control produced, on a date, the last time it ran. It does not say the policy is bad. It says the policy is answering the other question.
Check