Read somebody else's report the way they will read yours

For one lesson you are on the other side of the table.

A vendor's sales engineer has sent you a ninety-four page PDF with a note saying it is clean. You are not a security specialist, you have about forty minutes before a call, and whatever you decide becomes your company's position on a dependency you will have for years.

This is the reverse of everything in this track so far, and it is the half most people never practise. It is also the fastest way to understand what your own report will look like to somebody else.

Predict

Forty minutes with a ninety-four page report on a vendor you are about to depend on. Which part changes a buyer's decision most often, and is most often skipped?

Choose one answer

The opinion pageThe boundary: which systems, products and locations the description coversThe testing tables, where the exceptions areThe list of controls the report assumes you operate

Lock in answer

One buyer, doing it properly

Case file

◇ Composite, not a real person

A head of engineering at a sixty-person health-tech company, running the security review of a payments vendor because nobody else was going to

Sent a ninety-four page report by the vendor's sales engineer with a note saying it was clean, and given three days before the contract was due to be signed.

Pages

94

Period ended

11 months earlier

Product in the boundary

Not the one she bought

Customer-side controls

4, none in place

Work transferred to her team

~6 days a year

1. 1Read the opinion page first, saw no exclusion in it, and very nearly stopped there — the note said clean, and the page agreed.
2. 2Checked the period instead, and found it had ended eleven months earlier with nothing covering the gap since.
3. 3Read the description's boundary and found the service named in it was the vendor's enterprise platform; the product she was buying was their newer API offering, which appeared nowhere.
4. 4Found the section listing controls the report assumed the customer operated, and discovered four of them described things her team did not do at all.
5. 5Asked for the current period's report, and for the reports of the two suppliers carved out of the description. Received one of the three.

Signed, with three things written into the contract: a current report each year, notice if the described boundary changes, and named ownership on her side for the four customer-side controls. The last of those turned out to cost her team roughly six days a year that nobody had budgeted for.

What luck did

The sales engineer attached the full report rather than the two-page summary, which is what she would otherwise have been reviewing. She did nothing to cause that and would not have known to ask.

Who else tried this

A great many buyers in exactly this position sign on the strength of a clean opinion page and are perfectly fine, because the vendor is in fact well run and the mismatch between product and boundary never bites. The reading does not make the vendor better. It tells you which risks you are taking, which is a different and less satisfying product.

Why it is not a template

She had three days, a technical background and the standing to ask a vendor for things. A procurement analyst with forty vendors a quarter has none of the three, and for them the honest version is a two-question screen — which product, which period — rather than a thorough read performed badly.

The five checks, in detail

Which product. This is the one that catches people, because a vendor with several products often has one report covering the oldest and largest. The newest service — the one being sold hardest — is frequently the one outside the description. The manifest condition for this whole lesson is exactly that: a report for a product you are not buying is more common than it sounds.

Which period. Two dates on the cover and a third date that matters: today. A period ending eleven months ago is a document about a company that has since hired, reorganised and shipped. Some vendors supply a letter covering the gap between the period ending and now; that letter asserts far less than the report does and is signed by the vendor rather than the auditor.

What was carved out. Where a supplier runs part of the system, their controls may be excluded from the description — and the description says so, usually in a paragraph that reads like housekeeping. Read it as a map of what was not examined. If the carved-out supplier is the cloud provider holding the data you care about, the question is not whether the carve-out was legitimate. It was. The question is what you are relying on and who has looked at it.

The exceptions. Count them last. What matters is whether the cause is specific, whether the scope is bounded, and whether the response names a change with a date — which is lesson nine read from the other direction. A report with three well-described exceptions tells you considerably more about an organisation than one with none.

What the report assumes you do. Go to the back. The customer-side control list is the arrangement's other half, and if you have not implemented it, the system described in the report is not the system you are operating. Each line is a control you now own: who does it, how often, and what shows it ran.

Check

You read the carve-out paragraph and find that the vendor's entire database layer is operated by a cloud provider whose controls are excluded from the description. What does that tell you?

Choose one answer

The report is misleading, since the database is where your data livesNothing — large cloud providers publish their own reports, so the gap is coveredThe chain is two links long, and you have only examined one of themYou should require the vendor to bring the provider inside their boundary

Check answer

›The section that is outside the opinionOptional

Many reports carry a final part labelled as other information provided by management: roadmaps, a response to the exceptions written in a more flattering register, sometimes a description of controls that were not tested at all.

It is clearly labelled and it is routinely read as though it were part of the examined material. The practitioner's conclusion does not extend to it. Which makes it the single most quotable section of the document and the least supported, and it is where a vendor's most reassuring sentences tend to live.

Worth knowing from both sides: as a buyer, check which part of the document you are quoting before you rely on it. As the subject of a report, check what your own marketing has placed back there.

Worth remembering

• Read five things in order: which product, which period and how long ago it ended, what was carved out, what the exceptions were, and what the report assumes you do.
• The customer-side control list transfers real work to the buyer, sits near the back, and is the section that most often creates obligations nobody budgeted for.
• A vendor with several products often has one report covering the oldest, and the service being sold hardest is frequently outside the description.
• A letter covering the gap between the period ending and today asserts far less than the report and is signed by the vendor rather than the auditor.
• A carve-out is legitimate and is a map of what was not examined; where you depend on the carved-out party, that is a second document with its own boundary and period.
• Count exceptions last: the cause, the scope and the dated change tell you more than the number, and a report with three well-described exceptions can be more informative than one with none.
• The section labelled as other information provided by management falls outside the practitioner's conclusion, and it is where the most quotable sentences usually are.