Part VI · Chapter 12 · The Decisions · lesson 5
15 min
Builds on Will it fit on one box?, Say what would change your mind, before you look at options and Ship a number you measured, and say what would reverse it
A decision record from twenty-eight months ago. Four paragraphs, in a repository nobody has opened since. It explains why the team chose a managed search service rather than running their own cluster, and it is well written and entirely true.
You are the person being asked to move off it this quarter, because the search bill has tripled.
The document does not help you at all.
Predict
Here it is. Four paragraphs, in the house style, signed off.
ADR-0014: Search
Status: accepted, 2024-06
We need better search than our database can provide. Users complain that
results are poor and slow, and we have spent two sprints tuning queries
without much improvement.
We considered running our own search cluster and using a managed search
service. Running our own would require expertise we do not have and
operational attention we cannot spare with four engineers.
We will use a managed search service. This lets us ship search this
quarter and keeps operational burden low.
We will revisit if the cost becomes significant relative to our overall
infrastructure spend.
Nothing in it is wrong. It names a problem, names two options, picks one, gives a reason, and even sets a trigger. It would pass review in most organisations and it is better than the median, which is a commit message.
Four questions, which are the only questions the undoer has. What the document answers, and what it does not.
What was being optimised? Answered, implicitly: operational burden on a four-person team. The team is nineteen now, and two of them have run search clusters. The criterion has not been contradicted — it has quietly expired, and the document cannot tell you that, because it never recorded that the four was load-bearing.
What else was on the table, and what did each cost? Named, and unpriced. "Expertise we do not have" is a judgement, not a figure, and there is no way to check whether it is still true or by how much it was ever true.
What was true then that might not be true now? Nothing checkable. Not the corpus size, not the query rate, not the bill, not the relevance complaints. Every one of those was known in the room on the day and none of it survived the writing.
Who disagreed, and on what grounds? Silence — and somebody did. Had the record carried one line, an engineer arguing that the managed price scales with corpus size and the corpus grows with document volume rather than with paying users, the undoer today would be reading a prediction that came true, with a mechanism attached, and would know exactly which number to go and look at.
That is the shape of the failure. The record is a summary of a conclusion, and what the next person needs is the working.
Context, as facts with dates. Team size, corpus size, traffic, what was breaking and how often. The version everybody writes: context as motivation — "we needed better search". Motivation is not checkable. A fact with a date is.
Criterion, one sentence, no "and". The version everybody writes: three criteria sharing one sentence, so the decision cannot be re-derived because nobody knows which one dominated. The track-0 lesson on criteria is the long version; the one thing worth adding here is to name the criteria you deliberately did not use, because the undoer's first instinct will be to assume you were optimising cost.
Options, priced, with a source per figure. Each option in the two currencies that actually trade against each other: money per month, and engineer-time per month. Beside each figure, where it came from — a vendor's price page on a date, a measurement from a spike branch, or somebody's estimate, labelled as an estimate. The version everybody writes: options as prose paragraphs, where the preferred one gets the longest paragraph.
The choice, and what you are knowingly accepting. One sentence each. The version everybody writes: only the upside, which makes the document read as a justification and makes the undoer distrust the parts that are fine.
The dissent. Who disagreed and on what grounds, in one or two sentences, in their framing rather than yours. The version everybody writes: nothing, because the meeting ended in consensus — or because the document is written by the person who won. This is the field that makes a record arguable rather than merely readable, and it is the one that most often turns out to contain the mechanism that eventually reversed the decision.
The reversal condition: a threshold, a duration, and where it gets checked. The version everybody writes: "revisit if it becomes significant", which is an instruction to have the same argument later with less context.
The expiry. A date after which the figures in this record are void, and a name. The version everybody writes: nothing, so the document ages into authority instead of ageing into a prompt to re-measure.
ADR-0014: Search backend
Status: accepted 2024-06 | review due 2025-06 | owner: R. Okonkwo
CONTEXT (facts, with dates)
- 4 engineers. None has operated a search cluster. (2024-06)
- Searchable corpus: 38 GB across 2.1M documents, growing ~4%/month.
Growth is driven by documents per customer, not by customer count.
- Search traffic: ~95 queries/sec at peak, ~12 average.
- Today: relational full-text search. p95 1.9 s. Relevance complaints in
3 of the last 5 customer calls.
CRITERION
Ship usable search this quarter with the least ongoing engineer-time.
Deliberately NOT cost, and NOT relevance ceiling. If you think cost
should have been the criterion, that is the argument to have, and this
line is where to have it.
OPTIONS, PRICED AT TODAY'S SIZE
| $/month | eng-time/month | source of the figures
------------------+---------+---------------------+-------------------------
Managed service | $1,150 | ~0.5 day | vendor price list,
| | | 2024-06, 40 GB tier
Self-hosted, | $420 | ~3 days steady | compute: price list.
3 nodes | compute | + 15 days to set up | 3 days is an ESTIMATE
| | | by someone who has not
| | | run one. Low confidence.
Keep relational | $0 | ~1 day/month tuning | measured over the last
full-text search | marginal| and not converging | two sprints
CHOICE
Managed service. We are buying ~2.5 engineer-days a month for ~$730 a
month, which is well under our loaded cost per engineer-day.
WHAT WE ARE KNOWINGLY ACCEPTING
- The bill scales with corpus size, which grows with documents per
customer. If documents grow 20x and customers do not, the bill grows
20x and the revenue does not.
- Relevance tuning is limited to what the vendor exposes.
- Leaving later means re-indexing the whole corpus.
DISSENT
A. Mehta argued for self-hosting: the managed price scales with the
thing that grows fastest here (corpus) rather than with the thing that
pays us (customers), and the 3-day steady-state estimate is pessimistic
once a cluster is stable. Not accepted, because nobody could operate it
in 2024-06. NOTE: this objection gets stronger with every search-literate
engineer we hire.
REVERSAL CONDITION
Revisit when EITHER the monthly search bill exceeds $5,000, OR two or
more engineers have operated a search cluster, sustained for a quarter.
Checked quarterly in the infrastructure review, by the owner above.
EXPIRY
The cost figures here are void after 2025-06. If you are reading this
later, re-run the options table before arguing with the conclusion.
What that cost to write: roughly forty minutes more than the four paragraphs. Nearly all of it in the options table — looking up the index tier price, and pulling the corpus size and query rate off a dashboard that was already open. The dissent line took one question in the meeting: "Anil, you disagreed — how do you want that written down?"
What it is worth, twenty-eight months later: the undoer reads it in ten minutes and knows four things. Both reversal conditions have fired. The corpus is the input that moved, exactly as predicted. The person who predicted it is still here. And the comparison can be re-run in an afternoon rather than re-argued over a quarter, because every figure says where it came from and the expiry date already told them the figures are stale.
That is the whole difference, and it is not writing skill. It is whether the working was kept.
What would you do
Check
Not a future decision. A decision your team has already made and is currently living with — the database, the deployment platform, the queue, the vendor, the boundary between two services. Something you can check.
Seven lines, one page.
1. CONTEXT Three to five facts with dates. Team size, the sizes and
rates that mattered, what was breaking.
2. CRITERION One sentence, no "and". Plus the criteria you did NOT use.
3. OPTIONS Each one, in dollars a month and engineer-days a month,
with the source of every figure and the word ESTIMATE
next to every guess.
4. CHOICE One sentence, and one sentence on what you knowingly
accepted by choosing it.
5. DISSENT Who disagreed and on what grounds, in their words.
6. REVERSAL A threshold, a duration, and where it is checked.
7. EXPIRY The date these figures go void, and a name.
Two things will be missing, and they are the same two everybody finds.
You will not have a price for the option you rejected. Not an approximate one, not a bad one — none, because nobody priced it at the time; the rejection was a judgement and the judgement was probably sound. Go and get the figure now, even roughly. It takes twenty minutes and it is the difference between a record that can be re-run and one that has to be re-argued.
You will not have the dissent. Either nobody disagreed, which is worth being suspicious about, or somebody did and you cannot now reconstruct their argument in their framing rather than yours. Go and ask them. They remember. That conversation is also the single best predictor you will get of which input is going to move.
When you have both, you will notice something uncomfortable: the record you just wrote is more useful than any of the ones in your repository, and it took under an hour, and the only reason it did not exist is that nobody was asked for it.
For postmortems it was the share of action items that shipped. For decision records it is this: of the records in your repository, how many have had their reversal condition evaluated since the day they were written?
Count it. Not "how many have a trigger" — how many have been checked, with somebody looking at the threshold and writing down what they found. In most repositories the answer is zero, and the records are archaeology: read when somebody new asks why, never read to decide anything.
Zero is not a discipline problem and more templates will not fix it. It is a missing event. Nobody's calendar contains "read the triggers", so nobody does, and a trigger that is never evaluated is indistinguishable from one that was never written. The fix is small and dull: half an hour a quarter, one person, the records whose expiry has passed or whose threshold is close, and three outcomes allowed per record — still holds, expired and re-run, or superseded.
The second-order benefit is the one that changes behaviour. Once triggers get checked, people write better triggers, because a vague one produces a visibly useless half hour. Nothing improves these documents faster than somebody actually trying to use one.
The entire value of the document is in reaching somebody who was not in the room. Where there is no such person, it is overhead, and pretending otherwise is how a useful practice becomes ceremony.
Three engineers who have worked together for four years, deciding something they will revisit next month, in a codebase only they touch: the room is the record. Seven fields for that decision costs an hour and informs nobody, and the honest artefact is a sentence in the commit message. The same goes for anything inside a prototype that will be deleted, and for anything where the reversal cost is smaller than the documentation cost — which, after the first lesson in this track, you now have a way to estimate.
So the crossing is: when the expected number of people who will need the reasoning and were not in the room is less than one, do not write the document. That number is forecastable from things you know — whether the decision touches a boundary other teams depend on, whether anyone is likely to have left before the reversal condition could fire, whether the thing being decided will still exist in two years.
What survives the crossing is smaller and almost free: the reversal condition. Even a three-person team that writes nothing else benefits from one line in the commit message naming the threshold at which this stops being the right answer, because the one thing nobody in the room can rely on is remembering, in eighteen months, that there was a condition at all.
The first lesson in this subject asked whether five hundred people saving notes would fit on one server. The arithmetic said yes, with about a hundred and forty times the room to spare, and nearly everything in the eleven tracks since has been a reason to add something to that one box: a worker, a pooler, a replica, a cache, a broker, a second service, a shard, a second region.
Read it back and the pattern is not that the components are wrong. Every one of them is the right answer somewhere, and most of the length of this subject went on the measurement that makes each one right — the share of a response spent in one outbound call, the connection count, the hit ratio, the number of independent consumers, the blocked releases, the write rate on the busiest tenant.
The pattern is that the measurement arrives later than the appetite. The component is wanted before it is needed, and the work in between is refusing it. That work is unrewarding in a specific way: nobody is promoted for the service they did not build, an architecture that fits on a napkin does not look like expertise, and the refusal has to be made out loud, to people who are not obviously wrong.
This is not an argument for being minimal and it is not a position to hold. It is a claim about sequencing, and it has a real cost on both sides. Teams that add early pay what the first track's case file counted: a large share of a small team, for a long time, with nothing on a roadmap and nothing a customer could see. Teams that refuse too long pay in the outage they had the numbers to predict, and that bill arrives all at once, in public. The only honest asymmetry between the two mistakes is that one of them is visible.
So the thing worth taking out of here is not a default. It is being able to say out loud which measurement you are waiting for. "We shard when sustained write throughput on the busiest tenant passes two thirds of what the load test measured, for a week" is a sentence somebody can hold you to, and it is also a sentence that tells you what to instrument. "We shard when we need to" is neither.
None of which makes you right. This track opened by describing authority that comes from having been right before, and that is not something reading produces. It comes from writing down what you expected, shipping, and going back to look — and the going back is the part that gets skipped, because by then you are busy, the prediction is either embarrassing or irrelevant, and nobody is asking. The records are how you make yourself go back. That is the real reason they are the last thing here, and the reason the final exercise was a decision you have already made: it is the only kind you can check today.
Three things will still hold when every product named in this subject has been renamed. The arithmetic. The fact that a number beats an opinion in any room where somebody is willing to show their working. And the consequence of the second one, which is that it cuts against you exactly as often as it cuts for you — the whole price of working this way, and the only part nobody can make easier.