Part VI · Chapter 10 · Models in Production · lesson 4
A test suite for output with no right answer
14 min
Builds on Say what would change your mind, before you look at options
Part VI · Chapter 10 · Models in Production · lesson 4
14 min
Builds on Say what would change your mind, before you look at options
The pull request is open and CI is green, which means nothing: the only assertion in the test file checks that the response is a non-empty string. You changed two sentences of the prompt to fix a complaint about tone, and you cannot say whether the replies got better or worse. Review has been open four days, because nobody can work out what approving it would mean.
Predict
Most of what you want to assert about a generated output does have a right answer. It parses as the schema you asked for. It cites only documents that were in the context. It never contains a customer's account number. It is under the length the interface can display. It refuses when the policy says refuse.
Those are ordinary tests. They are deterministic, they run in milliseconds, and they catch most real breakage, because most production incidents in these features are structural rather than aesthetic.
What is left over is the part with no right answer: whether the summary is a good summary. That residue needs a different instrument — inputs with judgements attached, scored by a rubric, plus a head-to-head against the prompt currently in production.
Write the exact layer first. It is cheap, it is boring, and it is where the bugs are.
Layer one: properties with right answers. Written as ordinary assertions in whatever test framework you already use. Schema validity, every citation resolving to a document that was actually in the prompt, no personal data in the output that was not in the input, length bounds, required refusals, and language matching the request. Deterministic, so a failure is a failure and not a mood. Run them on every input in the eval set, on every pull request.
Layer two: a graded set. Thirty to a hundred real inputs with a reference answer or a rubric for each, scored by a model acting as a grader, reported as an average with the failures listed individually. The absolute number matters less than its movement: the useful question is whether this prompt scores worse than the last one on the same set with the same grader. Keep the set in the repository so a reviewer can see what the feature is supposed to do.
Layer three: pairwise preference. For each input, generate with the old prompt and the new prompt, show both to a grader without saying which is which, and ask which better satisfies the brief. Report a win rate. This is the layer that answers the pull request, and it is markedly more reliable than absolute scoring because the grader is comparing two things in front of it rather than calibrating against a scale it is inventing.
Two mechanical details decide whether layer three works at all. Randomise which output is shown first, because graders have a measurable preference for one position, and run each pair in both orders if you can afford it. And collect the grader's one-line reason, because the reasons are what you read when the win rate says something you do not believe.
What would you do
An internal tool drafts a first-pass reply to a customer support ticket. An agent reads the draft, edits it, and sends it. Roughly nine in ten drafts are sent with some editing.
The team ships prompt changes about twice a week. There is no eval set. The current test file asserts that the response is non-empty and that it does not contain the string "As an AI". Two agents have complained that replies got blunter after a change nobody can now identify.
For layer one, the test framework you already have. Pytest, Vitest, JUnit. These are ordinary assertions over a JSON payload and they do not need a framework with "eval" in its name. Resist the pull towards a dedicated tool here; the whole value is that these tests run in the same place and fail in the same way as everything else you test.
For layers two and three, promptfoo if you want it in the repository. It is a configuration file next to the code, it runs in CI, it does pairwise and rubric grading, and the results are a file somebody can diff. The tradeoff is that it is a thin layer over your own discipline: it will happily run a bad eval set forever and report a stable number.
Braintrust or LangSmith if the pain is traces and history rather than CI. Both are hosted, both keep a record of every run against every prompt version, and both are far better than a repository file at answering "when did this score move". The cost is that your eval set now lives in somebody else's product, your test results are not in the pull request by default, and a production sample with customer text in it has left your boundary.
Inspect if the checks themselves are the hard part. It is built for specifying graders and scorers properly rather than for wiring a pipeline, and it is the right reach when you are arguing about what the rubric should say. It expects more of you in return.
Whichever you pick, the eval set is the asset and the tool is not. Fifty real inputs with judgements attached, in version control, is worth more than any of these products, and it is the only part you cannot buy or migrate your way out of.
Asking a grader for an absolute score out of ten requires it to hold a scale in its head and apply it consistently across runs, prompt versions and months. It does not. The same output scores differently depending on what preceded it, how the rubric is worded, and which grader version you are on, and none of that drift is visible in the number.
Asking which of two outputs is better removes the scale entirely. Both candidates are in front of the grader, the comparison is local, and the result is far more stable across runs. It is also the question your reviewer has.
Pairwise has three limits worth naming. It tells you about direction and not about level — a long sequence of changes each winning narrowly can drift somewhere nobody wanted, and only the absolute layer notices. It is blind to shared failure: if both candidates miss the same requirement, the comparison is contented. And it costs two generations and a grading call per input, which makes the set size you can afford on every pull request smaller than the set size you want.
So keep both. The graded set for level and trend, the pairwise run for the merge decision, and the exact layer underneath both because it is the only one that cannot be talked out of a verdict.
Check
It stops when something downstream demands exact values anyway.
If the output is parsed into a payment amount, a date for a calendar entry, a database identifier or an argument to a function that changes state, then graded evaluation is not the right instrument for the part that matters. That part has a right answer, it belongs in layer one, and the correct test is equality against an expected value on a fixed set of inputs — the same test you would write for any other parser.
The useful move here is to narrow what the model produces until the exact layer can cover the consequential part. Have it emit a structured object rather than prose containing a number. Constrain it to a choice from a list rather than a free-text label. Resolve its citation to a document identifier that must exist. Each of those converts a judgement into an assertion, and a feature where every consequential field is an assertion needs the graded layers only for the prose wrapped around them.
The two-layer split also tells you where to spend. If the exact layer covers everything that can hurt a user, the graded layers are a product-quality instrument and can be sampled, run nightly, or run on a subset. If the exact layer covers nothing, every regression is a judgement call, and you are going to need the human calibration run from the debrief more often than feels reasonable.