Quality dropped for six weeks and nothing reported anything

Nobody filed a bug. The dashboards are flat — latency unchanged, error rate flat at zero, cost per call steady, every test green, no alert in the channel. Six weeks after a change nobody connected to it, an account manager forwards a customer email saying the assistant has become useless. You go looking, find the decline began on the day of a routine dependency upgrade, and discover that every signal you own was structurally incapable of seeing it.

Predict

A retrieval-backed answering feature degrades gradually over six weeks with no error, no latency change and no cost change. Which single measurement would most likely have caught it in the first week?

Choose one answer

A nightly run of the eval set, with the score plotted over timeA signal from inside the product about whether the answer was usedA distribution check on the retrieved material going into the promptAn alert when the quality score crosses a threshold

Lock in answer

Case file

◇ Composite, not a real person

A business-software company, about sixty engineers. An in-product assistant answers customer questions from the company's own documentation and help centre, retrieving passages from an index and answering from them. Roughly forty thousand questions a month, and about one in four conversations ends in a ticket to a human instead.

A routine dependency upgrade in the ingestion pipeline, shipped on a Tuesday with eleven other version bumps in the same pull request. The upgrade changed the default behaviour of the text splitter: where it had previously broken documents on heading boundaries, it now broke them at a fixed character count, cutting through sentences and separating headings from the text beneath them.

Weeks to detection

6

Dashboards that moved

0

Weeks from the right question to the cause

0

Post-incident changes that were instrumentation

3 of 3

1. 1Nobody reviewed the changelog for the splitter. The upgrade was one line in a dependency file and the pull request described itself as routine maintenance.
2. 2The nightly ingestion job re-chunks only documents that changed, so the bad chunking spread through the index at the rate the help centre was edited — a few per cent of the corpus a week.
3. 3Week two: the eval set still scored within its normal range, because its questions were about long-standing articles that had not been edited and so had not been re-chunked.
4. 4Week three: ticket volume rose, and was attributed to a marketing campaign that had genuinely driven more signups in the same period.
5. 5Week four: an engineer noticed answers quoting fragments that began mid-sentence, raised it in a channel, and was asked for an example that reproduced. Replaying the question returned a different answer, which read as a refutation.
6. 6Week six: the forwarded customer email. Somebody plotted the share of answers containing a citation whose text started with a lowercase letter, found it rising from almost nothing on the Tuesday of the upgrade, and had the cause within an hour of asking the right question.

The splitter was pinned to its previous behaviour and the whole corpus re-chunked overnight, which restored answer quality within a day. The ticket rate took three weeks to return to its previous level, partly because the campaign cohort was genuinely larger and partly because customers who had been given useless answers had stopped asking. The team's post-incident work was entirely instrumentation: an in-product signal on whether the answer was used, a weekly window comparison on it, and a distribution check on chunk lengths at ingestion time.

What luck did

The re-chunking was incremental rather than wholesale, which turned the decline into a six-week slope instead of a one-day cliff. That is what made it invisible, and it is also what made the eventual diagnosis unambiguous: the citation-shape metric, once somebody thought to plot it, traced a clean line back to a single Tuesday with a single deploy on it. A full re-index on the day of the upgrade would have produced a step change that somebody would have connected to the deploy within hours, and this case file would not exist.

Who else tried this

The same upgrade was applied by a very large number of teams in the same quarter and caused almost none of them any harm, because most document corpora have articles short enough that the splitting strategy barely matters. That is exactly why nobody reads that changelog entry, why pinning the dependency would have been argued down as paranoia in review, and why the lesson here cannot be about reading changelogs more carefully. The teams who were hurt by it were the ones whose documents relied on headings for meaning, and none of them knew that about themselves in advance.

Why it is not a template

This is one composite assembled to make a single ordering visible, and it is not evidence about how often ingestion pipelines cause quality declines, nor that chunking is a common culprit, nor that an eval set is weak instrumentation. The transferable part is narrow and it is this: a decline that no existing signal is sensitive to will be found by a customer, and the most sensitive signal available is nearly always already present in the product and merely unrecorded. Everything else here — the splitter, the incremental re-index, the campaign that supplied a false explanation — is scenery.

The four things that made this invisible

The failure had no error. Retrieval returned chunks. The model answered from them. The response was well formed, the status was a success, the answer was fluent. Every layer did exactly what it was built to do with worse input.

The eval set was a fixed sample of a moving distribution. It was not wrong and it was not neglected. It was asking questions about articles that had not changed, in a system where only changed articles were affected, which is the most ordinary form of this problem: an offline set measures the inputs you thought of at the time you thought of them.

Non-determinism destroyed the one report that came in. An engineer saw it in week four and was asked to reproduce it. Replay produced a different answer, and everybody in the thread read that as evidence against rather than as a property of the system. This is the cost of the thing with no lesson of its own.

A plausible alternative explanation arrived at the same time. The campaign genuinely did raise signups and ticket volume. A rising ticket rate had an innocent owner, and a coincidence that explains your symptom is more expensive than no explanation at all.

What I would actually measure

One in-product signal, chosen because the product already does it. Whether the user copied the answer, edited the draft before sending, clicked the citation, rated it, escalated to a human anyway, or rephrased and asked again. Pick whichever the interface already makes someone do, record it with the request identifier, and you have a quality measurement on every real input at no cost in model calls. This is the instrument that cannot go stale, because its sample is whatever arrived.

A refusal and abstention rate. The share of answers that say some version of "the documentation does not cover this". It is cheap, it is a single string match, and it is the earliest moving number in most retrieval declines — when retrieval gets worse, a well-built feature declines to answer more often before it starts answering badly.

Distribution checks on what goes into the prompt, not only on what comes out. Chunk length, number of chunks retrieved, retrieval score of the top result, prompt token count, share of answers citing nothing. In this case a histogram of chunk lengths would have changed shape on the Tuesday. These are cheap, they are ordinary metrics, and they catch cause rather than effect — at the price of only catching causes somebody anticipated.

The nightly eval run, with the set refreshed on a schedule. Keep it, plot it, and put a recurring calendar item on adding recent real inputs to it. An eval set with no refresh policy has a half-life, and the half-life is shorter than anybody expects.

Alert on a window comparison, never on a threshold. Compare the last seven days against the trailing thirty, on each of the signals above, and raise something when a signal moves by more than its own normal week-to-week spread. A threshold on a noisy quality number is either muted within a month or never fires at all. A trailing comparison survives the baseline drifting, which it will.

›Why the window comparison is the part people get wrongOptional

Every other alert in this subject is a threshold, because every other signal has a meaning in absolute terms. Error rate above some small figure is bad at any point in history. Latency above the budget is bad because the budget is a promise.

Quality signals have no such anchor. An acceptance rate is a number about your users, your prompt, your corpus and the questions people happen to be asking this month, and all four move for reasons that are nobody's fault. There is no level that is correct, so a threshold has to be set from the recent past — at which point you have reinvented the window comparison badly, with the baseline frozen at whatever it was on the day somebody configured it.

Compare windows instead. Last week against the trailing month, with a band derived from the signal's own variability over the preceding quarter. The property that matters is not that the number is low but that it fell, and that it stayed fallen — a one-day dip is Tuesday, and a two-week slope is this lesson.

The cost is detection time. A seven-day window cannot tell you anything in a day, and a sharp degradation will reach customers before this fires. So run a much shorter window alongside it for cliffs, and accept that the instrument for slopes is inherently slow. Six weeks became one is the win, not six weeks became an hour.

Check

You add an acceptance signal and a weekly window comparison. Three months later it fires: acceptance has fallen noticeably against the trailing month. What is the first thing to establish?

Choose one answer

Which prompt or model version changed in that windowWhether the inputs changed — whether the questions arriving are the questions that were arriving beforeWhether the signal itself is still being recorded correctlyWhether the eval set score moved in the same window

Check answer

Where this stops being necessary

It stops where a deterministic downstream check validates the output.

A feature whose answer is parsed into a schema, resolved against a database, executed as a query, or compared with a value that a non-model path also computes has an oracle. A decline in that feature produces parse failures, lookup misses, mismatches or rejections, which are errors — and errors already have alerting, dashboards and an on-call rotation pointed at them. The whole problem in this lesson exists because fluent prose has no oracle.

So the strongest version of this work is not better monitoring. It is arranging for as much of the output as possible to be checkable, and then letting your existing alerting cover the checkable part. A structured field that must resolve to a real identifier, a citation that must match a document in the context, a total that must agree with a figure computed in code — each one converts a silent decline into a loud failure.

That arrangement is not always available, and it is never available for the whole output. The part of the answer that is prose a human reads will always be the part with no oracle, and that part needs an in-product signal and a window comparison for as long as the feature exists. Know which part of your output is which, because the monitoring for the two is entirely different and teams routinely build one and assume it covered both.

Worth remembering

• A quality decline moves no signal that was built to catch a failure which announces itself: no error, no latency change, no cost change, no failing test.
• An offline eval set measures the inputs you thought of when you thought of them, so it is blind to a decline that affects newer material around them.
• Non-determinism destroys the first report that comes in, because replay returns a different answer and everyone reads that as a refutation.
• A coincidental innocent explanation arriving in the same window — a campaign, a launch, a seasonal shift — is more expensive than having no explanation.
• The most sensitive quality signal is almost always already in the product and merely unrecorded: copying, editing, clicking a citation, escalating, or asking again.
• The abstention rate is the cheapest early warning in a retrieval feature, because a well-built one declines to answer before it answers badly.
• Monitor the inputs to the prompt as well as the outputs — chunk lengths, retrieval scores, prompt sizes — which catches causes you anticipated rather than effects.
• Alert on a comparison between a recent window and a trailing one, never on a threshold: quality signals have no absolute level that is correct, and their baseline drifts.
• A short window alongside the long one catches cliffs; the instrument for slopes is inherently slow, and six weeks to one week is the realistic win.
• The strongest fix is to make as much of the output checkable as possible, so that your existing error alerting covers it and only the prose needs this machinery.