Part III · Chapter 5 · Cache and Queue · lesson 2
The cache is serving data from twenty minutes ago
13 min
Builds on You added a read replica and users started seeing old data
Part III · Chapter 5 · Cache and Queue · lesson 2
13 min
Builds on You added a read replica and users started seeing old data
An admin changes a price in the console at 09:12. At 09:31 a customer sends a screenshot of the product page showing the old one. The write landed in Postgres and the row is correct. Redis is still holding what it was handed this morning, because the write path has never heard of the cache.
Four strategies exist for that problem. The one everybody chooses is not the one that works.
Predict
Staleness has two dimensions and they trade against each other.
Share of reads served stale. How much traffic sees a value the record no longer matches. A time-to-live is the worst of the four here: every read after a write and before the expiry is wrong.
Maximum age of a stale read. How old the oldest wrong answer you will ever serve is. A time-to-live is the best of the four here, and the only one that gives you a number rather than a hope.
The strategies that minimise the share — write-through, delete on write — add a code path that must be correct everywhere a write happens. Where that path is missed, the age is unbounded.
So the design rule is not "pick one". It is: pick the one that minimises the share, then put a time-to-live underneath it to bound the age.
Not the same thing
Accepting that cached values drift, and capping how far they drift by expiring them on a clock.
Updating or dropping the cached value as part of the write that changed the record.
Both sides accept
Both agree the cache will be wrong sometimes. Neither claims a cached read is a consistent read, and anybody who needs one has to go to the record.
Which is which? Put each one on a side.
A nightly importer updates 40,000 rows through a path written before the cache existed
A price change that must be visible to the next customer, not the next window
A catalogue entry that changes about twice a year
A read-replica lag window, where the value written is not yet readable from the replica your cache refills from
Time-to-live. Wrong on every read in the window between a write and the
expiry. Also the only one that is never wrong for longer than the window, needs
no cooperation from any write path, and cannot be broken by a script somebody
adds next year. One argument, SET key value EX 300 in Redis, or
Cache-Control: max-age if the cache is HTTP.
Write-through. The write updates the cache as part of writing the record. Wrong only where a write bypasses it — and writes bypass it constantly in practice: the bulk importer, the admin console, a database trigger, a second service with its own connection to the same table, a migration. It also makes the cache part of the write path, so a cache outage is now a write outage unless you handle that explicitly.
Explicit delete on write. Cache-aside plus a DEL in the write. The popular
choice, and the right instinct: do not try to compute the new cached value,
just remove the old one and let the next read refill it. Wrong less often than
either of the above, and wrong without limit when it is wrong. Three failure
modes to know by name: a write path with no delete in it, a delete that failed
because the cache was briefly unreachable, and the race — a reader misses, reads
the old row, the writer then updates the row and deletes the key, and the reader
finally writes its stale value into the empty slot. That last one leaves a
stale value that no further event will remove.
Versioned keys. Put something that changes into the key: an updated_at
timestamp, a row version, an ETag, a counter bumped on publish. Then a stale
value is unreachable rather than wrong, because nobody asks for it. Almost never
serves a wrong answer. What it costs is that a reader needs the current version
before it can build the key, which usually means one cheap read of the version
from somewhere — and that somewhere now has the freshness problem you just
moved. Rails has carried this as cache_versioning for years; Fastly and
Cloudflare express it as surrogate keys and cache tags, which is the same idea
with the lookup on their side.
Cache-aside, delete on write, and a time-to-live underneath everything.
The delete is what keeps the common case fresh, because most writes do come through the path you instrumented. The time-to-live is what stops any single missed invalidation from being permanent, and it is the part people skip because it feels like admitting the invalidation does not work. It does not work. There is a nightly job you have not read.
Where the version is already free — an updated_at you are selecting anyway, an
object store ETag, a catalog_version the publish flow bumps — use versioned
keys for that data instead, and keep a long expiry to stop old versions
accumulating. Where the value genuinely cannot be stale at all, do not cache it;
go to the record and make the record fast.
The one thing I would not do is write-through on a value with more than one writer. It reads as the rigorous option and it is the one whose correctness depends on every future author of a write path knowing the cache exists.
What would you do
A catalogue of about 80,000 products behind a read-through Redis cache, keyed by product id. Prices change in three places: the admin console (a handful of changes an hour), a nightly supplier importer that rewrites a large fraction of the catalogue, and a quarterly repricing job that somebody runs by hand from a laptop.
The page is read heavily enough that removing the cache is not on the table. Finance has asked for a written answer to "how long can a wrong price be live", and they will hold you to the number.
Check
The expiry-underneath rule has one clear condition that breaks it: when the underlying value changes less often than the window.
A catalogue entry that changes twice a year behind a five-minute window is refetched about a hundred thousand times per real change. Every one of those refetches is origin load, and the reward is the same value you already had. Then the one time it does change, you still serve the old one for up to five minutes, because the window was never the thing keeping you fresh — the invalidation was.
The test is a ratio, and you can compute it from any write log: mean interval between writes to a key, divided by the window. Below about one, the window is doing the work. Far above it, the window is pure cost and the invalidation is load-bearing whether you admit it or not.
At that point stop pretending the clock is a safety net and change the design. Push the window out — hours, or no expiry at all — and make the invalidation something you can verify rather than hope for: a single publish event that bumps a version, so one write makes every dependent key unreachable at once. Then the staleness bound comes from the publish path, you can test it, and you have stopped paying a hundred thousand refetches for a change that happens twice a year.
The cost of that move is the stampede. Everything becomes unreachable in one instant, and every reader goes to the origin together. Which is the next lesson.