Part I · Chapter 1 · The Diagram · lesson 6
The box labelled cache is a copy that can be wrong
14 min
Part I · Chapter 1 · The Diagram · lesson 6
14 min
The cache is the box beginners find most mysterious, and it has the simplest definition of anything on the whiteboard.
A cache is a copy of an answer, kept somewhere faster, so the same question does not have to be answered twice.
That is the whole idea. Everything difficult about caching comes from one consequence of it, which is that the original answer can change while the copy is still sitting there. A cache is not a faster database. A cache is a bet that the answer has not changed yet, and you are the one who decides how long that bet runs.
Predict
What you buy. The answer comes back faster, and the thing that would have produced it — usually the database — is not asked at all. The second benefit is often the bigger one: a cache is frequently there to protect the database rather than to please the user.
What you pay. Some readers see an answer that is out of date, for a window you choose. You also add a box that can fail, and memory that costs money.
The payment is unusual in one respect worth noticing now: it does not arrive when you make the purchase. The speed shows up immediately and visibly. The staleness shows up later, to one confused user, usually as a support ticket that is very hard to reproduce. That asymmetry is why caches get added casually and removed reluctantly.
Two words you need:
A hit is when the answer is already in the cache. A miss is when it is not, so the real work happens and the result is usually stored on the way back.
The fraction of requests that are hits is the hit rate, and it is the number that decides whether the cache was worth adding at all. A cache that is hit almost every time removes nearly all the work. A cache that is hit rarely adds a step to every request and removes almost nothing — you have made the system slower and more complicated in exchange for nothing.
There is a real arithmetic for where that break-even sits, and it is a lesson in chapter six with a model you can put your own numbers into. What you need here is the shape: a cache is a bet, the hit rate is how often the bet pays, and a low hit rate means you are paying for a bet you keep losing.
Three mechanisms, in rising order of effort and precision.
Expiry. Each entry is kept for a fixed time and then thrown away. Simple, predictable, and the only one that is hard to get wrong. The cost is that you accept being out of date for up to that long, on everything.
Eviction. The cache has limited memory, so when it fills up it discards whatever has been least useful. This is not about freshness at all — it is about space — and it is why a cache may forget something before its time is up. Anything that assumes the cache still holds what it was given is relying on a promise the cache never made.
Invalidation. When the underlying thing changes, the code explicitly removes the copy. The most precise and the most work, because every path that writes must remember to do it — and the one path somebody forgets is where the mysterious stale data lives for months.
Most systems use expiry for nearly everything and invalidation for the few things where being wrong is unacceptable. Chapter six is entirely about this choice and what each option actually costs.
Check
Do not add one until something is actually slow. A cache is the most commonly added box that nobody can later justify. It is cheap to add, it feels like progress, and it makes every subsequent bug harder to reason about.
Decide the staleness window out loud, per thing. "Account balances must be current; the marketing page may be an hour old" is a product decision and somebody other than an engineer should be in the conversation.
Prefer expiry, and reach for invalidation only where you must. One mechanism applied uniformly is something a new person can understand. A mixture, with exceptions, is how the stale-data mystery starts.
Ask what happens when it is empty, before it is in the diagram. If the honest answer is "the database falls over", you have not solved a capacity problem; you have hidden one behind a box that will eventually restart.
The condition under which the entire bargain disappears: when the underlying value never changes.
A cache of something immutable has no staleness cost at all, because there is no newer answer to be wrong about. Compiled assets with a content hash in the filename, a finished invoice, last month's closed report, an image that is replaced by uploading a new one with a new name — these can be cached aggressively and more or less forever, and the entire difficult half of this lesson is irrelevant to them.
This is worth knowing because it suggests a design move you will use often: when something must be cached hard, see whether you can make it immutable first. Changing the name when the content changes turns a hard caching problem into a trivial one, and it is the single most effective caching trick there is.