The cache restarted and the database fell over

The Redis instance failed over at 14:02. It came back empty, as it does. Within a few seconds Postgres was at every connection it had, the pool was exhausted, the API was returning gateway timeouts, and the cache — now warm again — was being filled by requests that had already given up and been retried.

The cache was down for about four seconds. The outage ran to eleven minutes.

Predict

A cache running at a 95% hit rate restarts empty. What does the database see in the seconds that follow?

Choose one answer

Twenty times its usual read traffic, which is the hit rate invertedRather more than twenty times, because every concurrent request for the same key becomes its own queryIts usual traffic, just slower, because the cache refills as requests come inNothing unusual — connection pools exist to absorb exactly this

Lock in answer

The three mechanisms, and which one actually saves you

Request coalescing, also called singleflight. One in-flight origin call per key; every other caller waits on its result. In Go this is golang.org/x/sync/singleflight. On the JVM, Caffeine's LoadingCache holds a per-key lock during load and gives you this for free. In Python, a dict of asyncio.Future keyed by cache key, which is thirty lines and worth owning. nginx has proxy_cache_lock on, and Varnish coalesces by default.

The limit worth stating out loud: coalescing is per process. Twenty application instances with perfect singleflight produce twenty origin queries per key, not one. Across a fleet you need a shared lock — SET lock:key token NX PX in Redis, with the loser either waiting briefly or serving stale — and that lock lives in the thing that just restarted empty.

Staggered expiry. Add jitter to every time-to-live so keys do not expire together. Without it, any event that fills the cache in bulk — a deploy that warms it in a loop, a nightly job, the recovery from the last incident — writes a cohort of keys with one expiry instant, and you get this outage again on a schedule. Jitter of ten to twenty per cent of the window, applied at write time, costs one line.

Jitter does nothing for the case in the opening, where the cache lost all its keys at once. It prevents the self-inflicted repeat, which is the more common version of this incident.

Early refresh, so expiry is not a cliff. Serve the stale value and refresh in the background: stale-while-revalidate in HTTP, proxy_cache_use_stale updating in nginx, refreshAfterWrite in Caffeine, or probabilistic early expiry — refresh with rising probability as the expiry approaches, which is the XFetch result and about four lines of arithmetic. The hot keys then never expire under load, because something always refreshes them first.

And the one that actually keeps the database alive: a bounded connection pool, with a queue you have sized and a timeout shorter than the caller's. Not a cache mechanism at all. Postgres degrades badly past a few hundred active connections, so the pool's maximum is the thing that decides whether a cold cache is a slow minute or a dead database. SetMaxOpenConns in Go, HikariCP's maximumPoolSize, PgBouncer in transaction mode with a small server pool in front of a large client pool.

Order of operations, if you do one thing: bound the pool. Coalescing and jitter reduce how often you need that bound to save you. The bound is what saves you.

What I would do

Singleflight in process, jitter on every write, stale-while-revalidate on anything whose staleness you can tolerate for a few seconds, and a connection pool whose maximum is a number somebody chose deliberately with a queue timeout below the client's.

Then the part that is not a mechanism: a load-shedding decision made in advance. When the pool queue is full, the service returns an error immediately rather than waiting — and it returns it to the cheapest requests first if you can tell them apart. Shedding a tenth of traffic to keep nine tenths is a decision somebody has to make while nothing is on fire, because during the incident it will look like making things worse.

What I would not do is size the database to serve the full read load unaided. That is the correct answer for a system where losing the cache must be a non-event, and it costs you the entire saving the cache was bought for.

What would you do

A read-heavy API behind a single managed Redis with automatic failover. A warm hit rate in the mid nineties. One Postgres primary, shared with the write path, sitting behind a pool whose maximum nobody has ever changed from the framework default.

Two weeks of engineering time. The failover above was the second in six months, and both times the database was the thing that fell over rather than the cache.

Optimising for

Losing the whole cache at peak degrades latency and sheds a stated minority of requests, but does not exhaust the database's connections or take the write path down — and it is demonstrated in a load test, not argued about in a document.

You cannot know

• What share of the read traffic concentrates on the hottest few hundred keys
• How many of the retries in the incident came from clients you do not control
• Whether the write path was actually failing or merely queued behind read connections
• How long a full refill takes at the origin's sustainable query rate, which nobody has measured

Choose an option

Make the cache durable: enable persistence so a restart reloads the keyspace instead of coming back empty.Add singleflight in the application and jitter to every time-to-live.Bound and separate the pools, shed when the queue is full, then add singleflight and jitter — and prove it with the cache disabled in a load test at peak-equivalent traffic.Scale the database so it can serve the full read load with no cache at all.

Check

You add perfect per-process singleflight and deploy to forty instances. A single very hot key expires at peak. How many origin queries?

Choose one answer

One — that is what singleflight guaranteesForty, one per instance, and that is the ceiling rather than the floorForty times the number of concurrent requests per instanceUnknowable without the retry configuration

Check answer

Where this stops holding

Everything above assumes the database cannot serve the full read load. Check that, because it is cheap to check and it changes the entire design.

Run a load test at peak-equivalent traffic with the cache disabled in configuration — not flushed in production, disabled in a test environment with production-shaped data. If the origin holds, you own an optimisation: the cache makes things faster and losing it makes them slower, and none of the machinery in this lesson is worth building. Jitter is still free. Skip the rest.

If the origin does not hold, you own a dependency, and the honest consequence is that your availability is now the product of two services rather than one. A cache you cannot survive losing belongs on the dependency list, in the architecture diagram, and in whatever number you have promised anybody about uptime.

The test has a shelf life. Traffic grows, and a system that could serve its full read load unaided last year is a system with a cold-cache dependency this year, discovered during the incident. Put the load test on a calendar, and re-run it whenever the hit rate or the traffic moves by a factor that matters.

›Why the cache is usually missing from the dependency diagramOptional

Caches get drawn as an optimisation because that is how they were introduced. Somebody added one to make a page faster, and the diagram it appeared in was a performance diagram.

The transition from optimisation to dependency happens quietly and without a commit: traffic triples, the origin is never resized because the cache absorbed the growth, and one day the system cannot run without it. No code changed. The diagram did not change. What changed is that the hit rate became load-bearing.

This is why the load test above is worth more than any amount of reasoning. It is the only artefact that distinguishes the two states, and the answer it gives expires.

Worth remembering

• A cold cache is a load generator: the origin receives the full read workload through a path sized for the miss rate.
• Three multipliers stack — hit-rate inversion, concurrent duplicate misses, and retries arriving as the service rate falls.
• Request coalescing is per process, so a fleet of forty instances produces forty origin queries per hot key, not one.
• Jitter on every expiry prevents the self-inflicted repeat, where a bulk refill creates a cohort of keys that expire in the same instant.
• Serving stale while refreshing in the background means hot keys never expire under load.
• The thing that actually saves the database is a bounded pool with a queue timeout below the caller's, plus a shedding decision made before the incident.
• Separate the read pool from the write pool, or a read stampede takes the write path with it.
• The test that tells you whether any of this is needed: a peak-equivalent load test with the cache disabled. If the origin holds, the cache is an optimisation; if not, it is a dependency and belongs in the diagram.