Should you put Redis in front of it?

The product-detail endpoint reads one row from Postgres, serialises it, and returns it. Averaged across a month it serves about 120 requests a second, and the query comes back in roughly 55ms. A ticket has been opened: put Redis in front of it. The ticket does not say what the hit rate will be. It does carry a managed cache instance at $500 a month.

Nobody in the thread has asked what hit rate makes that instance worth buying.

Predict

That endpoint, that traffic, that cache bill. Below what hit rate does the cache cost more than it saves?

Choose one answer

There is no such rate — a cache always saves money, because every hit is a query you did not runAround a tenth — caches are so much cheaper per request than a database that almost any hit rate winsAround four fifths, and five points either side of that changes the sign of the savingIt cannot be computed without knowing the working-set size

Lock in answer

Cache economics

≈ Hypothetical model, not a forecast

Effective latency and monthly cost at a given hit rate, and the hit rate below which the cache tier stops paying for itself.

It assumes

• The origin price is the database instance cost divided by the requests it serves, attributed wholly to this endpoint.
• A miss pays the full origin latency, so the cache check itself is taken as free on a miss.
• The hit rate is a steady monthly average, with no cold start and no eviction storm inside it.
• The cache tier is charged whether it is hit or not, which is what makes a breakeven exist at all.
• Nothing here prices the engineering time, the extra dependency, or the staleness the next lesson is about.

It ignores

• Cache invalidation, staleness, and what being wrong costs
• The write path, and the cost of filling the cache in the first place
• Layered caches, where a miss in the first tier may still be cheap
• Engineering time, usually the largest line in this table
• What a slow page costs in lost revenue, which is the real reason to cache

Leave the hit rate at 90% and find the traffic level, or the origin price, at which the breakeven hit rate drops below 20% — the point at which the cache's cost case stops depending on a guess about hit rates.

Hit rate90%

Requests120/s

Requests arriving each second, averaged across the month.

Origin cost per million requests2

Effective latency

6.4 ms

Latency saved against no cache at all

49 ms

Origin traffic divided by

10

Share of the remaining origin traffic five more points removes

0.5

Bytes the origin still ships each second

141 KiB

Origin cost per month, with the cache

62.21

Origin cost per month, with no cache

622

Total per month, cache plus origin

562

Saved per month

59.87

Hit rate below which the cache costs more than it saves

80.376%

The cache is cheaper than no cache

1

• Saved per month against no cache

◷ Not there yet — keep moving the controls.

About 483 requests a second, or about $8 per million origin requests. Four times the traffic, and the question stops being interesting: the cache pays for itself at almost any hit rate worth deploying, and the argument moves on to whether the data can be stale.

At 120 requests a second it is interesting, and the numbers are uncomfortable. At a 90% hit rate the cache saves $59.87 a month and takes the endpoint from 55ms to 6.4ms. Two figures, two different arguments, and only one of them is worth $500.

The latency one. Nearly 49ms off a page load, repeatedly, is a real product change. The $59.87 is noise, and if you present the cache as a cost saving to somebody who later reads the bill, you will be explaining yourself.

You can measure the hit rate before you build the cache

This is the hour that most cache tickets skip, and it is the highest-value hour in the whole exercise.

Take a day of access logs. Reduce each request to the cache key you intend to use. Replay that sequence through an in-memory LRU of the size you are proposing to buy, and count. Fifty lines of Python, and the output is the number the ticket was guessing at — including the shape of the curve as you vary the size, which tells you whether the next tier up is worth anything.

If the logs do not carry enough to reconstruct a key, that is itself the finding: you do not yet know what you would cache.

Already running a cache and want the real number instead of an estimate? Redis reports keyspace_hits and keyspace_misses in INFO stats, and the ratio is the figure the model above wants. Beware that it counts every key in the instance, so one chatty prefix will flatter or ruin the endpoint you care about.

What people reach for, in the order I would try them

The database's own cache, first. Postgres has a buffer cache, and a row read every second from a warm 12KB page is not a disk read. Before adding a tier, check whether the 55ms is actually the row fetch or a missing index, a sequential scan, or serialisation in the application. A cache in front of a bad query plan hides the plan and keeps the cost.

An in-process cache, second. Caffeine on the JVM, lru-cache in Node, functools.lru_cache or cachetools in Python. No network hop, no new dependency to operate, microsecond hits. It costs you heap, it is cold after every deploy, and each instance has its own copy — so a hundred instances means a hundred first misses per key and a hundred independently stale views. That last part is what rules it out when staleness has to be bounded tightly.

Redis, third, and for a specific reason. Reach for it when the instances must share one view of the cached value, when the working set is larger than you want in heap, or when you need the data structures — sorted sets for a leaderboard, SETNX for a lock, counters for a rate limiter. If the only requirement is "make this endpoint faster", Redis is a bigger answer than the question.

Memcached, when it is purely a key-value cache with multiple clients and you want the simplest possible operational story. Fewer features, less to go wrong, and no persistence to misconfigure. The reason people pick Redis anyway is that the features arrive later and the migration does not.

A CDN, if the response is a cacheable GET with no per-user content. Cloudflare or Fastly in front of the endpoint moves the hit out of your infrastructure entirely, and the cheapest cache is the one in somebody else's building. It stops being an option the moment the response varies per session.

For this endpoint, with this traffic: fix the query first, then an in-process LRU with a short time-to-live, and Redis only when the fleet grows past the point where per-instance staleness is visible to one user hitting refresh.

Check

You raise the hit rate on a cache from 90% to 95%. What did that buy, and what did it not?

Choose one answer

It halved the origin's load and took a little under three milliseconds off the average responseIt improved both load and latency by about five per cent, since that is how much the hit rate movedVery little either way — above 90% a cache is in diminishing returns on everythingIt made the cache pay for itself, where it previously did not

Check answer

Where this stops holding

Everything above assumes the working set fits in the tier you bought. The moment it stops fitting, the arithmetic changes shape rather than degree.

A cache at its memory limit evicts, and what it evicts next is decided by policy, not by your model. Redis defaults to noeviction on a plain install, which means writes start failing rather than old keys disappearing — a surprise the first time the instance fills. allkeys-lru is what most caches want and almost nobody sets. Once the working set is larger than memory, the hit rate stops being the number you measured and becomes a function of how fast the distribution's tail cycles through.

The symptom is specific and worth recognising: evicted_keys climbing in INFO stats, hit rate drifting down week over week with no code change, and origin load rising faster than traffic. That is not a tuning problem. It is the model above telling you that you are buying a cache whose job is now to churn.

Two honest responses. Buy more memory, and re-run the breakeven with the new tier price, because the breakeven moves with it. Or cut the working set: cache fewer fields, cache the expensive derived value rather than the whole row, drop the long tail deliberately by not caching keys whose popularity you can see is one-shot. The second is nearly always the better engineering and always the harder sell.

›Why the cost case is usually the wrong case to makeOptional

Caches are mostly bought for latency and justified with cost, because cost is the argument that fits in a ticket.

The problem is that the cost case is the one that collapses first. It depends on a per-request origin price that is itself an attribution choice — the database instance was already running, and it will carry on running when the cache absorbs 90% of the reads. The saving only becomes real if you then buy a smaller instance, which nobody does, because the instance is also serving the write path and four other endpoints.

So the honest version of the ticket says: this takes roughly 49ms off a page a user hits constantly, it costs $500 a month, and it introduces a second thing that can serve wrong answers. All three of those are true, and a reviewer can weigh them. The version that promises a cost saving is making a claim that the finance report will quietly contradict in a quarter.

Worth remembering

• The breakeven hit rate is the cache bill divided by the origin bill you would otherwise pay. For this endpoint it is 80.4%, and five points either side changes the sign of the saving.
• The breakeven falls as traffic rises: at about 483 requests a second it drops below 20% and stops being a real constraint.
• Effective latency is linear in the hit rate; origin load relief is not. Five points at 90% halves the traffic still reaching the database.
• Three things predict the hit rate before you build: working set against memory, reads per write, and how much staleness you can tolerate.
• You can measure the hit rate from a day of access logs replayed through an LRU of the proposed size, which is an hour's work and settles the argument.
• Fix the query, then an in-process cache, then Redis when instances must share one view or the working set outgrows heap.
• Once the working set exceeds the tier's memory, the measured hit rate stops predicting anything and `evicted_keys` becomes the metric that matters.