Part III · Chapter 4 · The Database · lesson 3
They save, hit refresh, and see the old value
13 min
Part III · Chapter 4 · The Database · lesson 3
13 min
A user edits a project name, clicks Save, sees the success toast, hits refresh. The old name comes back. They edit it again. On the third refresh the new name sticks. No error is logged, and support cannot reproduce it, because by the time they try the replica has caught up.
This is read-your-own-writes, the first distributed-systems bug most developers meet without being told it has a name.
Predict
Case file
Composite, not a real person
Support had seventeen tickets over six weeks describing edits that did not save. Engineering closed eleven of them as not reproducible. The twelfth came from a customer with a recorded screen share showing the save, the toast, the refresh and the old value, in that order, in under four seconds.
The echo now lied twice. Names with trailing spaces displayed as the user typed them and were stored trimmed, so the value changed under the user on their next visit and a second round of not-reproducible tickets began. Worse, a queued rename rendered as applied: the user saw the new name, the approver saw the old one, and the two disagreed for as long as the approval sat unactioned. The second bug reached a customer's audit before it reached engineering.
Sticky reads. Pin a session to one replica for its lifetime, usually by hashing the session id in the pooler or the proxy. What it buys is monotonic reads: the user never sees a value go backwards. What it does not buy is read-your-own-writes, because the replica they are stuck to is still behind the primary. It costs you even load distribution, and it turns one lagging replica into one cohort of users having a bad day rather than everyone having an occasional bad second. Reach for it when the complaint is flickering, not when the complaint is staleness.
Read from the primary after a write. Record the time of a session's last write — in the session, in a cookie, in Redis — and route that session's reads to the primary until the window expires. This is the one I would ship. It costs a small amount of per-session state, it costs the primary whatever fraction of reads come from recently-writing sessions, and it costs you a number you have to choose. Choose it from measured lag at peak with margin, not from the lag you see at four in the afternoon, and re-derive it when the write rate changes.
Monotonic tokens. The write returns a position in the replication stream;
the client sends it back with the next read; the router either waits for a
replica to reach that position or sends the read to the primary. This is the
only one of the four that is actually correct rather than probabilistically
correct, because it compares positions instead of guessing at elapsed time.
MySQL gives you the machinery directly: the write returns a GTID and
WAIT_FOR_EXECUTED_GTID_SET blocks until a replica has applied it, which
ProxySQL and Vitess both build causal reads on top of. Postgres has no
equivalent wait call, so you compare the log position the commit reached
against pg_last_wal_replay_lsn() on the replica and fall back to the primary
when it is behind. It costs a token on every response and every request, a
router that understands it, and a timeout policy for when no replica ever
catches up.
Client-side echo. Render what was submitted rather than re-reading. Nearly free, instant, and the case file above is what it costs: it is a claim about the database made by something that cannot see the database. Use it where the server is a dumb store for that field, and stop using it the moment anything server-side can transform, reject, queue or enrich the value. The honest version echoes the response body the server returned rather than the request body the client sent, which survives transforms and still lies about anything asynchronous.
Check
Read-from-primary-after-write, with the window derived from measured peak lag and an alarm that fires when lag reaches a fraction of it. It is the cheapest fix that is about the database rather than about the screen, and it degrades in an understandable direction: when it fails, it fails by sending more reads to the primary.
Then echo the server's response body, not the client's request body, because it is free and it removes the single-request case entirely.
Then, only if the product has a surface where a stale read is a wrong decision rather than a stale screen, monotonic tokens on that surface alone. It is the correct fix and it is the one that puts a position token into every layer of your request path, so it is worth the cost where the cost is justified and not across the board.
Every fix above assumes the staleness comes from one hop: the user's write is on the primary, and the read went somewhere that had not caught up. Replicas are only the most legible copy, and most products have three or four more.
A CDN or a cache in front of the read path has exactly this bug with a longer window and no replication position to compare against. A search index built from a change feed has it with a window measured in whatever the indexer's backlog is. A materialised view refreshed on a schedule has it with a window equal to the schedule. A mobile client with an offline store has it in both directions at once.
The useful habit is to stop asking "is the data stale" and start listing, for one feature, every copy of the value and the maximum age each is allowed to reach. That list is the milestone of this track, and read-your-own-writes is the case where the list has exactly two entries and still goes wrong.