Part VI · Chapter 11 · Choosing · lesson 4
Kafka, SQS, or a table with a status column?
15 min
Builds on Should this go on a queue, or just be slower? and Your queue says "at least once", which means twice
Part VI · Chapter 11 · Choosing · lesson 4
15 min
Builds on Should this go on a queue, or just be slower? and Your queue says "at least once", which means twice
A sign-up should send a welcome email, and the email provider takes about a second and fails sometimes. Somebody says "put it on a queue", everybody agrees, and by the end of the week there is a ticket to stand up a Kafka cluster. The volume is a few thousand sign-ups a day.
Three things answer "put it on a queue", and they differ by roughly two orders of magnitude in what they cost to run. Nothing about this workload picks the expensive one.
Predict
Not "how much volume" first. Volume is the question that eliminates least, for the same reason it did in the Postgres-or-DynamoDB lesson: all three options carry more than most products produce.
One: does the enqueue need to be atomic with a database write? If the row and the job must both exist or neither must — an order created and its confirmation queued, a payment recorded and its receipt queued — then a table in the same database wins outright, because the insert is in the same transaction and there is nothing to reconcile. Choosing a broker here means building the outbox pattern: a table, a publisher, and a reconciliation story for the window where one landed and the other did not. You end up with the table anyway, plus a broker.
Two: does anything other than this consumer need these messages? If a second team wants the same stream, or the same events need replaying next month into something that does not exist yet, you need a log that retains messages after they are read. This is the question that genuinely selects Kafka, and it is about readership rather than throughput. A queue deletes a message when it is acknowledged; a log keeps it for its retention period and lets a new consumer start from the beginning.
Three: what ordering do you actually need, and per what? Almost nobody needs total ordering and almost everybody says they do. Per-entity ordering — all events for one account processed in sequence — is cheap in a log (partition by account) and available in a managed queue only through its ordered variant, with throughput limits attached. Global ordering across all entities means one partition, one consumer, and a throughput ceiling set by one thread.
Answer those three and the field is usually down to one. Volume then tells you whether the survivor can actually carry it, which is a sanity check rather than a choice.
The technical limits and the operational cost, in the units each vendor publishes them in. Figures in this block are the perishable part of the lesson and every one of them is enumerated in the dated caveat at the end.
A TABLE WITH A STATUS COLUMN (Postgres)
claim SELECT ... FOR UPDATE SKIP LOCKED (Postgres 9.5+)
enqueue same transaction as your business write — nothing to reconcile
ordering any ordering you can express in ORDER BY, per anything
replay a WHERE clause
retention until you delete it; DROP a daily partition if partitioned
delivery at-least-once; exactly-once effects need an idempotent handler
ceiling write + vacuum churn on one table, order of a few thousand/s
ops cost zero new systems; one more table in a database you already run
what bites dead tuples and index bloat on the status column; a poll
interval that is either latency or load; a stuck job with
no visibility timeout unless you build one
SQS (managed queue, AWS)
standard queue at-least-once, best-effort ordering, effectively unlimited TPS
FIFO queue ordering per message group; deduplication window 5 minutes
FIFO throughput 300 transactions/s, 3,000/s with batches of 10;
high-throughput mode raises this by region, not uniformly:
70,000 unbatched / 700,000 batched in the three largest,
down to 2,400 / 24,000 in the smallest
message size 256 KB maximum payload
retention 4 days default, 14 days maximum
visibility 30 s default, 12 h maximum — redelivery is built in
replay none. An acknowledged message is gone. Use a dead-letter
queue for failures and archive elsewhere if you need history
ops cost per-request billing; no cluster, no brokers, no rebalances
what bites no replay, no second consumer, per-queue FIFO limits, and
long-polling vs short-polling getting billed differently
KAFKA (log, self-managed or managed)
ordering total within a partition; none across partitions
retention time- or size-based; a new consumer can start from offset 0
consumers many independent consumer groups read the same messages
replay reset the offset
ceiling partitions x per-partition throughput; add partitions to grow
ops cost three brokers minimum for a production footprint, plus
coordination, plus the people who understand rebalances
what bites partition count is hard to reduce; consumer lag is the metric
that matters and nothing warns you by default; a slow
consumer blocks its partition; ordering guarantees evaporate
the moment someone adds a retry topic
The row to read twice is the operational cost. A table adds no system. A managed queue adds a billing line and an API. A log adds a distributed system with its own failure modes, and the honest cost of that is a person who understands it being reachable at three in the morning.
What would you do
An order service on Postgres. Creating an order must also schedule three background jobs: a confirmation email, a warehouse notification, and a fraud re-check twenty minutes later. Orders peak around a few per second on promotion days, so the job rate peaks at roughly ten a second.
An order that exists with no confirmation sent is a support ticket; a confirmation sent for an order that rolled back is worse. Four engineers, none with Kafka experience, no platform team.
Three reasons, and none of them is that brokers are bad.
The atomicity argument wins constantly and gets noticed rarely. A large fraction of background jobs are created as a consequence of a database write, and for every one of those the table is the only option with no reconciliation problem. Teams that reach for a broker first usually discover this later and bolt an outbox underneath, which is the table, arrived at expensively.
The operational cost of the table is borrowed, not new. Backups, failover, monitoring, access control and the on-call rota for the database already exist, and the queue inherits all of them for free. A broker's equivalents are all new, and new operational surface is paid for in attention rather than dollars, which is why it never shows up in the comparison spreadsheet.
The volume almost never arrives. The job rate that would force a broker is a few thousand a second sustained, and most products' background work peaks two orders of magnitude below that. Designing for a rate you will not reach costs you the operational surface every single day between now and never.
What the table genuinely cannot do: serve a second consumer that needs its own copy, retain history for replay after acknowledgement, survive a sustained rate beyond one primary's write capacity, or carry a stream to a team that has no access to your database. Those are real, they are the reasons brokers exist, and the honest move is to check for them explicitly rather than assume them.
Check
The recommendation above has one numeric boundary and one structural one.
The numeric one: a jobs table stops being the answer somewhere around a few thousand jobs a second sustained, and the thing that gives is not the claim query but the churn. Every job is an insert, at least one update and a delete, and each of those leaves dead row versions and index entries for vacuum to reclaim. The hot index on the status column is the usual first casualty, because a workload that writes and deletes the same narrow key range all day bloats it faster than it bloats the heap. The symptom is autovacuum falling behind on one table while everything else looks healthy.
Before you migrate at that point, two cheaper moves exist and both are worth trying first: partition by day so retention is a dropped partition rather than a delete, and move the jobs table off the instance that serves product traffic so the churn competes with nothing that matters. Those two together buy an order of magnitude in most cases, and they are days of work against months.
The structural boundary has no number: the day a second independent consumer needs its own retained position in the stream, the table is no longer sufficient at any volume, and that day usually arrives as a request from a team you do not report to.