Your queue says "at least once", which means twice

Some customers got the receipt twice. Not many — the ones whose consumer was redeployed between sending the email and acknowledging the message. The queue's documentation says "at least once", and it was telling the truth. It reads like a floor on reliability, which is why nobody read it as what it also is: a statement that some messages will be delivered more than once, by design, forever.

Predict

Your broker offers an exactly-once mode. You turn it on. The consumer sends an email and writes a row to Postgres. What is now guaranteed?

Choose one answer

Each message is processed exactly once, which is what the mode is calledEach message is delivered once, but the consumer may still process a delivery twice if it crashesReading a message, producing derived messages and committing the offset happen atomically inside the broker — and the email and the row are outside that boundary entirelyNothing useful — exactly-once is marketing and the mode is a no-op

Lock in answer

The four ways to make a consumer safe to repeat

Natural idempotence — assignment, not accumulation. SET status = 'shipped' applied twice is one outcome. UPDATE counter = counter + 1 applied twice is a bug. A surprising share of handlers can be rewritten from the second shape to the first, and when they can, everything else on this list becomes unnecessary. Carry the final value in the message rather than the delta.

A unique constraint in the same transaction as the effect. Insert the message's identifier into a processed_messages table with a primary key, in the same transaction that writes the effect, and let the second attempt fail the insert. INSERT ... ON CONFLICT DO NOTHING and check whether you inserted. This is the only mechanism on this list that is actually correct, and the reason is atomicity: the record of having processed the message and the effect of processing it commit or roll back together, so there is no window where one exists without the other.

Conditional writes. UPDATE ... WHERE version = 4, or a DynamoDB ConditionExpression that fails if the item already reflects the change. The same atomicity property, expressed in the data instead of in a side table, and it needs no extra storage. It works where the effect has a natural version and not where it does not.

A duplicate-suppression store with a window. Redis SET dedupe:id 1 NX EX 300 before doing the work: if the key already existed, skip. This is the one people reach for, and it is the weakest of the four, for two reasons. The store is not in the same transaction as the effect, so a crash between the key and the work loses the message entirely — you have marked as done something that never happened. And it only suppresses duplicates that arrive while the key is still there, which makes correctness a function of a retention window you guessed at.

That last point is arithmetic, so here it is.

Duplicate suppression window

≈ Hypothetical model, not a forecast

How many keys a duplicate-suppression store holds, what it costs in memory, and how many duplicates slip through because they arrived after the key expired.

It assumes

• One key is written per distinct message and kept for the retention window, never refreshed by a hit.
• A duplicate is suppressed only while the original's key is still in the store.
• The delay between an original and its duplicate is exponentially distributed around the stated mean.
• Two per cent of arrivals are duplicates, which is a stand-in for a redelivery rate you would measure.
• The bytes-per-key figure ignores index and allocator overhead, which for keys this small is usually larger than the keys themselves.

It ignores

• The second half of exactly-once: suppressing a duplicate request is not the same as applying its effect once, which needs the write and the key to commit together
• Index, pointer and allocator overhead on top of the stated bytes per key, which for small keys is usually larger than the keys
• Replication and partitioning, which multiply the store and add a consistency question of their own
• Lookup cost and the latency the check adds to every request, duplicate or not
• A real duplicate delay distribution, which is usually bimodal: an immediate retry, or a replay hours later, with little in between
• Clock skew and ordering, which decide whether a key has expired at all

At a five-minute window, about 1,642 duplicates per million requests get through. Find the retention window that brings that under 100 per million, then read what the store costs there.

Requests2k/s

Everything arriving each second, duplicates included.

Retention window5 min

How long one key is kept before it expires.

Distinct requests each second, which is the key insert rate

2k/s

Keys held in a steady state

588,000

Memory the store holds

35.9 MiB

Duplicates arriving each second

40/s

Duplicates suppressed each second

36.72/s

Duplicates that arrive after the key has expired

3.28/s

Share of duplicates suppressed

0.918

Share of duplicates that get through

0.082

Duplicates that get through, per million requests

1,642

Retention window that would suppress 99% of duplicates

9.2 min

There are duplicates to suppress

1

• Duplicates that get through, per second

◷ Not there yet — keep moving the controls.

A little over ten and a half minutes, and the store roughly doubles: from 588,000 keys to about 1.25 million. That is the trade in one sentence — suppressing the last stragglers costs memory linearly while buying correctness logarithmically, and the window that catches 99% of duplicates here is about nine minutes, which is far longer than the five somebody picked because it sounded safe.

Then the uncomfortable part. Even at ten and a half minutes you are letting about one duplicate per ten thousand requests through, and the real distribution of redelivery delays is not the smooth curve the model assumes. It is two spikes: the immediate retry seconds later, and the replay hours or days later when somebody reprocesses a topic or drains a parked queue. No window catches the second spike. A unique constraint catches it for the lifetime of the table.

Where exactly-once genuinely exists, stated honestly

Kafka transactions. Since the 0.11 release, a producer can write records and commit consumer offsets in one transaction, and a consumer reading with isolation.level=read_committed never sees records from a transaction that was aborted. Kafka Streams wraps this as processing.guarantee=exactly_once_v2. The scope is Kafka to Kafka: records in, records out, offsets moved, all or nothing. It does not and cannot cover a write to Postgres, a charge at a payment provider, or an email. It also costs throughput and adds a transaction coordinator to your failure modes, which is why it is off by default.

A queue inside the database. If the messages are rows in Postgres and the effect is also in Postgres, claim the job and apply the effect in one transaction and you have exactly-once for real, with no extra machinery. This is the quiet advantage of a queue table over a broker, and it stops being available the moment the effect leaves the database.

SQS FIFO deduplication. A deduplication id suppresses repeat sends inside a fixed window — five minutes, as of this writing, not configurable. Useful, and note what it is: the model above, implemented by somebody else, with the window chosen for you. Same shape, same limitation.

The transactional outbox, for the producing side. Write the business row and a row in an outbox table in one transaction; a relay reads the outbox and publishes. That makes "the event exists if and only if the change happened" true, which removes the producer's half of the problem. The consumer still has to be safe to repeat, because the relay can publish twice.

What I would do

A unique constraint on a business key, in the same transaction as the effect. Not a dedup store — a constraint, in the database that holds the thing being changed, on an identifier the producer supplies and does not reuse.

If the effect is at a third party, use their idempotency mechanism and store the key: a payment provider's idempotency header makes the remote side safe to repeat, and recording the key locally in the same transaction as your own row makes the pair safe. If the third party has no such mechanism, you cannot have exactly-once against it, and the honest design is to make the duplicate harmless, visible, or cheap — in that order of preference.

The Redis dedup store I would use for one thing only: cost. If a duplicate triggers an expensive computation, a short window in front of it saves most of the expense most of the time, and the correctness still lives in the constraint underneath. Dedup store as an optimisation, constraint as the guarantee. Using the store as the guarantee is the specific mistake this lesson exists to stop.

Check

A consumer writes the dedup key to Redis first, then performs the effect, then acknowledges. It crashes between the key and the effect. What happened?

Choose one answer

The message is redelivered and processed correctly, because the key expires eventuallyThe message is lost: the redelivery sees the key, concludes it has been handled, and skips work that never happenedNothing — the broker never acknowledged, so the message is redelivered and the key is ignoredIt depends on whether Redis persisted the key before the crash

Check answer

Where this stops holding

All of this machinery — the window, the store, the memory, the arithmetic — is the price of a consumer that is not naturally idempotent. Make it naturally idempotent and the entire lesson becomes unnecessary.

Three questions tell you whether that is available. Does the message carry the final value rather than a change to apply? Can the effect be expressed as setting a field rather than incrementing one? Is there an identifier in the data that could carry a unique constraint without inventing a side table?

Where the answer to all three is yes, applying the message twice costs a wasted query and nothing else. No dedup store, no window to tune, no memory bill that grows with traffic, and — the part that matters most — no new failure mode at the boundary between two systems that cannot commit together. The replay from six months ago is safe. Reprocessing the whole topic to fix a bug is safe, which is the capability that makes a log worth keeping in the first place.

So the first move on any duplicate problem is not to reach for suppression. It is to ask whether the handler can be rewritten to not care, and to spend the afternoon finding out. Suppression is what you build when the answer is no.

Worth remembering

• At most once acknowledges before working and loses messages; at least once works before acknowledging and duplicates them. Duplicates are the mechanism, not a defect.
• Exactly-once exists only inside a transactional boundary. Kafka transactions cover read-process-write within Kafka and cannot cover an email, a payment or a row in another database.
• A queue table in the same database as the effect gives genuine exactly-once, because the claim and the effect commit together.
• The four consumer mechanisms, strongest first: natural idempotence, a unique constraint in the same transaction, a conditional write, and a dedup store with a window.
• A dedup store outside the effect's transaction has two orderings and both are unsafe: mark-then-do loses messages, do-then-mark duplicates them.
• At a five-minute window against a two-minute mean redelivery delay, about 1,642 duplicates per million requests get through; reaching 100 per million takes about ten and a half minutes and roughly doubles the store.
• Real redelivery delays are bimodal — seconds later or days later — so no window catches the replay, and a unique constraint catches it indefinitely.
• Use the dedup store to save cost and the constraint to provide the guarantee. Treating the store as the guarantee is the central mistake.
• If the handler can be made naturally idempotent, all of this machinery is unnecessary, and finding out takes an afternoon.