Should this go on a queue, or just be slower?

Signing up for the service does five things: write the user row, send a welcome email, generate a thumbnail, notify the analytics pipeline, and provision a workspace the user is dropped into on the next page. The handler does all five inline and takes about four seconds on a bad day.

The proposal is to put all five behind a queue. Three of them belong there, one is already fine, and one would get materially worse.

Predict

Of those five, which one is made worse by moving it onto a queue?

Choose one answer

The analytics notification — a queue is overkill for something nobody reads synchronouslyThe workspace provisioning, because the user is on the next page waiting for it to existThe welcome email, because email providers already queue and you would be queueing a queueThe user row, since a queue would make the write eventually consistent

Lock in answer

The three cases

It must survive a restart. The welcome email, the invoice, the webhook to the customer's endpoint. Characterised by: somebody notices if it never happens, and nobody notices if it happens a minute late. This is what queues are for, and the whole value is in the acknowledgement — the message stays visible until a consumer says it is done, so a process dying mid-work costs you a retry rather than the work.

It can be dropped. Analytics events, cache warm-ups, a search index update that a nightly rebuild also does. Characterised by: the loss is statistical rather than individual. These do not need a durable queue, and giving them one is how a team ends up operating a broker to protect data they would not pay to recover. A bounded in-memory channel that drops on overflow, with a counter on the drops, is the correct shape — and the counter is the part people skip, which turns a deliberate drop into a silent one.

Somebody is waiting. The provisioning step. Characterised by: there is a human or a calling service blocked on the answer. A queue here buys you nothing and costs you a hop, a serialisation boundary, and a mechanism for the caller to discover completion — polling an endpoint, a websocket, or a page that lies pleasantly while something finishes.

That third case has one honest exception, and it is the most interesting thing in this lesson. If the work must also survive a restart — a payment capture, an order submission — then it does belong on a queue, and the caller stops waiting for the result. It gets an identifier and a status to poll. That is a product decision, not an infrastructure one: somebody has to design what the user sees while it is pending, and what they see if it fails twenty minutes later with nobody on the page.

Teams that skip that conversation ship the queue anyway and invent the pending state by accident. It is usually a spinner that never resolves.

What I would reach for, in order

Postgres, with SELECT ... FOR UPDATE SKIP LOCKED. If the project already runs Postgres and the job rate is in the tens or low hundreds per second, this is the first answer and it is not a compromise. One fewer system to operate, transactional enqueue — the job and the row it refers to commit together, which removes an entire class of ghost-job bug — and you can query the backlog with the tool you already have. It costs you table bloat under churn and a database that now carries queue load alongside everything else.

SQS, when you want somebody else to own the retries. Visibility timeouts, redrive to a dead-letter queue after a configured number of receives, and no broker to operate. The tradeoffs are real and specific: at-least-once delivery on the standard queues, no ordering unless you take the FIFO variant with its lower throughput ceiling, a maximum message size that pushes large payloads into object storage, and a maximum retention after which unprocessed messages are gone. It is the default I would pick for "must survive, nobody waiting" on a team without a platform group.

RabbitMQ, when routing is the requirement. Exchanges, bindings and per-queue semantics are genuinely better than a flat topic when one event has five different consumers that each want a slice. It is a broker you operate, with cluster and queue-mirroring decisions that have bitten people.

Kafka, when several independent consumers need the same stream, or replay matters. It is a partitioned log, not a work queue, and that distinction decides whether it fits: ordering is per partition, consumers track offsets rather than acknowledging individual messages, and redelivering one bad message without blocking its partition is something you build rather than configure. For a stream of events feeding analytics, search indexing and a notification service, it is the right shape. For "retry this one email", it is the wrong one.

Sidekiq, Celery, BullMQ — the queue inside a framework, backed by Redis. Excellent ergonomics, and the honest caveat is durability: a Redis instance without persistence loses in-flight jobs on a restart, and the default configuration in several of these has historically favoured throughput over that guarantee. Fine for work you can drop or regenerate, and worth a hard look before the invoice goes through it.

Temporal, when the work is a multi-step process with state. Provision a workspace, wait for a third party, retry step three, compensate if step four fails. That is a workflow, not a job, and expressing it as five queue messages with a status column is how you end up writing a worse Temporal.

For the sign-up handler: the email, the thumbnail and the analytics event go through whatever queue the project already has, with the analytics event going through a path that is allowed to drop. Provisioning stays inline and gets made faster, and if it cannot be made fast enough, the next page changes to show a workspace being prepared — which is a design ticket, not a queue ticket.

What would you do

Checkout does three things after the card is authorised: writes the order, calls a warehouse API that is sometimes unavailable for minutes at a time, and emails a receipt. Today all three are inline; the warehouse call is what makes the page slow, and when it fails the customer sees an error after their card has been charged.

The team runs Postgres and already uses a background worker library for email. There is no platform team. Deploys happen several times a day and restart every process.

Optimising for

Checkout returns in well under a second, no paid order is ever lost including during a deploy, and a customer who refreshes the confirmation page is never shown a state the system cannot later justify.

You cannot know

• Whether the warehouse API is idempotent on retry, which nobody has tested
• How long the warehouse outages actually last, as opposed to how long they are remembered as lasting
• Whether finance can tolerate an order that is paid for and not yet submitted to the warehouse
• What the confirmation page is allowed to promise, which is a question for whoever owns the funnel

Choose an option

Keep all three inline and add retries with a short timeout to the warehouse call.Write the order and enqueue the warehouse submission and the receipt in the same database transaction, drained by a worker. The page returns as soon as the order row commits.Put the whole checkout on a queue: the request enqueues an order intent and returns a job id, and the page polls for the result.Order and receipt inline, warehouse submission onto SQS with a dead-letter queue.

Check

Which of these is the strongest reason to prefer a queue table in your existing database over a managed broker?

Choose one answer

It is cheaper, because you are already paying for the databaseThe job and the data it refers to commit or fail together, so there is no window where one exists without the otherIt has one less moving part, so there is less to operateYou can inspect the backlog with SQL rather than a console

Check answer

Where this stops holding

All of the above assumes you can decide where the work goes. The decision disappears in one case: when the caller needs the result before it can reply.

Then no queue design helps, because the thing you are trying to remove is the wait, and the wait is the contract. A synchronous API whose consumer has no callback, a page that must render a provisioned resource, a card authorisation whose outcome belongs in the response — for these, moving the work does not reduce the latency anybody experiences. It adds a hop to it.

What is left is unglamorous and it is the actual answer: make the work faster, run the independent parts in parallel instead of in sequence, cache the expensive part, or cut it. Then put a timeout on it and decide what you return when the timeout fires, because an inline dependency without a timeout is a request that hangs until something else gives up.

And if the work genuinely cannot be made fast enough, the thing that changes is the contract. Return an identifier and a status, give the caller a way to find out, and design the pending state deliberately — a queue behind an unchanged synchronous contract is the one configuration that is worse than either choice made honestly.

Worth remembering

• Three questions in order: must it survive a restart, is anybody waiting for the result, and is it safe to retry.
• Work that must survive and that nobody waits for is what queues are for, and the acknowledgement is the entire value.
• Work that can be dropped wants a bounded buffer that drops on overflow, plus a counter on the drops — not a durable broker.
• Work the caller is waiting for is made worse by a queue: the wait is relocated, not removed, and two failure modes are added.
• The exception: work that is both waited-for and must-survive belongs on a queue, and the caller gets an identifier instead of a result. That is a product decision about the pending state.
• Every queue retries, so a queue in front of non-idempotent work converts a latency problem into a correctness problem.
• A queue table in Postgres with SKIP LOCKED buys a transactional enqueue, which no separate broker gives you without an outbox.
• Kafka is a partitioned log rather than a work queue: right for several consumers over one stream, wrong for retrying one message.
• When the caller needs the result before it can reply, the answer is faster work, parallel work, or a changed contract — never a queue behind the same contract.