Part III · Chapter 5 · Cache and Queue · lesson 5

The queue has 400,000 messages and is growing

14 min

Builds on How many requests can be inside your service at once

The consumer-lag graph says 400,000 and the line is going up. Four partitions, four consumers, roughly 1,000 messages a second arriving. Each consumer clears about 300 a second, so the group is draining 1,200 against 1,000 and the backlog is genuinely shrinking — at 200 messages a second, which is not what the graph appears to show.

Somebody is about to add consumers. Two of the three levers available are the wrong one here.

Predict

Four partitions, 400,000 behind, draining 200 a second faster than messages arrive. You add four more consumers to the group. What happens?

The drain rate roughly doubles and the backlog clears in about half the timeNothing changes, because the four new consumers sit idle with no partition assigned to themThroughput improves somewhat, since consumers share partitions and even out the loadIt gets worse, because more consumers means more contention on the database they write to

Little's law

Hypothetical model, not a forecast

Ties arrival rate, latency and concurrency together, solving for whichever of the three is left implied.

It assumes

The backlog is read per partition, because one partition is consumed by exactly one member of the group.
The arrival-rate control here is the surplus: messages a second the backlog actually loses, drain minus arrivals.
The surplus is treated as steady, which it is not — arrivals have a daily shape and the backlog drains fastest overnight.
Little's law is an identity about averages in steady state, and a draining backlog is not in steady state.
Per-message work is taken as constant, so nothing here accounts for a batch that is cheaper per message.

It ignores

Warm-up and shutdown periods, where the law simply does not apply
The cost of concurrency itself — memory, context switches, lock contention
Whether the implied concurrency is physically achievable on the hardware
Variance: this is an identity about averages and says nothing about tails

The backlog on one partition is 100,000 and it is losing 50 messages a second. Keep the backlog where it is and find the surplus needed to clear that partition inside ten minutes.

Arrivals50/s

Requests entering the system each second.

Requests in flight100,000

Implied latency: 33.3 min
Which quantity is implied: 0 concurrency, 1 arrivals, 2 latency: 2
Arrivals: 50/s
Latency: 33.3 min
Requests in flight: 100,000
Arrival rate this concurrency sustains: 50/s

Not there yet — keep moving the controls.

167 a second. The partition is draining 50 a second faster than it fills, and clearing it within ten minutes needs a surplus of 167 — more than half of everything one consumer can do, on top of keeping up with arrivals. At the surplus it actually has, one partition takes about thirty-three minutes, and that is the honest answer to "when will this be fixed".

Now set the control to 300, the full drain rate, and read it differently. A message joining the back of a 100,000-deep partition waits about five and a half minutes before anybody looks at it. That is the number the customer complaining about delayed notifications is experiencing, and it is nearly six times better than the time to clear — because the queue does not have to be empty for the next message to be served, it only has to get through the ones in front.

Two numbers, both correct, answering different questions:

Time to clear is backlog divided by surplus. It is what you tell the person asking when the alert will stop.
Wait for a new arrival is backlog divided by drain rate. It is what the user is experiencing right now.

Reporting the first when somebody asked the second is how a thirty-three minute estimate gets attached to a five-minute symptom, and then nobody trusts the graph.

Three levers move a backlog, and they are not interchangeable.

Raise the drain rate. More consumers, up to the partition count and not one further. More throughput per consumer — larger fetch batches, batched writes, moving a slow call out of the hot path. The lever everyone reaches for, and the one most often already at its stop.

Lower the arrival rate. Shed at the producer, sample, or turn off whatever started emitting more than it used to. Unpopular because it refuses work, and frequently the only free lever.

Reduce the work per message. Usually the largest available factor and always the slowest to ship, because it is a code change rather than a configuration change.

Note what is missing: waiting. A backlog with a positive surplus clears on its own, and the arithmetic above tells you whether it clears before anybody cares.

When draining faster is the wrong move

The consumer writes to the same Postgres primary the checkout path uses. Right now the backlog is the only thing keeping that write load bounded: the consumer group is a rate limiter that happens to be made of lag.

Double the consumers — assuming partitions to put them on — and you have taken the limiter off. The database now carries the full drain rate plus the online traffic, checkout latency rises, and you have converted a delayed-notifications problem into a revenue problem. The backlog was doing something useful and nobody labelled it.

The honest move in that case is to keep the drain rate where the shared dependency can take it, and shed. Three shapes of shedding, in increasing order of how uncomfortable they are to propose:

Drop what has expired. A push notification about a ride that arrived twenty minutes ago is not late, it is wrong. If the message carries a timestamp and a meaning that decays, discard anything older than its useful life as the first thing the consumer does. This is a one-line change and it often clears the entire backlog, because most of it was already worthless.

Split by value and starve the cheap stream. Move the low-value traffic onto its own topic with its own consumers and its own small share of the capacity. Analytics events wait; payment webhooks do not. A single queue cannot express a priority it was never given.

Reset to the end, deliberately. For a stream whose contents are a current state rather than a sequence of obligations — metrics, presence, cache invalidations where a full refresh exists — seeking the group to the latest offset throws the backlog away and fixes the symptom instantly. Write down what you dropped and tell whoever depends on it. Done quietly, this is how a team discovers in a quarterly review that a dashboard has been missing a day.

What I would do, in this order: drop the expired messages, check that the partition count is not the ceiling, batch the writes, and only then add consumers — against a database I have confirmed has headroom for them. And if the shared dependency has no headroom, shedding is not the fallback. It is the answer.

Check

Lag is flat at 400,000 — not rising, not falling — and consumer CPU is at a healthy level. What does that tell you?

The drain rate exactly matches the arrival rate, so capacity is the problem and more of it is the fixThe consumers are stuck, and the flat line is the absence of progressNothing — a flat lag with healthy consumers is a system working normallyThe lag metric is broken, since a real queue never sits exactly level

Where this stops holding

Every number above assumes the backlog is made of load. There is a second cause that produces the same graph and for which all of this arithmetic is nonsense: one message the consumer cannot process.

The signature is specific. Lag is enormous and flat on exactly one partition while the others are near zero. The consumer logs the same error repeatedly, at whatever interval the retry is set to. The committed offset does not advance, and consumer CPU is either pinned by a retry loop or almost idle. Throughput across the group looks like three quarters of what it should be, because it is: three partitions working and one stuck.

No lever in this lesson touches that. Adding consumers gives the poison message a new host. Raising the batch size makes each attempt fail more expensively. Shedding at the producer does nothing, because the thing blocking is already written. The surplus is not small, it is zero at one offset, and dividing by it is the arithmetic error the model will happily perform for you.

The fix is a dead-letter path, and it is worth having before you need it. SQS gives you one with a redrive policy and a receive-count threshold. Kafka does not: you build it, typically by catching the failure, publishing the record to a parking topic with the error attached, committing past it, and alerting on the parking topic's depth. Spring Kafka ships this as an error handler with a dead-letter publishing recoverer, which is worth copying even if you are not on that stack.

The rule underneath it: a consumer must never retry one message indefinitely in place. Bound the attempts, put the message somewhere a person can look at it, and keep going — because the alternative is that one malformed payload stops a partition for as long as it takes somebody to notice that the backlog is not, in fact, made of traffic.

Worth remembering

Time to clear is the backlog divided by the surplus. The wait a new message sees is the backlog divided by the drain rate. They differ by a large factor and answer different questions.
At a 100,000 backlog losing 50 a second, clearing takes about thirty-three minutes; clearing inside ten minutes needs a surplus of 167 a second.
In a consumer group, partition count is the parallelism ceiling: extra consumers sit idle and the rebalance makes the graph briefly worse.
Three levers: raise the drain rate, lower the arrival rate, reduce the work per message. The first is usually already at its stop.
A backlog in front of a shared dependency is a rate limiter nobody labelled, and draining faster takes the limiter off.
Shedding, in order of comfort: drop messages whose value has expired, split by value and starve the cheap stream, or reset to the latest offset and say so.
Flat lag with busy consumers means a surplus of zero — keeping up, not catching up, with a standing latency penalty on every message.
If lag is huge and flat on one partition only, the cause is a poison message and none of the arithmetic applies. Bound the retries, dead-letter it, and move on.