Part III · Chapter 3 · Under Load · lesson 6
You doubled the workers and throughput went up 10%
13 min
Builds on How many requests can be inside your service at once
Part III · Chapter 3 · Under Load · lesson 6
13 min
Builds on How many requests can be inside your service at once
The order consumer went from 8 workers to 16. Throughput rose 10%. The pods are not CPU-bound, the pool has no waiters, and nobody can explain where the other 90% went.
One measurement explains it: 36% of each job's wall clock is spent inside a
transaction holding SELECT … FOR UPDATE on a single inventory row. That part
does not divide, and it is already deciding your ceiling.
Predict
Amdahl's law. If a share p of the work parallelises and the rest does not, the speedup from n workers is
S(n) = 1 / ((1 − p) + p/n)
and as n grows without limit, S approaches 1 / (1 − p).
That limit is the only number in this lesson worth memorising. Not a trend, not a diminishing return — a hard ceiling set entirely by the share that refuses to divide. 10% serial caps you at 10x. 36% serial caps you at 2.8x. And the workers needed to reach half the ceiling is p / (1 − p), which at 36% serial is two.
The serial part is rarely labelled as such. It is a row lock, a single-writer table, a leader election, a queue every worker claims from with the same query, a shared rate limiter, or one Redis key.
The flat line is the ceiling. The rising line is you.
Amdahl's law
Hypothetical model, not a forecast
Speedup from adding workers when only part of the work parallelises, and the ceiling that part implies.
It assumes
It ignores
Find the parallel share you would need for the ceiling to reach 10x — the point where adding workers is worth planning around at all.
| Workers | Speedup achieved | Ceiling |
|---|---|---|
| 1 | 1 | 2.78 |
| 2 | 1.47 | 2.78 |
| 4 | 1.92 | 2.78 |
| 8 | 2.27 | 2.78 |
| 16 | 2.5 | 2.78 |
Not there yet — keep moving the controls.
90% parallel for a 10x ceiling. Which reframes the whole exercise: the question is never "how many workers" and always "what is the serial share, and can I make it smaller". Dragging the worker slider is the cheap lever and it runs out almost immediately. Dragging the parallel share is the expensive lever and it is the only one that moves the ceiling.
The model also reports the share of each worker that buys nothing. At 16 workers and 64% parallel it is 84%, which is a direct statement about the bill: you are paying for sixteen workers and getting the work of two and a half.
You do not have to guess at p, and you do not need a profiler. Two measurements solve for it.
Run the workload at one worker, then at two, and take the throughput ratio r = T₂ / T₁. Then
p = 2 − 2/r
At 64% parallel, two workers give r = 1.47 and the formula returns 0.64. A ratio of 1.9 returns 0.947 and a ceiling of 19x. A ratio of 1.1 returns 0.18 and a ceiling of 1.22x, which is a workload that should be run on one worker.
Two runs, five minutes, and you know the ceiling before provisioning anything. The reason this is better than reasoning about the code is that the serial part is usually something nobody wrote down — it is a side effect of a query plan or a lock a library takes.
Then find out what it is. For Postgres, the sessions tell you directly:
psql -c "SELECT wait_event_type, wait_event, count(*)
FROM pg_stat_activity WHERE state = 'active'
GROUP BY 1, 2 ORDER BY 3 DESC;"
A pile in Lock / transactionid is sessions waiting for a row another
transaction has. A pile in Lock / tuple is the queue to acquire it. LWLock
is internal contention, usually on buffer mapping or the WAL. Each one has a
different fix, and from outside all three look like one undifferentiated
complaint about the database.
More workers. The move that produced this lesson. Past half the ceiling, every doubling buys less than the last doubling and costs exactly as much. It is still the right first move, because you find out where the ceiling is by hitting it, and two runs is cheaper than a redesign.
Shard the contended row. The standard fix for a single hot counter: keep
sixteen rows instead of one, have each worker update (id, shard) where shard
is a hash of the worker or a random draw, and sum them on read. The serial
section is now sixteen independent serial sections, so p rises and the ceiling
rises with it. The cost is on the read side — an exact total needs an aggregate
over all sixteen rows, and anything that was a single-row read becomes a
SUM. Worth it for counters, usually wrong for inventory, where the exact
total is the business rule.
Batch, so the serial section runs once per group. Accumulate a hundred
decrements in memory and apply one UPDATE … SET qty = qty - 100. The serial
work per item drops by the batch size, which raises p sharply. Two real costs:
per-item latency now includes waiting for the batch to fill, and a crash loses
the unflushed batch unless you persisted the intent first — which usually means
you have moved the problem to a durable log and are now holding a different
serial resource.
Claim work with SKIP LOCKED. This is the specific fix for "a queue nobody
counted as shared". A worker pool claiming jobs with
SELECT … FROM jobs WHERE state = 'ready' ORDER BY id LIMIT 1 FOR UPDATE is
fully serialised at the claim: every worker queues for the same first row. Add
SKIP LOCKED and each worker takes a different row, and a serial section
disappears entirely. It has been in Postgres since 9.5 and in MySQL since 8.0,
and it is the highest-value one-word change in this lesson.
Move the lock somewhere else. The trap. Replacing a Postgres row lock with a Redis lock, or with a lock service, does not change p at all — the same fraction of the work is still one-at-a-time. It usually makes things worse, because the serial section now contains a network round trip, and it adds a failure mode where the lock holder dies and the lease outlives it. If the answer to "what is the serial part" is "a lock", the useful question is whether the work needs to be serialised, not where the lock lives.
Make the operation not need serialising. The real fix when it is available: optimistic concurrency with a version column and a retry on conflict, or an append-only design where workers write independent reservation rows and a reconciler resolves them. Both convert a lock into a conflict rate, and a low conflict rate parallelises. Both also introduce behaviour the product has to accept — a retry the user may notice, or a brief window where the total can be wrong.
What I would actually do here, in order: two runs to get the ceiling, then
pg_stat_activity to name the lock, then SKIP LOCKED if it is a claim and
batching if it is a counter. I would not shard an inventory row, and I would not
move the lock to Redis.
Check
Amdahl holds the problem size fixed and asks how fast you can finish it. That is the right question for a nightly batch with a deadline. It is the wrong question for a service.
Gustafson's observation: in practice a bigger fleet is given a bigger problem. If the serial share is a fixed cost per run — start-up, a leader election, one summary write at the end — then processing ten times the work with ten times the workers leaves the serial part the same absolute size and a much smaller fraction. The effective p rises with the workload and the ceiling moves out of the way.
So the test is whether the serial work grows with the volume. A row lock taken once per order is serial work proportional to the order count: Amdahl applies and the ceiling is real. A single summary row written once per batch run is fixed: the fraction shrinks as the batch grows and the ceiling is an artefact of the batch size you measured at.
Measure the serial share at two volumes as well as at two worker counts. If it falls as volume rises, you have the comfortable case. If it holds, the ceiling is yours to keep until you change the design.