Part III · Chapter 3 · Under Load · lesson 4

You made the slow function twice as fast and nothing changed

13 min

Builds on Your endpoint takes 400ms and the budget was 200

The flame graph was unambiguous: serialise_order was the widest bar in the application, 40ms of it. Two days of work got it to 20. The endpoint went from 210ms to 190ms, the pool-size arithmetic from the last lesson barely moved, and the ticket is still open.

The profiler was telling the truth about the thing it measures. It does not measure the thing that owns the total.

Predict

A request passes through six stages in series: edge and TLS 5ms, authentication 15ms, application 40ms, database 120ms, serialisation 10ms, response transfer 20ms. Which single change gets the endpoint under 120ms?

Make the application stage twice as fast — it is the biggest thing you controlDelete the edge, authentication, serialisation and transfer stages entirelyCut the database stage to under 30msRun the stages in parallel instead of in series

Stages in series add. Which means two things follow immediately, and both are usually skipped:

The share is the ceiling on the gain. A stage owning share s of the total can, at absolute best, remove s of it — by taking zero time. Halving it removes s/2. Before optimising anything, divide its latency by the total: that fraction over two is your optimistic upper bound on the improvement, and if that number is smaller than the gap you need, the work cannot succeed.

The dominant stage is usually off-CPU. In a service that talks to a database, a cache and one other service, most of the wall clock is spent waiting. A profiler that samples on-CPU time sees the smallest part of the request and reports it in exhaustive detail.

Split the request before you profile any part of it. Then profile the part that owns the total.

The budget, and what each change buys

Each stage is a slider. Move one and watch both the total and the share.

Latency budget

Hypothetical model, not a forecast

Adds up the stages a request passes through and names the one stage that owns the total.

It assumes

The six stages run strictly one after another with no overlap between them.
Each figure is the average for that stage rather than a worst case or a tail.
Nothing is retried, so every stage is paid for exactly once per request.
Queueing between stages is excluded and would add to every figure under load.

It ignores

Work that happens in parallel, which this would badly overcount
Variance and tails — averages add, but p99s emphatically do not
Queueing between stages, which grows faster than the stages themselves
Network time not attributed to one of the named stages

Get the total under 120ms. You may move authentication, the application stage and the database stage — find the combination that works, and notice how few of them there are.

Stage 2 — authentication15 ms

Stage 3 — application40 ms

Stage 4 — database120 ms

Total latency: 210 ms
Largest contributor: stage 4, database: 4
Latency of the largest contributor: 120 ms
Share of the total it owns: 0.571
Total if the largest contributor were twice as fast: 150 ms
Total if every other stage took no time at all: 120 ms
Everything that is not the largest contributor: 90 ms

Elapsed by end of stage
This stage alone

A staircase climbing from zero to the total, with one step far taller than the rest; the per-stage line is flat and low except for a single spike at the database stage.
Stage, in order	Elapsed by end of stage	This stage alone
0	0 us	0 us
1	5 ms	5 ms
2	20 ms	15 ms
3	60 ms	40 ms
4	180 ms	120 ms
5	190 ms	10 ms
6	210 ms	20 ms

Not there yet — keep moving the controls.

Two of the model's readouts are the argument in one line. Total if the largest contributor were twice as fast: 150ms. Total if every other stage took no time at all: 120ms. Deleting five stages outright is worth less than halving one — and it was never going to be otherwise, because that one stage is more than half the total.

Note also what the two-day optimisation actually bought under load. From the previous lesson, requests in flight is arrival rate times hold time. The hold time went from 210 to 190, so at 400 a second the concurrency needed fell from 84 to 76: a 9.5% reduction in pool pressure for two days of work. The same two days spent on the database stage, getting 120 to 60, would have taken it to 60.

Measuring it, with the thing that actually sees waiting

A distributed trace. The right tool, and the one that answers this question directly. OpenTelemetry auto-instrumentation for Python, Java, Node and Go will produce spans around the HTTP server, the database driver and the outbound HTTP client with no code changes, which is already five of the six stages above. The shape to look for is the one from the previous lesson: a span whose own duration is most of its parent's with nothing underneath it.

The cost is real — a collector to run, sampling to configure, and a per-span storage bill. For one endpoint and one question, head-based sampling at a low rate is enough; you need a handful of slow traces, not all of them.

A wall-clock profiler, if you are not going to install tracing. This is the specific fix for the mistake the lesson opens with. py-spy record --idle includes threads that are blocked rather than running. async-profiler -e wall does the same on the JVM, against the default -e cpu. The resulting flame graph attributes the 120ms to the driver's read call, which is less precise than a span but costs one command and no deploy.

Four timers in the handler. Unfashionable and frequently sufficient. Record a monotonic clock reading at each boundary and emit the four deltas as a histogram per stage. You get the budget, per endpoint, in your existing metrics system, for an afternoon of work and no new infrastructure. What you do not get is the ability to follow one request across services, which is the thing tracing exists for and the reason I would still install it.

What I would actually do, in this order: wall-clock profile first because it is one command and rules out the most common error; four timers second if the answer is "the database" and you need to know which query; tracing third as the permanent answer, scoped to the services on this path rather than to everything.

Check

A stage takes 15% of your request's total time. Your endpoint is 210ms and the budget is 150ms. How much can optimising that stage contribute?

Up to 31ms, and only if the stage becomes freeUp to 60ms, which closes the gapIt depends how much faster the stage can be made15% of the gap, so about 9ms

Where this stops being true

Everything above assumes the stages run one after another. The moment any of them overlap, adding them up overstates the total — sometimes enormously.

A handler that issues three independent queries concurrently has a database stage equal to the slowest of the three, not their sum. A client that opens its TLS connection while the DNS lookup for the next host is still in flight has overlapped two stages you would have counted twice. If you add up a budget and it comes to more than the measured end-to-end latency, that difference is your evidence of overlap, and it is worth chasing rather than rounding away.

The practical test: compare the sum of your per-stage timers against the one timer that spans the whole handler. Equal within noise means series, and this lesson applies. Materially smaller means parallel, and the stage that owns the total is the longest one rather than the biggest share — a different question with a different answer.

Why averages are the wrong budget for a p99 targetOptional

The model adds averages, and averages add cleanly. Percentiles do not.

If each of six stages has a p99 and they are independent, the p99 of the total is nowhere near the sum of the six p99s — it is much lower, because the chance of all six being simultaneously at their own worst is vanishingly small. It is also higher than the sum of the six medians. Neither sum is the answer.

What this means in practice is that a latency budget built from averages tells you where to work, which is what you want it for, and does not tell you what your p99 will be after the work. For that you need either the distribution or a measurement — and the next lesson is about the specific case where this gets much worse, which is one request waiting on many calls at once.

The safe habit: build the budget from averages or medians to decide what to fix, then verify the p99 by measuring it, never by adding it up.

Worth remembering

Stages in series add, so a stage's share of the total is the hard ceiling on what optimising it can contribute.
Halving a stage buys half its share; deleting it buys all of its share. Both are knowable before any work starts.
In the worked budget the database stage owns 57% of 210ms, and deleting all five other stages is worth less than halving it.
A sampling CPU profiler cannot see off-CPU time, which is where the dominant stage usually lives in a service that talks to a database.
py-spy record --idle and async-profiler -e wall measure wall clock instead, which is one command and no deploy.
Four monotonic-clock timers in the handler give you a per-endpoint budget in your existing metrics system; tracing adds following one request across services.
Before agreeing to an optimisation, say the optimistic bound out loud: share of the total over two, against the gap you need.
If the per-stage timers sum to materially more than the end-to-end timer, the stages overlap and the sum is the wrong model.
Build the budget from averages to decide what to fix, then measure the p99 rather than adding percentiles up.