Part IV · Chapter 6 · Several Services · lesson 2
One request, five services, and nobody owns the latency
14 min
Builds on Your endpoint takes 400ms and the budget was 200 and You made the slow function twice as fast and nothing changed
Part IV · Chapter 6 · Several Services · lesson 2
14 min
Builds on Your endpoint takes 400ms and the budget was 200 and You made the slow function twice as fast and nothing changed
A customer taps Pay and waits. The gateway calls checkout, checkout calls pricing, pricing calls inventory, inventory reads its Postgres replica. The median was 180ms in March and it is 310ms today. Four teams have each checked their own dashboard and each one reports no change in their own service. Nobody has measured the request, because the request belongs to nobody.
Commit to an answer before the arithmetic appears.
Predict
The model below is the same latency budget from track 0, pointed at a path that crosses process boundaries rather than one that stays inside a single service. Its stage labels are generic, so here is the mapping for this request:
Every hop waits for the next one. That is the whole reason these add rather than overlap, and it is why the total is a sum you can audit.
Latency budget
Hypothetical model, not a forecast
Adds up the stages a request passes through and names the one stage that owns the total.
It assumes
It ignores
The four hops that are not the inventory read come to a tenth of a second between them. Leave those four alone and find the largest the inventory hop can be while the whole path still fits inside 200ms.
Counted from the first; later stages are ignored entirely.
| Hops completed | Cumulative latency |
|---|---|
| 0 | 0 us |
| 1 | 12 ms |
| 2 | 30 ms |
| 3 | 75 ms |
| 4 | 285 ms |
| 5 | 310 ms |
Not there yet — keep moving the controls.
Three hundred and ten milliseconds, and the inventory hop owns two hundred and ten of them — sixty-eight per cent of the total. The two numbers the model reports side by side are the whole argument about where to spend the quarter:
Deleting four services outright is worth less than halving one hop. That is the sentence to take into the meeting where three teams are each volunteering to shave a few milliseconds off their own piece.
The challenge lands at 100: a path with a 200 budget and a hundred milliseconds of unavoidable overhead leaves the deep hop a hundred, which is less than half of what it currently takes. Either the budget moves or the inventory read does.
A budget tells you where the time goes when everything works. A timeout decides how long the request can take when something does not — and this is where most call chains are configured backwards.
The gateway here gives the request two seconds, because that is what the mobile client waits. Every service below it was configured independently, and each one has the framework default its team never changed: ten seconds, with one retry.
Timeout budget
Hypothetical model, not a forecast
Worst-case time through a nested call chain where every hop has its own timeout and its own retries, against the budget the caller allows.
It assumes
It ignores
Keep four hops and one retry at each. The gateway's two-second budget is fixed. Find the per-hop timeout that brings the whole worst case inside that budget — and get it only just inside, not far inside.
How deep the nesting goes below the caller.
| Hops below the gateway | Worst case | When nothing goes wrong | The gateway's budget |
|---|---|---|---|
| 0 | 0 us | 0 us | 2 s |
| 1 | 20 s | 120 ms | 2 s |
| 2 | 40 s | 240 ms | 2 s |
| 3 | 1.3 min | 360 ms | 2 s |
| 4 | 2.7 min | 480 ms | 2 s |
Not there yet — keep moving the controls.
The opening state is the one almost every chain is in, and it reports three things worth saying out loud.
The worst case is 160 seconds against a two-second budget. Not 40, not 10. Two attempts per hop, four hops deep, is two to the fourth — sixteen times the leaf timeout. The attempt count multiplies once per level; it does not add.
The caller gives up before its callee does. Every hop below the gateway is willing to wait five times longer than the gateway is willing to wait for all of them. The practical consequence is that the callee's own timeout and retry logic never run: the caller has already abandoned the request, so the retry is spent on work nobody will read, by a service that is now under more load than it was when it got slow.
The budget is gone at the first hop. The model names the depth at which the worst case has already eaten the whole allowance, and it is hop one. Hops two, three and four are, in the worst case, entirely work paid for and thrown away.
The challenge lands at 120: four hops, two attempts each, sixteen times 120 is 1,920 inside a 2,000 budget. That number is the point of this lesson. The deepest hop gets the smallest timeout, not the largest, and the ratio is not gentle — the leaf is allowed about a sixteenth of what the top is allowed.
Static per-hop timeouts that shrink by hand are correct arithmetic and a maintenance problem: the moment someone adds a hop, every number below it is wrong and nothing fails loudly.
So I would propagate a deadline instead of configuring a timeout. If the chain
speaks gRPC, this is already built: a deadline set by the caller travels in the
grpc-timeout header and every hop computes its own remaining budget from it.
In Go, the same thing falls out of passing the request's context down the
chain and never calling context.Background() in a handler. For a plain HTTP
chain there is no standard header for this, so you pick one — Envoy sets
x-envoy-expected-rq-timeout-ms on requests it proxies, which is a reasonable
thing to read if a mesh is already in the path.
Two costs, and both are real. First, a deadline is now a runtime value rather than a line in a config file, so "what is the timeout on this call" stops being answerable by reading the repository, and you need a trace to answer it instead. Second, a deadline arriving from outside is a number a client controls: without a server-side cap, a caller that asks for sixty seconds gets to hold your connections for sixty seconds. Propagate the deadline, and clamp it on arrival.
Check
Inventory serves eleven callers. Your checkout path is one of them and it uses one query shape: a lookup by SKU with a join onto warehouse stock. The other ten callers mostly ask for a single row by primary key.
When the SKU query lost its index, inventory's median moved by a few per cent, because the query that got slower is a small share of its traffic. Its p99 moved more, and the owning team read that as noise from a batch job. Both readings were honest and both were taken over the wrong denominator.
The fix for this is not a better alert threshold. It is cutting the metric by caller and by route, so that "inventory latency for checkout's SKU lookup" is a series somebody can see. That cut costs cardinality, which is the thing metric backends charge for, and it is the single most useful cut in a system where one service has many callers.
Track 4's last lesson is about the other way to get this: a trace carries the caller's identity with the request, so the attribution happens without anybody having decided in advance to cut by caller.
All of the above holds because each hop waits for the next one. Nest the calls and the latencies add; the timeouts multiply; the deepest hop needs the smallest allowance.
Run the calls in parallel instead — one coordinator firing five requests at once and waiting for all five — and every claim in this lesson changes. The total becomes the maximum rather than the sum, so the budget is set by the slowest single call and adding a sixth parallel call costs almost nothing in the median. The timeout ladder collapses to one level, because there is no nesting for the attempt count to compound through. And the number that starts hurting is the tail rather than the median: waiting for all of five calls means waiting for the slowest of five, which is a worse draw than any one of them.
So before using this arithmetic on your own path, check the shape. If the calls nest, this lesson applies. If they fan out, the budget here will overcount badly and the model you want is tail amplification, not a sum of stages.