Part II · Chapter 2 · One Service · lesson 2

Your endpoint takes 400ms and the budget was 200

13 min

GET /api/notes?folder=inbox returns in 400ms at the median. The number somebody wrote in a ticket is 200.

Four things happen on the way through: TLS and routing at the edge, the auth middleware, your handler, and one query. Everyone on the call assumes it is the query, because it is always the query.

Before anybody opens a profiler: which stage owns most of those 400ms?

Predict

Four stages — edge and TLS, authentication, your application code, one database query. Which one owns most of the 400ms?

The database query, which is where the time usually goesAuthentication, because it makes a network call you forgot was thereYour handler, which is doing more work than it looks likeThe edge, because TLS and routing cost more than people think

Latency in a serial path is a sum, and a sum has one dominant term.

That has two consequences people routinely act against. First, the biggest stage sets a floor: if one stage costs 250, no amount of work on the other three can take the total below 250. Second, improving a small stage is capped at the size of that stage — halving a 10 cost you 5, and 5 against a 400 total is not a result you can show anybody.

So the order of operations is forced. Split the total into stages. Find the dominant one. Ask what the total would be if every other stage were free, because that is your best case. Then work on the dominant stage, and only then re-split, because the dominant stage will have moved.

Averages add. Tails do not, which is a different lesson.

Here are the four stages, measured at the median rather than guessed.

Latency budget

Hypothetical model, not a forecast

Adds up the stages a request passes through and names the one stage that owns the total.

It assumes

The four stages run strictly one after another, so the total is their sum.
Each figure is a median for that stage, taken from server-side spans rather than guessed.
Stage 2 is a token introspection call to the identity provider, uncached, on every request.
Stage 4 is one indexed query returning fewer than a hundred rows from a warm cache.
Nothing in the path is retried, so every stage is paid for exactly once.

It ignores

Work that happens in parallel, which this would badly overcount
Variance and tails — averages add, but p99s emphatically do not
Queueing between stages, which grows faster than the stages themselves
Network time not attributed to one of the named stages

Leave the other three stages exactly where they are. Bring the authentication stage down until the total response time meets the 200 budget, and find the largest value that still does.

Stage 1 — edge and TLS10 ms

Stage 2 — authentication250 ms

Stage 3 — application95 ms

Stage 4 — database45 ms

Total latency: 400 ms
Largest contributor: stage 2, authentication: 2
Latency of the largest contributor: 250 ms
Share of the total it owns: 0.625
Total if the largest contributor were twice as fast: 275 ms
Total if every other stage took no time at all: 250 ms
Everything that is not the largest contributor: 150 ms

Time spent so far
Time spent in this stage

The cumulative line barely leaves the floor for the first stage, jumps by 250 at the second, and then climbs gently to 400. The per-stage line is a single spike at stage two and nearly nothing anywhere else.
Stages completed	Time spent so far	Time spent in this stage
0	0 us	0 us
1	10 ms	10 ms
2	260 ms	250 ms
3	355 ms	95 ms
4	400 ms	45 ms

Not there yet — keep moving the controls.

Authentication owns 250 of the 400, which is 62.5% of the response. The other three stages together are 150.

Two outputs are worth sitting with. If every other stage took no time at all, the endpoint would still take 250ms — the budget is unreachable without touching auth. And halving the auth call gets you to 275, still over budget. The only move that works is removing the network call, not speeding it up.

What I would actually do

The stage is a token introspection request — RFC 7662, the endpoint that takes an opaque access token and asks the identity provider whether it is still valid. Called per request, synchronously, with no cache. Two fixes, in the order I would try them.

Validate the token locally. If the provider issues JWTs, verify the signature in-process against the provider's JWKS, fetched once and cached for the key rotation period. The jose library in Node and PyJWT with PyJWKClient in Python both do this in a few lines. The stage drops to single-digit milliseconds, because it becomes a signature check rather than a round trip, and the total lands near 155 — inside budget with room left.

If the tokens are opaque and introspection is unavoidable, cache the result. Redis, keyed on a hash of the token, with a TTL that is the shorter of the token's remaining lifetime and about thirty seconds. At any reasonable request volume the hit rate is high enough that the stage all but disappears from the median — and the price is that a revoked token keeps working for up to the TTL, which is a decision for whoever owns the security requirement, not for you.

What I would not do is add an index to the query in stage 4. It is 45 of 400, it is already indexed, and the best possible outcome of a day's work there is a response that is still over budget.

How to get the four numbers in the first place

This whole lesson is worthless without the split, and the split is cheap.

OpenTelemetry auto-instrumentation gives you spans per middleware and per database call with no code changes in most frameworks, and a trace view shows the 250 immediately. If you do not want a tracing backend yet, the Server-Timing response header costs ten lines: time each phase in the handler, emit auth;dur=250, app;dur=95, db;dur=45, and read it in the browser's network panel. Either way the goal is the same — four numbers that add to the total, so the argument stops being about who is most suspicious.

Check

The team fixes authentication and the median lands at 155. A week later the same endpoint is at 310 again, and the auth stage is still single-digit. What is the most likely explanation?

The cache hit rate fell, so introspection is backA different stage became the dominant one, and it is probably the application or the queryTail latency got worse and the median is reflecting itThe budget was never achievable and 310 is the honest number

Where the sum stops being a sum

The condition under which every number above is wrong: when stages overlap instead of running one after another.

If the handler fires the auth check and the database query concurrently — Promise.all, asyncio.gather, two goroutines and a wait group — the total is not 250 plus 45. It is roughly the larger of the two, so adding the stages overstates the response by the whole of the smaller one. Worse, the lever changes: in a parallel path the only stage worth optimising is the longest one, and shaving anything off the others buys you precisely nothing.

So before you add the four numbers, check whether the path is actually serial. A trace answers this instantly — overlapping spans are visible as overlapping bars — and a spreadsheet of stage timings never will.

Worth remembering

In a serial path the total is a sum, so the largest stage sets a floor nothing else can get you under.
At 10, 250, 95 and 45, authentication owns 62.5% of a 400 total and the other three stages together are 150.
If every other stage were free the endpoint would still take 250, and halving the dominant stage only reaches 275 — the fix has to remove the call, not speed it up.
An auth middleware that calls the identity provider per request is a round trip hiding in configuration nobody has touched; verify JWTs locally against a cached JWKS, or cache the introspection result with a short TTL.
Optimising the second-largest stage is capped at the size of that stage, which is why indexing an already-indexed 45 is a day that cannot succeed.
Removing the dominant stage promotes the next one, so re-split after every fix instead of reasoning from the old picture.
The sum is only valid for stages that run one after another; if they overlap, the total is roughly the largest stage and the smaller ones are not worth touching.