Part IV · Chapter 6 · Several Services · lesson 6

Which of the six was slow? Nobody can tell

13 min

Builds on One request, five services, and nobody owns the latency

A support ticket: "checkout took about eight seconds this morning, around 09:14." Six services handle that request. All six have a dashboard, and all six show a p99 comfortably inside target for the 09:00 to 09:15 window. All six have logs you can grep, and no two of them share an identifier.

You have an email address and a timestamp. Nothing in the system can tell you which of the six spent the eight seconds.

Predict

All six dashboards look healthy and the request genuinely took eight seconds. What is the structural reason none of them can answer this?

The dashboards average over too long a window to show a fifteen-second spikeA metric has already aggregated the individual request away before it was stored, so no query can get it backThe six services are not time-synchronised, so the timestamps cannot be comparedThe slow service is not logging at the right level

The three, and what each one structurally cannot do

A metric is a number per time bucket, with the requests summed into it. That is why you can keep years of it for almost nothing, and it is why it can never answer a question about one request. It answers "is the service healthy" and "did this get worse after the deploy", which are the questions you ask most often.

A log line is a record of one event in one process. It keeps the detail a metric threw away, and it has no idea what happened before or after it in another process. Give every log line a request identifier that is the same across all six services and you have most of what you need — the lines can now be joined, sorted by time, and read as one story.

A trace is the remaining part, and it is two things logs do not give you even with a shared identifier.

The first is causal structure. A trace is a tree: each span names its parent, so the record says inventory's call happened inside pricing's call, rather than merely after it. That turns a list of timestamps into a critical path, and the critical path is the answer to "which one was slow".

The second is the wait-versus-work split. A log line saying the checkout handler took eight seconds does not distinguish eight seconds of its own work from seven and a half seconds of waiting on something else. A span with children does, by arithmetic: parent duration minus the time covered by children is self-time. This is the number that ends the argument between two teams, and nothing else produces it.

What it costs to run

Three costs, and only the first one is the one people budget for.

Volume. Spans a second is requests a second times spans a request. The service tree from the previous lesson — a three-way fan-out, three levels deep — is roughly forty internal calls per request, so instrumenting every call at two hundred requests a second is on the order of eight thousand spans a second. Most managed backends price on spans ingested or bytes ingested, so this is the line that decides whether anyone keeps paying for it in year two.

Sampling, and the fact that the two kinds are not interchangeable. Head sampling decides at the root — keep one request in a hundred — and it is cheap, stateless, and throws away the interesting requests, because the slow one is not chosen by a coin flip. Tail sampling decides after the whole trace has arrived, so it can keep every trace that errored or exceeded a duration. That is what you want and it has machinery: every span of a trace has to reach the same collector instance before the decision, which means a load-balancing exporter in front of the collectors and memory for buffered traces. The OpenTelemetry Collector ships a tail-sampling processor for this, and its buffer size is a real capacity decision rather than a configuration detail.

Propagation completeness, which is the cost nobody prices. A trace is only as good as the weakest hop. One service that receives traceparent and does not forward it on its outbound calls does not degrade the trace — it cuts it in two, and everything below the cut appears as a separate root. The break is almost always at a boundary nobody owns: a queue consumer that does not restore context from the message, a batch job, a third-party SDK, a thread pool handoff. You find out by looking for traces with suspiciously few spans.

The smallest version that answers the ticket

In the order I would do it, stopping as soon as the ticket is answerable.

One: propagate traceparent and put the trace identifier in every log line. This is the W3C Trace Context header, it is two headers and some middleware, and on its own it converts six unjoinable log streams into one joinable one. If you do nothing else in this lesson, do this. It answers a large share of tickets without a tracing backend existing at all.

Two: one span per boundary, not per function. A server span where the request arrives and a client span for each outbound call. That is what the OpenTelemetry auto-instrumentation for your HTTP server and client gives you with no code changes, and it is enough to produce a critical path across the six services. Function-level spans multiply volume and add almost nothing to this question.

Three: head-sample low, and keep everything that went wrong. Start with a small percentage at the root plus a rule that retains traces with an error or a duration over your target. Until you have a collector doing tail sampling, the pragmatic version is to keep the sample rate low and accept that you will catch slow requests statistically rather than every time — which is usually enough, because the ticket describes a pattern rather than a unique event.

What I would run: OpenTelemetry SDKs in the services, an OpenTelemetry Collector as the only thing the services know about, and Grafana Tempo behind it with object storage underneath. The reason is the collector: it means the backend is a configuration change rather than a redeploy of six services, and you will change the backend.

The tradeoff with Tempo specifically is that its low cost comes from indexing almost nothing — trace identifier and a small set of attributes — so looking up a known trace is cheap and asking "show me every trace slower than five seconds for this customer last week" leans on how much of TraceQL your version supports and how much data it has to scan. Honeycomb and Datadog APM invert that trade: high-cardinality search is the product and you pay per event for it. If the people answering tickets are not the people operating the storage, the paid option is usually the cheaper one in total, and that sentence is about an organisation rather than about a database.

What would you do

Six services, two of them owned by another team. No tracing today. Everything speaks HTTP and three of the six are behind an Envoy sidecar already. There is a Prometheus and a Loki, both self-hosted by a platform team of two. Tickets like this arrive about twice a week and currently get closed as "could not reproduce".

Optimising for: Be able to answer 'which service spent the time' for a ticket like this one, within a month, without a budget approval.
You cannot know: Whether the other team will add propagation middleware on your timeline
How much of the eight seconds is inside a component that emits no spans at all
What the span volume will actually be, since nobody has counted calls per request
Whether the sidecars are already generating spans that nobody is collecting

Propagate traceparent everywhere and log the trace identifier into Loki. No tracing backend yet.Turn on span generation in the three existing Envoy sidecars and send them to a self-hosted Jaeger.Full OpenTelemetry instrumentation across all six, with a collector and tail sampling.Add a latency histogram per endpoint per caller to Prometheus instead.

Check

You have traces now. The checkout span is eight seconds and the sum of all its children is six hundred milliseconds. What does that tell you?

Roughly seven and a half seconds happened somewhere that emits no spans — in checkout's own code, or waiting for something it does not instrumentThe children are being sampled out, so the trace is incompleteCheckout is the slow service and the fix belongs to its teamThe clocks are skewed between checkout and its callees

The cheapest tracing that is not tracingOptional

Before any of this existed, the move was to carry a request identifier in a header, log it everywhere, and ship the logs somewhere searchable. It still works, and it still answers most tickets.

What you lose is the tree, so you are reading interleaved lines and inferring nesting from indentation that is not there. What you keep is everything else: one query, six services, time order, and the actual parameters of the request, which a span's attributes often omit because somebody worried about cardinality.

There is a middle version worth knowing about, because it costs an afternoon. Log a duration with every outbound call — the caller's measurement of how long the callee took — alongside the request identifier. That single field gives you the wait-versus-work split for one level of the tree, which is the part of a trace that answers "which of the six", without a tracing backend, a collector or a sampling policy existing anywhere.

It is not a replacement and it is a good first week, and it has the property that nothing is thrown away when you do adopt tracing properly: the request identifier becomes the trace identifier and the log lines become span events.

When the slow component emits nothing

A trace shows you what is instrumented and is silent about everything else, and the silence looks identical to speed.

The eight seconds can be inside a component that emits no spans at all: a database that is seen only as the duration of its client span, a load balancer between two of the six, a DNS resolution, a TLS handshake, a queue the request sat in before any handler touched it, or time in the client's own network. In every one of those cases the trace is complete, correct, and does not contain the answer.

The tell is the one in the check above: a parent span much longer than the sum of its children. That names the gap without explaining it, which is a real service — it tells you which process to go into and that the cause is not a downstream call. Past that, tracing has handed the problem back, and the method changes to bisection: add timing around the suspect regions yourself, in that one service, and redeploy. Not elegant, and it is what people actually do, and knowing in advance that this is where the trace stops is the difference between an afternoon and a week.

As of 2026-10-31

Several claims here are tied to current versions and products. W3C Trace Context — the traceparent and tracestate headers — is a published recommendation and is the format to standardise on; tracestate semantics have been revised since the first version. The OpenTelemetry Collector's tail-sampling processor, and the load-balancing exporter that has to sit in front of it so a trace's spans land on one instance, are both current components whose configuration keys have changed across releases, and several OpenTelemetry components were still outside their stability guarantee in 2026. Grafana Tempo's indexing model and the subset of TraceQL it supports have both expanded release by release, so the limitation described above is a statement about late 2026 rather than a permanent property. Pricing models for Honeycomb, Datadog APM, AWS X-Ray and Google Cloud Trace all changed within the last two years and should be read from their current pages; the only durable statement is that they charge on the volume you send, which is why sampling is an architecture decision rather than a setting.

Worth remembering

A metric has summed the individual request away before storing it, so no query can recover one request — and a healthy p99 is consistent with the complainant's request being slow.
Logs keep the detail and cannot cross a process boundary; a shared request identifier in every log line supplies most of what a trace gives you.
The two things a trace adds beyond joined logs are causal structure — each span names its parent, which yields a critical path — and the wait-versus-work split from parent duration minus children.
Running cost is spans a second, which is requests a second times calls a request, and it is what managed backends charge for.
Head sampling is cheap and discards the interesting requests; tail sampling keeps errors and slow traces and needs every span of a trace to reach one collector instance.
A service that receives traceparent and does not forward it cuts the trace in two rather than degrading it, and the break is usually at a queue consumer, a batch job or a thread handoff.
The smallest useful version is propagating traceparent and logging the trace identifier, then one span per service boundary, then sampling — in that order, stopping when tickets become answerable.
A parent span much longer than the sum of its children means the time is somewhere uninstrumented, which is where tracing hands the problem back and bisection takes over.