Part IV · Chapter 6 · Several Services · lesson 3

Your diagram has four dependencies. You have nineteen

12 min

Builds on You promised three nines. That is nine hours a year

The diagram on the wiki has four boxes: the service, Postgres, Redis, and the payments provider. Last Tuesday the service would not start on new nodes. The cause was an expired intermediate certificate in the container registry's trust chain, which no running pod cared about and every new pod did. The registry is not on the diagram. Neither are the fourteen other things a pod touches before it answers a request.

Predict

Which of these unlisted dependencies has taken down the most services in practice?

The container registry, because no new pod can start without pulling an imageDNS, because almost every other dependency is reached through a nameThe identity provider, because a request with no valid token is a rejected requestThe certificate authority, because an expired certificate stops every connection

Three censuses, in the order worth running them

Each of these produces a list from something the system already does, rather than from somebody remembering. That is the only property that matters: a list assembled from memory omits exactly the dependencies nobody thinks about.

One: the outbound connection census. Watch every remote endpoint one instance talks to, for a full hour, so that periodic work is included. On a node, ss -tnp on a loop into a file, deduplicated on the remote address, is crude and sufficient. In a cluster with a mesh, the Envoy sidecar's access log already has it and you can read it off. On AWS, VPC Flow Logs for the pod's network interface give the same answer without touching the host. Every distinct remote endpoint is a dependency, including the ones your code never mentions — the telemetry collector, the token endpoint, the clock.

Two: the start-up census. Everything consumed between process launch and the readiness probe passing. For a Kubernetes workload this is readable off the pod spec rather than measured: every imagePullSecrets entry, every init container, every envFrom and every volume mounted from a Secret or ConfigMap, plus whatever the entrypoint fetches before it binds a port. These are the dependencies that are invisible while the fleet is stable and mandatory the moment it is not.

Three: the deploy-time census. Every step that must succeed for a new version of the code to exist and reach a node. The CI runner, the registry, the artefact signing step, the Terraform state bucket, the admission webhook that mutates the pod. These do not break serving. They break your ability to change anything, which is a problem precisely when you most want to change something.

The list this service actually has

Four on the diagram, fifteen not. Each line is the dependency, the named thing that currently plays that role, and when it is needed.

Public DNS — Route 53. Needed per request.
In-cluster DNS — CoreDNS. Needed per request.
The orchestrator's API — the Kubernetes API server. Needed at start-up.
The registry, and the token to pull from it — Amazon ECR. At start-up.
Public certificate issuance and renewal — Let's Encrypt, via cert-manager. Per request, once the current certificate expires.
Internal identity certificates — istiod's built-in certificate authority. Per request.
End-user identity — Okta. Per request.
The pod's own cloud identity — IAM Roles for Service Accounts. At start-up, and again at every token refresh.
Secrets — AWS Secrets Manager. At start-up.
Configuration and feature flags — LaunchDarkly, AWS AppConfig. Per request, unless there is a local fallback.
The mesh control plane — istiod serving xDS. At start-up, and whenever routing changes.
The sidecar proxy itself — Envoy, in the pod. Per request.
The webhook that injects that sidecar — the mesh's mutating admission webhook. At start-up.
Log shipping — Fluent Bit. Needed by neither, in theory.
Metrics and traces — the OpenTelemetry Collector. Needed by neither, in theory.

That is fifteen, and it is a floor rather than a count. It leaves out the CNI plugin that allocates the pod an address, the clock the token validator trusts, the node's local image cache, the load balancer in front of the gateway, and the package index your build pulls from. Every one of those has taken somebody's service down.

The column that decides what to do about each one

The useful classification is not critical versus non-critical. It is when the dependency is needed, because that decides what its failure looks like and therefore what the mitigation is.

Needed per request. Failure is an outage now. These are the ones worth a fallback: a cached key set for the identity provider, last-known-good configuration on local disk, a resolver cache with a sane negative TTL.

Needed at start-up. Failure is a latent outage. Nothing happens, then a node is replaced or traffic rises, and the fleet cannot grow back. This class is the most dangerous because it hides: the registry broke at noon and the incident starts at the evening peak, by which time the two events look unrelated. Pin images by digest, keep an image cache on the node, and have one instance restart on purpose every day so that a broken start path fails while someone is awake.

Needed neither per request nor at start. Telemetry, mostly. The correct treatment is to make sure it cannot take you down: a log shipper that blocks the application when its backend is unreachable has converted an observability problem into an outage, and this is a configuration choice in Fluent Bit and in most collectors. Check which way yours is set before you need to know.

What would you do

You have just run the three censuses on the checkout service and have the fifteen above. One engineer, two weeks. The last two incidents were both start-up failures: an expired registry certificate, and a mesh webhook that rejected every new pod for forty minutes after a control-plane upgrade. Nothing has gone wrong at request time in a year.

Optimising for: Reduce the number of dependencies that can stop the fleet from growing, within two weeks of one engineer's time.
You cannot know: Whether the registry incident was a one-off or a symptom of how that certificate is managed
How long the node image cache would actually survive a registry outage
Whether the mesh webhook can be configured to fail open without breaking its security model
Which of the fifteen your platform team already owns and is already working on

Pin every image by digest, enable a pull-through cache, and set the mesh's mutating webhook to fail open.Add synthetic start-up checks: restart one pod every hour and alert on failure.Write the fifteen into the runbook with an owner and an escalation path for each.Remove the mesh, which accounts for three of the fifteen on its own.

Check

Your log shipper's backend becomes unreachable and the application's request latency triples. What has been configured wrongly?

Logging is blocking the request path — a dependency needed by neither the request nor the start-up has been made a request-time oneThe log backend needs more capacityThe service should retry the log writeNothing — logs are important enough to be worth the latency

The census that finds nothing, and what it meansOptional

Run the outbound connection census and you may get a short list, because a sidecar proxy is terminating everything and the host only ever sees connections to the sidecar. That is not an absence of dependencies; it is the proxy doing its job.

In that case the census moves: the sidecar's own configuration is the list. For Envoy, the set of configured clusters is the set of places it will send traffic, and curl localhost:15000/clusters on a pod running Istio's sidecar dumps it. This is a better list than the connection census produces, because it includes destinations that happen to have had no traffic in your observation hour.

The general shape of the problem: every layer that abstracts the network also hides the dependency list, and the list then lives in that layer's configuration. Service discovery, a mesh, a connection pooler, an egress gateway. Find the layer, read its config, and the hour of packet watching becomes a cross-check rather than the method.

When a dependency is only needed at deploy time

The classification above has a fourth case the three censuses handle badly, and it is the one this lesson stops being useful for.

A dependency that is needed neither per request nor at start-up but only to produce a new version — the CI runner, the artefact signing key, the Terraform state bucket, the registry you push to rather than pull from — will not appear in an hour of connection watching, because nothing in production talks to it. It will not appear in the pod spec either. Its failure mode is not an outage: the fleet keeps serving, indefinitely, exactly the version it has.

That makes it both less urgent and worse to discover, because you find out during an unrelated incident, at the moment you want to ship a fix. So the mitigation is a different kind: a documented path to produce and deploy an image without the usual pipeline, tested at least once, by somebody who was not the person who built it. Not a fallback, not a cache. A break-glass procedure, and a date on the last time it was exercised.

As of 2026-10-31

This list is a snapshot of one common stack in late 2026 and several entries are version-dependent. curl localhost:15000/clusters reaches Envoy's admin interface on the port Istio's sidecar injector has used for several minor versions; both the port and the endpoint are Envoy's own choices and have moved before. IAM Roles for Service Accounts is the AWS-specific mechanism for giving a pod a cloud identity, and the equivalents elsewhere are named differently and work differently. Docker Hub's anonymous and free-tier pull limits — the reason a pull-through cache is worth configuring — have been revised more than once and should be read from the current pricing page rather than from here. And whether a mesh's mutating webhook can be set to fail open at all depends on the mesh version and on how its admission configuration ships by default.

Worth remembering

The dependency list on a diagram is assembled from memory, which means it omits exactly the things nobody thinks about.
Three censuses produce the real list: an hour of outbound connections, everything consumed between launch and readiness, and every step needed to produce a new version.
Classify each dependency by when it is needed — per request, at start-up, at deploy time, or never — because that decides the failure mode and the mitigation.
Start-up dependencies are the dangerous class: the breakage and the outage are hours apart, so they look like unrelated events.
Pinning images by digest, a pull-through cache, and one deliberate restart a day turn a latent start-up failure into a visible one.
Telemetry that blocks the request path has converted an observability problem into an outage; bound the buffer and count the drops instead.
Where a sidecar or a mesh terminates the network, the dependency list lives in that layer's configuration rather than in the connection table.