Part V · Chapter 8 · Running It · lesson 6

Every deploy drops requests for eight seconds

14 min

Builds on How many requests can be inside your service at once and Under overload you refuse work, or you lose all of it

Every deploy, the error-rate graph shows a spike lasting about eight seconds. Connection resets and gateway errors, a few hundred of them, then clean. The rolling update is configured, the readiness probe is green, the application logs show nothing — no exception, no shutdown message, nothing at all. The team has decided this is normal and deploys at night.

It is four settings, and none of them is the readiness probe.

Predict

A Kubernetes rolling update replaces one pod. The readiness probe is configured and passing. Where do the dropped requests come from?

The new pod is taking traffic before it is ready, because the readiness probe passes before the application can serveThe old pod keeps receiving new connections for a second or two after it was told to shut down, because removing it from the routing rules on every node is eventually consistent and runs independently of the shutdownThe load balancer health check needs two consecutive failures, so it keeps sending traffic until the second oneIn-flight requests are cut off when the container is killed at the end of the grace period

The sequence, in the order it actually happens

A pod is deleted — by a rolling update, a scale-down, an eviction, a node drain. From that instant:

The API server marks the pod for deletion and sets a deletion timestamp.
In parallel, the endpoint controller removes the pod's address from its EndpointSlice. That change then has to reach the proxy on every node and be written into that node's forwarding rules. This is eventually consistent and takes a short but real amount of time — longer on a larger cluster, longer still if any controller in the path is behind.
Also in parallel, the kubelet runs the container's preStop hook, waits for it to finish, and then sends SIGTERM to process one.
The grace period runs. At its end, anything still alive gets SIGKILL.

Nothing in that list makes step 3 wait for step 2. If the container stops accepting connections the moment it gets the signal, every connection routed to it by a node whose rules have not caught up is refused — and refused by the proxy, which is why the application logs are empty.

The four settings

One: a preStop hook that does nothing but wait. The signal is not sent until the hook returns, so a hook that sleeps is the only mechanism that makes the container outlive the routing update. It looks absurd and it is the single highest-value line in this lesson.

lifecycle:
  preStop:
    exec:
      command: ["/bin/sh", "-c", "sleep 10"]

Pick the duration from the propagation delay you measure, not from a blog post. The honest way to measure it: delete a pod while offering steady load, and record the time between the deletion timestamp and the last request the pod received. Double it.

Two: terminationGracePeriodSeconds longer than the hook plus the longest request. The grace period is the budget for everything in steps 3 and 4, hook included. A hook that sleeps for ten seconds inside a default grace period of thirty leaves twenty for the application to finish in-flight work, and if your slowest request takes longer than that the kill arrives mid-request.

spec:
  terminationGracePeriodSeconds: 45
  containers:
    - name: api
      lifecycle:
        preStop:
          exec:
            command: ["/bin/sh", "-c", "sleep 10"]

Three: an application that actually handles the signal. This is the setting that is not a setting, and the one most often missing. On the signal the application should stop accepting new connections, finish what it holds, close idle keep-alive connections, and then exit — and the keep-alive clause is the one that gets forgotten, because a client holding an idle connection to a departing pod will send its next request down it.

The lab's own service is a worked example of getting this wrong. It installs no handler at all, so the default disposition applies and the process terminates at once, abandoning every connection on its queue. You can watch it happen:

docker compose run -d --rm load --rate 40 --seconds 60
docker compose stop service      # SIGTERM, then SIGKILL after the timeout
docker compose logs load

Every request in flight and every request queued becomes an error in the generator's output. Nothing appears in the service's log, because there is no code that runs on the way out.

What the real frameworks need, named: gunicorn finishes current requests on SIGTERM and bounds that with --graceful-timeout; a Starlette or FastAPI application needs its shutdown work in the lifespan handler, which runs on the signal; a Node HTTP server needs server.close() plus closeIdleConnections or the keep-alive sockets stay open until their own timeout; Spring Boot needs server.shutdown=graceful set explicitly, because the default is immediate, and spring.lifecycle.timeout-per-shutdown-phase to bound it.

Four: a rolling update that never dips below capacity, and probes that do different jobs. The default update strategy allows a quarter of the fleet to be unavailable at once, which is a capacity decision disguised as a deployment setting. And the readiness probe decides whether a pod receives traffic while the liveness probe decides whether it is restarted — pointing both at the same endpoint with the same thresholds means that a pod which is merely overloaded gets killed, which is the most reliable way to turn a latency incident into an outage.

strategy:
  type: RollingUpdate
  rollingUpdate:
    maxUnavailable: 0
    maxSurge: 1
readinessProbe:
  httpGet: { path: /ready, port: 8080 }
  periodSeconds: 2
  failureThreshold: 2
livenessProbe:
  httpGet: { path: /alive, port: 8080 }
  periodSeconds: 10
  failureThreshold: 3
  timeoutSeconds: 5
startupProbe:
  httpGet: { path: /alive, port: 8080 }
  periodSeconds: 5
  failureThreshold: 30

/ready should answer for the dependencies the pod needs to serve. /alive should answer for the process being able to run at all, and nothing else — a liveness probe that checks the database restarts every pod in the fleet the moment the database has a bad minute.

What each of these costs

This is the part the setting names hide, and it is why the defaults are what they are.

setting	what it buys	what it costs
`preStop` sleep	the routing race closes	every pod replacement takes that much longer
longer grace period	in-flight requests survive	a node drain, an eviction and a rollback all take longer
graceful shutdown in the app	in-flight requests survive	code that must be written and tested once per service
`maxUnavailable: 0` with `maxSurge: 1`	capacity never dips	one extra pod's resources for the length of the rollout

The arithmetic worth doing before you set these: the sleep plus the grace period multiplied by the number of pods, divided by the surge, is how long your rollout now takes. On a fleet of a few pods that is a minute. On a fleet of a few hundred with a one-pod surge it is hours, and the setting that made deploys safe has made a rollback slower than the incident it is meant to end. The balance is maxSurge — a larger surge is the thing you buy with capacity to keep the rollout short — and the reason it is a cost question rather than a correctness one.

What would you do

Thirty pods, one deployment, a rolling update with the default strategy. A cloud load balancer in front of the cluster, targeting node ports, with its own deregistration delay still at the provider's default of several minutes.

The application exits immediately on the shutdown signal — there is no handler. The readiness and liveness probes both point at /health, which returns a constant. The error spike on each deploy lasts roughly eight seconds per pod replacement and the deploy takes a few minutes in total.

You can change the manifest today. Changing the application needs a review and a release.

Optimising for: Eliminate the error spike on deploy without making a full rollback take longer than five minutes, on a fleet of thirty pods behind a cloud load balancer.
You cannot know: How long endpoint removal actually takes to reach every node in this cluster, which nobody has measured and which depends on its size
Whether the errors are resets from the in-cluster proxy, gateway errors from the external balancer, or both in unknown proportion
How long the slowest request to this service takes, which bounds the grace period you need
Whether the load balancer is targeting pod addresses directly or going through node ports, which decides whether its deregistration delay is even in the path

Add a preStop sleep and raise the grace period, leaving the application and the probes aloneWrite the graceful shutdown handler first, since that is the real bugSet maxUnavailable to zero with a surge, so capacity never dips during the rolloutLower the load balancer's deregistration delay to a few seconds so targets leave the pool quickly

Check

You add the preStop sleep and the error spike on deploy closes completely. Six months later it comes back, and nothing in the deployment manifest has changed. What is the most likely cause?

The cluster grew, so endpoint removal now takes longer to reach every node than the sleep coversThe application got slower, so in-flight requests no longer finish inside the grace periodA new Kubernetes version changed the default grace periodThe readiness probe is now failing before termination, so the pod is removed from the pool twice

Where this stops holding

All four settings exist to keep traffic flowing to an instance that is on its way out, which assumes the deploy is incremental and some of the fleet is always serving.

The crossing to watch for: a deploy that replaces the whole fleet at once on purpose — a recreate strategy, a stop-the-world migration, a cutover between two clusters — has no instance to drain traffic to, so draining is not the mechanism and none of this applies. There the window is the full startup time of the new version, the honest answers are a maintenance window or a blue-green switch with both versions running, and the thing to get right is the order of the database migration rather than the shutdown signal.

The same is true at one replica. A single pod being replaced has nowhere to send the traffic during the gap, so a surge is not optional, it is the only fix — and at one replica maxUnavailable: 0 with maxSurge: 1 is the difference between a drop and no drop rather than a tuning choice.

As of 2026-10-31

Dated as of this page, and these are the parts most likely to have moved: Kubernetes field names and defaults — terminationGracePeriodSeconds and its default of thirty seconds, the lifecycle.preStop exec hook, the RollingUpdate strategy's maxUnavailable default of twenty-five per cent and maxSurge, the readinessProbe / livenessProbe / startupProbe fields and their threshold defaults, and EndpointSlice as the object the endpoint controller writes. The claim that endpoint removal and container termination begin concurrently and are not ordered is a property of the design rather than a version detail, but the time it takes depends on your cluster and must be measured there. Also dated: gunicorn's --graceful-timeout, Node's server.closeIdleConnections, Spring Boot's server.shutdown and spring.lifecycle.timeout-per-shutdown-phase, and the claim that a cloud load balancer's target deregistration delay defaults to several minutes — that is a per-provider default that has been changed before. Check the version you run.

Worth remembering

Deleting a pod starts two independent processes: the container terminating and the address being withdrawn from every node's routing rules. Nothing orders them, so a container that exits promptly stops serving while traffic is still arriving.
The refusals therefore appear in the proxy and not in the application log, which is why the application looks innocent.
A preStop hook that only sleeps is the single highest-value fix, because the shutdown signal is not sent until the hook returns.
The grace period is the budget for the hook and the shutdown together, so a long hook inside a default grace period leaves less for in-flight work than it appears to.
The signal handler must close idle keep-alive connections as well as finishing in-flight requests, or a client sends its next request down a connection to a departing instance.
A readiness probe decides whether traffic arrives and a liveness probe decides whether the container is restarted; pointing both at the same endpoint turns an overload into a fleet-wide restart.
Order the fixes by what they cover: the sleep closes the arrivals-after-shutdown window, the handler closes the in-flight window, and the surge setting covers capacity during the rollout.
Every one of these settings is paid for in rollout duration, which multiplied by the fleet size is the rollback budget — so the surge is a cost decision and not a safety one.
A sleep duration chosen from a measurement on a small cluster expires silently as the cluster grows, and the symptom is the original spike returning with nothing changed on your side.