Ingress returns 502

Reported symptoms

The checkout API sits behind the cluster’s Ingress controller. Two or three times a week, for twenty to sixty seconds, a fraction of requests come back as 502 Bad Gateway. Then it stops.

The application team has looked, more than once, and has nothing:

• The service’s own request log has no entry for the failed requests. Not an error entry - no entry.
• Its 5xx counter is flat across every burst.
• kubectl get pods shows the full replica count Running and Ready before, during and after, with a restart count of zero.

The platform team has also looked, and also has nothing. The Ingress controller Pods have not restarted. Their CPU and memory are flat. Their own liveness and readiness probes have never failed. Two other services behind the same controller have never produced a single 502, which is the observation that keeps sending the investigation back to the application team.

There is one more data point that everybody has, and nobody has used. The 502 rate roughly doubled the week the application team shipped a change that made shutdown faster - they had been annoyed by slow deploys, so they made the process exit immediately on SIGTERM instead of winding down over half a minute. Deploys did get faster. The 502s got worse. Nobody connected the two, because “we made shutdown cleaner” is not a plausible cause of an increase in errors.

Evidence provided

$ kubectl logs -n ingress-nginx deploy/ingress-nginx-controller | grep ' 502 '

10.0.4.7 - - [12/Aug/2026:12:41:07 +0000] "POST /v1/checkout HTTP/2.0" 502 150 412 0.001 [prod-checkout-8080] 10.244.3.41:8080 0 0.001 502
10.0.4.9 - - [12/Aug/2026:12:41:07 +0000] "POST /v1/checkout HTTP/2.0" 502 150 409 0.001 [prod-checkout-8080] 10.244.3.41:8080 0 0.001 502
10.0.4.7 - - [12/Aug/2026:12:41:08 +0000] "GET /v1/cart HTTP/2.0"      502 150 331 0.001 [prod-checkout-8080] 10.244.3.41:8080 0 0.001 502

The upstream address is the same on every line, and the upstream response time is a millisecond. Whatever happened, it happened at connection setup - the controller did not wait on a slow application.

$ kubectl get pods -n prod -l app=checkout -o wide

NAME                        READY   STATUS    RESTARTS   AGE   IP             NODE
checkout-7f9c4d8b6-2xk9q    1/1     Running   0          4m    10.244.1.18    worker-02
checkout-7f9c4d8b6-9wvbt    1/1     Running   0          4m    10.244.2.27    worker-03
checkout-7f9c4d8b6-lm4dz    1/1     Running   0          4m    10.244.4.11    worker-01

$ kubectl logs -n ingress-nginx deploy/ingress-nginx-controller | grep 10.244.3.41 | grep error

2026/08/12 12:41:07 [error] 34#34: *8891 connect() failed (111: Connection refused) while connecting to upstream, client: 10.0.4.7, server: checkout.example.com, request: "POST /v1/checkout HTTP/2.0", upstream: "http://10.244.3.41:8080/v1/checkout", host: "checkout.example.com"

$ kubectl get events -n prod --sort-by=.lastTimestamp | grep -i 'replica set'

4m    Normal   ScalingReplicaSet   deployment/checkout   Scaled up replica set checkout-7f9c4d8b6 to 3
4m    Normal   ScalingReplicaSet   deployment/checkout   Scaled down replica set checkout-6b8d5f2a4 to 0

$ kubectl get deploy checkout -n prod -o yaml | grep -nE 'terminationGracePeriodSeconds|lifecycle:'

42:      terminationGracePeriodSeconds: 30

$ kubectl logs -n prod checkout-6b8d5f2a4-t7rp2 --previous | tail -3

{"ts":"2026-08-12T12:41:06.418Z","level":"info","msg":"SIGTERM received, closing listener"}
{"ts":"2026-08-12T12:41:06.601Z","level":"info","msg":"in-flight requests drained"}
{"ts":"2026-08-12T12:41:07.104Z","level":"info","msg":"shutdown complete"}

Work the evidence before reading on

Nothing here is broken. The application shuts down correctly. The controller is healthy. The rollout succeeded. And a fraction of real customer requests were answered with a 502.

1. The upstream address in the 502 lines belongs to no running Pod. What was it a moment earlier, and what does that tell you about when the controller learned of the change?
2. connect() failed (111: Connection refused) is not a timeout and not a reset mid-response. What state must a target host be in to refuse a connection that quickly?
3. Compare the timestamp of shutdown complete in the application log with the timestamp of the first 502. Which came first?
4. Two other services on the same controller never see this. What might be different about them that has nothing to do with the controller?

Before continuing: the application closed its listener at 12:41:06.418. The controller sent it a request at 12:41:07. Who was wrong?

Root cause

1. Two clocks that nobody synchronises

When a rollout deletes a Pod, two things start at once and finish independently.

On the node, the kubelet sees the deletionTimestamp, runs the preStop hook if there is one, and sends SIGTERM. That path is short and local.

In the control plane, the EndpointSlice controller reacts to the same deletion and rewrites the Service’s slice; the API server publishes the change; every watcher picks it up; and the Ingress controller then applies it to its own running configuration. That path crosses three components and a network.

There is no barrier between them. Nothing in Kubernetes holds SIGTERM until the address has been withdrawn everywhere, because nothing in the control plane knows who “everywhere” is. The gap between the two is the window in which the Ingress controller still holds an address that has stopped listening.

2. The application is fast, and that is the problem

This application does everything the course asks of it. It catches SIGTERM, it stops accepting new connections, it drains what is in flight, and it exits with code 0 in under a second.

The window is bounded by how long the withdrawal takes to propagate. It is not bounded by how long the process lives. So a process that exits in one second spends the rest of that window as a closed port at an address the controller still believes in - and a closed port refuses connections instantly, which is exactly the connect() failed (111: Connection refused) in the error log.

The team’s “make shutdown faster” change did not create the fault. It widened it, by removing the accidental delay that a slow shutdown had been providing. That is why the improvement looked like a coincidence: it was a real change, in the right direction, that made a latent defect visible.

3. The application’s logs were never going to show it

The requests were refused at connection setup. They never reached the process, so they were never logged, never counted, and never appeared in the service’s 5xx metric.

An entire class of failure lives in this gap: the Ingress controller records what it observed, the application records what it served, and requests that died between the two appear only in the first. Reconciling the controller’s 5xx rate against the application’s own is the check that finds them, and almost nobody runs it until an incident forces them to.

Resolution

1. Measure the window before changing anything. Watch the Service EndpointSlice with kubectl get endpointslices -w while deleting one Pod, and record how long the terminating address stays listed and how long the controller keeps sending to it. That number, not a default from a blog post, is the size of the problem.
2. Fix it in the application if you can. On SIGTERM: report not-ready immediately, keep serving, wait out the measured window, then close the listener and exit. The application knows when it is safe to stop; the kubelet does not.
3. Use a preStop hook only where the application cannot be changed. A hook that sleeps for the measured interval buys the same delay from outside, at the cost of holding a Pod that is already doomed.
4. Raise terminationGracePeriodSeconds to cover the delay plus the longest in-flight request plus headroom. The grace period includes the preStop hook, so a hook added without raising the grace period takes its time from the drain.
5. Change one Deployment first. Roll it under load and confirm the count of non-200 responses is zero before the pattern goes anywhere near the other services.
6. Reconcile the two error rates as a standing check: the controller 5xx rate for this backend against the application own 5xx rate. A divergence is the signature of this whole failure class.
7. Roll the pattern out deliberately, service by service, measuring each. A cluster-wide default applied in one change gives you no way to tell which service it helped and which one it slowed down.
8. Remove any client-side retry that was added to mask the bursts, or at least confirm it is scoped to idempotent requests. A retry over a non-idempotent checkout call is a duplicate order, not a fix.

Verification

1. A rollout under load produces zero non-200 responses. Drive steady traffic through the Ingress, run kubectl rollout restart, and count. Lower is not the goal; zero is, because the window either closes before the listener does or it does not.
2. The test runs at production shape. Two replicas at ten requests per second can miss a window that twenty replicas at two thousand will find every time.
3. The mechanism is confirmed independently of the outcome. During a termination, the application still answers on its own port after its address has left the EndpointSlice. That is the ordering the fix exists to guarantee.
4. No Pod is being killed at the deadline. Check the exit codes of terminated Pods: an exit code of 137 means SIGKILL arrived, which means the delay is being paid out of the time in-flight requests needed.
5. The controller 5xx rate for this backend and the application own 5xx rate now agree. While they disagree, requests are still dying between the two.
6. A deliberately broken canary still fails the test. Remove the drain delay from a scratch Deployment and confirm the rollout test goes red. A test that has never failed is not evidence.
7. Rollout duration is measured and accepted. The fix makes every termination slower by the delay; the team should know the new number for a full rollout and for a node drain before it is needed in an incident.

Prevention

• Make rollout-under-load a staging gate. A Deployment that drops requests during its own rollout should fail in the pipeline, not in production. This is the single control that would have caught the original defect and the “faster shutdown” change that widened it.

• Gate the manifest shape too. A workload behind an Ingress that declares neither an in-process drain nor a preStop hook is a defect an admission policy or a CI check can catch for almost nothing:

spec:
  terminationGracePeriodSeconds: 60
  containers:
    - name: checkout
      lifecycle:
        preStop:
          exec:
            command: ["sleep", "15"]

Prefer the in-process version of this; ship the hook where you cannot.

• Alert on the controller’s 5xx rate per upstream, not in aggregate. A 60-second burst on one backend vanishes into a cluster-wide error rate and reappears only as a customer complaint.

• Record the measured propagation delay for this cluster. It is a property of the cluster’s size, its controller and its configuration, and every workload that must survive a rollout needs the same number. Leaving each team to rediscover it guarantees a range of wrong guesses.

• Reconcile controller-observed errors against application-observed errors. Any failure that happens between the proxy and the process is invisible on one side and obvious on the other. Comparing them is cheap and it is the only routine check that sees this class at all.

• Treat “we made shutdown faster” as a change to traffic handling. It reads like a build-time optimisation and behaves like a routing change. It belongs in the same review as a probe change or a Service edit.