Service wrong targetPort

Reported symptoms

At 11:04 the ingress controller started returning 502 for /pay. Within a minute, five services calling payments.prod.svc.cluster.local were failing too. The callers are not timing out; they are being refused, which is fast enough that the retry storm arrived before the page did.

The on-call ran the standard first check and cleared the Service in two minutes:

$ kubectl describe service payments -n prod | grep Endpoints
Endpoints:  10.244.3.11:8080,10.244.5.22:8080,10.244.7.14:8080

Three addresses. Not an empty EndpointSlice, not a selector problem. The Pods are 1/1 Ready and have been continuously through the incident, and every readiness probe has passed on every interval.

The application is also fine. Port-forwarding to a payments Pod and calling it directly returns 200 for /healthz and, more convincingly, 200 for /pay with a real request body.

So: DNS resolves, the Service has endpoints, the Pods are ready, the application works. That combination is what sent the first thirty minutes into the CNI, the NetworkPolicy that a different team had applied that morning, and the ingress controller’s upstream configuration.

One fact did not fit, and it is the one that solves the incident. The nightly reconciliation job is completely unaffected. It calls the same three Pods, from the same cluster, over the same network - through a second Service named payments-internal.

The change log for the day has one entry, at 11:03. It removed a sidecar proxy from the payments Deployment. It was reviewed as “no functional change”, and the mesh team confirms the removal was intentional and that the application never depended on the proxy for anything.

Evidence provided

$ kubectl describe service payments -n prod | grep -E 'Port:|TargetPort|Endpoints'

Port:              http  80/TCP
TargetPort:        8080/TCP
Endpoints:         10.244.3.11:8080,10.244.5.22:8080,10.244.7.14:8080

$ kubectl describe service payments-internal -n prod | grep -E 'Port:|TargetPort|Endpoints'

Port:              http  80/TCP
TargetPort:        9090/TCP
Endpoints:         10.244.3.11:9090,10.244.5.22:9090,10.244.7.14:9090

$ kubectl get pod payments-6c5f8b9d47-2vhwq -n prod -o jsonpath='{.spec.containers[*].name}'

payments

$ kubectl get pod payments-6c5f8b9d47-2vhwq -n prod -o jsonpath='{.spec.containers[0].ports}{.spec.containers[0].readinessProbe.httpGet.port}'

[{"containerPort":9090,"name":"http","protocol":"TCP"}]9090

$ kubectl run probe --rm -it --image=curlimages/curl --restart=Never -- sh -c 'curl -s -o /dev/null -w %{http_code} http://10.244.3.11:9090/healthz; curl -s -m 3 http://10.244.3.11:8080/healthz'

200
curl: (7) Failed to connect to 10.244.3.11 port 8080: Connection refused

Work the evidence before reading on

Every layer of the canonical flow answers correctly until one of them does not. Walk it: DNS, Service, EndpointSlice, Pod IP, application port.

1. Two Services select the same three Pods. One works and one does not. Rule out, one by one, every explanation that lives outside the Service objects themselves - DNS, CNI, NetworkPolicy, the ingress controller, the Pods. What is left?
2. The endpoints are populated. Read the port on each address rather than the address. What port is the EndpointSlice telling kube-proxy to send traffic to, and where did that number come from?
3. The readiness probe passes on every interval. Write down, precisely, what a passing readiness probe proves about the Pod - and what it does not.
4. The change removed a container. What did that container have that the Deployment manifest is no longer the only file to have referenced?

Before continuing, answer this: the Pods are in the EndpointSlice because they are Ready. What would have had to be true for them to be excluded, and would that condition have caught this fault?

Root cause

1. The Service targets a port that nothing listens on

Until 11:03 the payments Pods ran two containers: the application on 9090, and a sidecar proxy on 8080 that forwarded to it. The Service was written against the sidecar. targetPort: 8080 was correct, and had been for as long as the sidecar existed.

The change removed the sidecar. It removed the container, and with it the only process listening on 8080 inside the Pod’s network namespace. The Service was never touched, so it still sends every request to port 8080 on a Pod where nothing is bound to it, and the Pod’s kernel refuses the connection immediately.

That is why callers are refused rather than timed out, and it is a useful distinction to keep: a refusal means the packet arrived somewhere and was rejected by a live host, so the routing worked and the listener did not exist. A timeout would have meant the opposite.

2. Nothing validates a targetPort against anything

There are three independent statements about ports in this workload, and the platform does not compare them:

The API server accepts a Service whose targetPort matches no container. The EndpointSlice controller copies the resolved number into the slice without examining it. kube-proxy programs whatever the slice says. At no point does any component ask whether anything is listening there.

3. Readiness answered a different question

This is the part that cost thirty minutes.

A Pod appears in an EndpointSlice because it matched the selector and its readiness probe passed. The probe targets 9090, the application is healthy on 9090, so the probe passes and the Pod is Ready - correctly, and it will stay correct for as long as the outage lasts.

Readiness is a claim about the container. Reachability through the Service is a claim about a port number written in a different object. Kubernetes verifies the first and assumes the second, so on this fault the endpoint list is populated, green, accurate, and completely irrelevant.

The second Service is the control experiment the incident handed you for free. payments-internal selects the same three Pods over the same network, and it works, because it was written later against the application instead of against the proxy in front of it. When two Services in front of one workload disagree, the disagreement is in the Service objects. Nothing else can produce it.

Resolution

1. Confirm the port mismatch from the EndpointSlice rather than from the Service. kubectl get endpointslices -n prod -l kubernetes.io/service-name=payments -o yaml shows the port kube-proxy is actually programming, which is the number that matters.
2. Confirm what the Pod is listening on from outside it. A request from a debug Pod to the Pod IP on each candidate port answers this without needing a shell, a package, or a debug sidecar in the target image.
3. Set targetPort: 9090 on the payments Service in the repository that owns it, and apply from there. Patching the live object first is defensible while callers are down, but only if the repository change lands in the same incident.
4. Do not roll back the sidecar removal unless the Service change is genuinely blocked. It is slower, it undoes work another team did on purpose, and it repairs the outage by restoring the thing that was correctly removed.
5. Do not make the application listen on 8080 as well. That treats a stale manifest as fixed and leaves the next reader two listeners and no explanation for either.
6. Watch the EndpointSlice rewrite before testing traffic. The controller updates the slice and kube-proxy reprograms from it; if the slice still says 8080, the change has not taken effect and no amount of curling will help.
7. Check the other Service. payments-internal was working throughout and must still be working afterwards - a repair that regresses the one healthy path is not a repair.
8. Once traffic is restored, decide whether to move to a named target port. targetPort: http against a container declaring name: http, containerPort: 9090 keeps the number in one place. It is the better shape and it has its own failure mode, so make it a deliberate follow-up rather than part of the incident fix.

Verification

1. The EndpointSlice carries the new port. kubectl get endpointslices -n prod -l kubernetes.io/service-name=payments -o yaml shows port: 9090 on each endpoint, and kubectl describe service payments -n prod shows TargetPort 9090/TCP.
2. A request through the ClusterIP succeeds. From a debug Pod, call the payments ClusterIP on port 80 and require a 200. Do not verify with a port-forward and do not verify against a Pod IP - reaching the application directly is precisely what worked throughout the outage.
3. The ingress path returns 200 for /pay. That caller reached the Service by a different route and raised the incident, so it is the one that proves the recovery is complete.
4. The callers stopped retrying. Connection-refused counters on the five affected services return to zero rather than merely falling, since a partially reprogrammed dataplane looks like improvement.
5. payments-internal still returns 200. The healthy path must survive the fix.
6. The blind spot is demonstrated, not assumed. On a staging Service, set a targetPort that nothing listens on and confirm the Pods stay Ready and the endpoints stay populated. This is what makes the case for the new check to whoever has to approve it.
7. The new post-deploy check fails against that staging Service. A validation step that passes on a cluster you have deliberately broken is not a validation step.

Prevention

• Assert through the Service after every rollout. One request to the ClusterIP is the check that none of readiness, rollout status or a port-forward performs. All three were green for the entire outage, and all three were answering questions nobody had asked.
• Name the ports. A container port declared with a name, and a Service that targets the name, keeps the number in exactly one place. The number then moves with the container instead of being maintained in two files by two teams.
• Treat a sidecar’s port as a published interface. It looks like an implementation detail and it is referenced by Services, network policies and probes in other files. Removing a sidecar means enumerating what pointed at its port, not reviewing the Deployment on its own.
• Add the comparison the platform declines to make. containerPort is documentation, but it is checkable documentation: a rule that flags a Service whose numeric targetPort is declared by no container it selects would have failed this change in review, in about a second.
• Learn the three signatures. Ready endpoints with connections refused is a port mismatch. No endpoints at all is a selector or readiness problem. A timeout rather than a refusal is a network path problem. Those separate cleanly, and picking the right one is worth more than any single command in this file.
• Keep a second Service in mind as a diagnostic. When two Services front one workload and only one of them fails, everything outside the Service objects is eliminated at once. That is a rare gift and it is worth noticing quickly when the cluster hands it to you.