Service has no endpoints

Reported symptoms

At 14:07 six services in four namespaces started failing against checkout.prod.svc.cluster.local. The ingress controller began returning 503 for /checkout in the same minute. The failures are immediate: the callers are not waiting for a timeout, they are being refused.

The checkout team sees none of this. Their Deployment reports 6/6 ready. kubectl rollout status says the rollout completed successfully. Their Pods are consuming from their queue, their latency panel is flat, and their error rate is zero, because nothing is reaching them to fail.

The first responder ran the obvious command and got a result that sent everyone in the wrong direction:

$ kubectl get pods -n prod -l app=checkout
No resources found in prod namespace.

That was escalated as “the checkout Pods are gone”, which produced ten minutes of work on scheduling, node capacity and quota. The Pods are not gone. Without the label selector, the same command lists six of them, Running and Ready.

DNS was ruled out early and correctly: the Service name resolves to its ClusterIP from every Pod anyone tested.

The change log for the day contains one deployment at 14:06. It is titled “adopt the recommended Kubernetes labels”, it was reviewed and approved, and it touches only the application repository.

Evidence provided

$ kubectl describe service checkout -n prod | grep -E 'Selector|Endpoints|IP:'

Selector:          app=checkout
IP:                10.96.71.204
Endpoints:         <none>

$ kubectl get endpointslices -n prod -l kubernetes.io/service-name=checkout

No resources found in prod namespace.

$ kubectl get pods -n prod --show-labels | grep checkout

checkout-7f4b96c8d4-4rlq2   1/1   Running   0   14m   app.kubernetes.io/instance=checkout,app.kubernetes.io/name=checkout,pod-template-hash=7f4b96c8d4
checkout-7f4b96c8d4-8wnkd   1/1   Running   0   14m   app.kubernetes.io/instance=checkout,app.kubernetes.io/name=checkout,pod-template-hash=7f4b96c8d4
checkout-7f4b96c8d4-nq6vt   1/1   Running   0   14m   app.kubernetes.io/instance=checkout,app.kubernetes.io/name=checkout,pod-template-hash=7f4b96c8d4

$ kubectl get deployment checkout -n prod -o jsonpath='{.metadata.creationTimestamp}'; echo; kubectl get deployment checkout -n prod -o jsonpath='{.spec.selector.matchLabels}'

2026-08-18T14:07:03Z
{"app.kubernetes.io/instance":"checkout","app.kubernetes.io/name":"checkout"}

$ kubectl exec -n orders deploy/orders -- getent hosts checkout.prod.svc.cluster.local

10.96.71.204    checkout.prod.svc.cluster.local

Work the evidence before reading on

The Deployment is healthy, the Pods are healthy, DNS resolves, and the Service has an IP. Work these in order.

1. The Service’s selector and the Pods’ labels are both in the evidence above. Compare them character by character. Which Pods, anywhere in the cluster, would that selector match?
2. The Deployment’s creationTimestamp is fourteen minutes old, but the workload has been in production for a year. What kind of change deletes and recreates a Deployment rather than updating it, and why would the tooling have had no choice?
3. Callers are being refused immediately rather than timing out. What does that tell you about which of the five layers - DNS, Service, EndpointSlice, Pod IP, application port - is still working?

Before continuing, answer the question that decides the fix: which of the two objects is wrong, the Service or the Deployment? Note that “wrong” here is a question about ownership and intent, not about the API, because the API accepted both of them.

Root cause

1. The selector matches nothing

The Service selects app: checkout. The Pods carry app.kubernetes.io/name=checkout and app.kubernetes.io/instance=checkout and nothing else. There is no Pod in the prod namespace with an app label, so the EndpointSlice controller has nothing to put in a slice, and it does not create one.

That is the entire fault. Everything else on the page is a consequence.

2. Why the change was instantaneous

A Deployment’s spec.selector is immutable. The migration changed the labels the Deployment selects on, which the API server will not accept as an update, so the tooling did the only thing available to it: it deleted the Deployment and created a new one.

Deleting a Deployment deletes its ReplicaSets, which deletes their Pods. The six Pods carrying app: checkout disappeared at 14:07:03 and six Pods carrying the new labels appeared immediately afterwards. The Service’s endpoint count went from six to zero and stayed there.

This is why the incident has no gradual phase, no partial failure and no window in which somebody might have noticed a rising error rate. It also explains the timing precisely enough to identify the change, which is the one thing the evidence gives you for free.

3. The contract nothing enforces

The Service and the Deployment do not reference each other. There is no owner reference, no foreign key, no validating admission on this relationship, and no warning printed by any of the commands involved. Both objects are internally consistent and both are reported healthy.

Consider what each team could see:

Nobody misread anything. The information required to see the fault existed in exactly one place, and it was a place neither team had a reason to look at during a deploy.

Resolution

1. Establish which object is authoritative before editing either one. The Service lives in the platform repository and the Deployment in the application repository; the fix has to be applied where the object is owned, or it will not survive.
2. Decide between the two repairs explicitly and say which one you are doing. The live Service patch is instant and creates drift; adding the legacy label back to the Pod template is durable and costs a rollout. Both are correct answers to different questions.
3. If you patch the Service: kubectl patch service checkout -n prod --type=merge -p with the new selector, then watch the EndpointSlice appear. kube-proxy reprograms from the slice, so the recovery is measured in seconds and needs no Pod to move.
4. If you patch the Deployment: add app: checkout to spec.template.metadata.labels as an additional label. Do not touch spec.selector, which is immutable - adding a Pod label is legal precisely because the existing selector still matches.
5. Do not restart, scale or roll the Deployment in the hope of dislodging the endpoints. The Pods are healthy and every replacement carries the same labels; a restart costs capacity and changes nothing.
6. Confirm the callers recover before declaring the incident over. Six services and an ingress path were failing, and the endpoints reappearing is not the same statement as traffic flowing.
7. Land the corresponding change in the repository that owns the object you patched, inside the incident. This is the step that decides whether the outage recurs.
8. Write the label migration down as a two-release plan before anyone attempts it again: both labels present, selectors moved, old label removed. The change that caused this tried to do all three at once.

Verification

1. The EndpointSlice exists and is populated. kubectl get endpointslices -n prod -l kubernetes.io/service-name=checkout lists six addresses, and each endpoint shows ready: true rather than merely being present.
2. kubectl describe service checkout -n prod lists the addresses under Endpoints. This is the same fact from the other side, and it is the one a responder will check first next time.
3. A caller can connect. Curl the Service name from a Pod in one of the affected namespaces. Every object on this page reported health throughout the outage, so an object-level check is not evidence of reachability.
4. The ingress path returns 200 for /checkout. That caller reached the Service by a different route and noticed the failure first, so it is the one that confirms the recovery is complete rather than partial.
5. The fix survives a reconcile. Force a sync from the platform repository and re-read the endpoints. If the repair was a live patch that has not yet been landed, this is where it vanishes - which is exactly the point of checking now rather than later.
6. The zero-endpoint alert fires when it should. Point a test Service at a selector that matches nothing in a staging namespace and confirm the alert appears. An alert that has only ever been silent has not been tested.
7. The old label is still in place if you took the additive path. Confirm both app=checkout and app.kubernetes.io/name=checkout select the same six Pods, which is the state the migration should have passed through in the first place.

Prevention

• Alert on Services with zero ready endpoints. This is the highest-value rule in the whole scenario. It costs one query, it fires within a minute of the fault, and it names the broken object directly instead of leaving six calling teams to work out whose problem it is.
• Migrate labels additively, over two releases. Add the new label, move every selector to it, verify, then remove the old one. A rename is atomic on the object you edit and manual on everything that refers to it, and nothing will tell you what you missed.
• Do not let kubectl rollout status be the deployment gate. It is a statement about the ReplicaSet. A pipeline step that asserts the Service has the expected number of ready endpoints after the rollout would have failed this change before any caller noticed.
• Review the contract, not the diff. The change that caused this was correct, well-reviewed and complete within its own repository. It was unreviewable there, because the object it broke was not in it.
• Prefer one source for the selector. Where the Service and the workload are templated together, derive both from a single value so that they cannot disagree. Where they cannot be templated together, say so in both files, next to the label.
• Teach the string-match rule. Anything joined by labels in Kubernetes is joined by exact string equality, within one namespace, with no component watching the join. Once an operator holds that, this entire family of faults becomes one hypothesis instead of five.