Broken selectors and orphan services — the routing failure

A broken selector is a Service whose selector does not match the Pods. The Service has no endpoints; the synthetic traffic fails. The diagnostic is the selector and the labels. The discipline is to compare the Service’s spec.selector to the Pod’s labels.

The selector

A Service’s spec.selector is the set of labels that the EndpointSlice controller uses to populate the Service’s endpoints. The controller watches all Pods and matches their labels against the selector. A Pod whose labels match is included in the EndpointSlice.

flowchart LR
    A[Service selector: app=billing] --> B{Pod labels match?}
    B -->|Yes| C[In EndpointSlice]
    B -->|No| D[Not in EndpointSlice]

A broken selector is one that does not match the Pod’s labels. The Service has no endpoints. The synthetic traffic fails.

The diagnostic

A real kubectl describe service for a broken selector:

Name:              billing
Namespace:         prod
Labels:            app=billing
Annotations:       <none>
Selector:          app=billing,tier=api
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.96.45.123
IPs:               10.96.45.123
Port:              http  8080/TCP
TargetPort:        8080/TCP
Endpoints:         <none>

The diagnostic is:

• Selector: app=billing,tier=api — the Service requires both labels.
• Endpoints: <none> — no Pods match.

The Pod’s labels:

kubectl get pods -n prod -l app=billing -o wide

The output may show:

NAME                       READY   STATUS    RESTARTS   AGE   LABELS
billing-7d8f-abcde         1/1     Running   0          5m    app=billing
billing-7d8f-def01         1/1     Running   0          5m    app=billing

The Pods have app=billing but not tier=api. The selector requires both. The Service has no endpoints.

The remediation

The remediation is to either:

1. Fix the Pod’s labels. Add the missing label to the Pod’s spec.
2. Fix the Service’s selector. Remove the tier=api requirement from the Service’s selector.

The remediation depends on the intent. If the Service is intended to route only to the API tier, the Pods need the tier=api label. If the Service is intended to route to all billing Pods, the Service’s selector is wrong.

# Option 1: Fix the Pod's labels
kubectl label pod billing-7d8f-abcde -n prod tier=api

# Option 2: Fix the Service's selector
kubectl patch service billing -n prod -p '{"spec":{"selector":{"app":"billing"}}}'

Common causes of broken selectors

• Typo in the selector. The selector has a typo (app=bililng instead of app=billing).
• Missing label on the Pod. The Pod’s spec is missing a label that the selector requires.
• Renamed Pod label. The Pod’s label was renamed in the Deployment’s template, but the Service’s selector was not updated.
• Namespace mismatch. The Service is in a different namespace than the Pods.

Orphan services

An orphan service is a Service whose selector does not match any Pods in the cluster. The Service exists but has no endpoints. The orphan service is the rare case where the Deployment was deleted but the Service was not.

The diagnostic is the same:

kubectl get endpointslices -n prod -l kubernetes.io/service-name=billing
kubectl get pods -n prod -l app=billing

The output shows the EndpointSlice is empty and the Pods exist (with different labels).

Production discipline

A broken selector is the cluster’s hypothesis. The discipline is to compare the Service’s selector to the Pod’s labels. The diagnostic is the EndpointSlice and the Pod labels. The remediation is to fix the selector or the labels.

• Use kubectl get pods —show-labels. The command shows the Pod’s labels.
• Use kubectl describe service. The command shows the Service’s selector and the Endpoints.

Quiz