low riskservice affecting~25 min

Runbook: Troubleshoot Service Connectivity

1 · Prerequisites

Confirm every item is in place before any state change.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Capture the Service definition: kubectl get svc <name> -n <ns> -o yaml
· Capture the EndpointSlice: kubectl get endpointslices -n <ns> -l kubernetes.io/service-name=<name> -o yaml
· Capture the Pods that should back the Service: kubectl get pods -n <ns> -l <selector> -o wide
· Confirm the client and the Service are in the same cluster (no cross-cluster confusion)
· Capture the cluster DNS view of the Service: kubectl run -it --rm --image=registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7 --restart=Never dnstest -- nslookup <name>.<ns>.svc.cluster.local

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Confirm the client can resolve the Service: nslookup <name>.<ns>.svc.cluster.local from inside the cluster
2If resolution fails, see kubernetes-rb-troubleshoot-coredns first
3Confirm the Service exists and has the right selector: kubectl get svc <name> -n <ns> -o jsonpath='{.spec.selector}{" "}{.spec.ports}'
4Confirm the EndpointSlice has at least one address: kubectl get endpointslices -n <ns> -l kubernetes.io/service-name=<name> -o jsonpath='{.items[*].endpoints[*].addresses}'
5If the EndpointSlice is empty, the Service selector does not match any Pod labels (most common cause)
6If endpoints exist, confirm the Pod IPs are reachable from a different Pod: kubectl exec <client> -- nc -vz <pod-ip> <port>
7Confirm the Pod is listening on the Service targetPort: kubectl exec <pod> -- ss -tlnp | grep <port> or curl the probe
8Confirm the Service port-to-targetPort mapping matches: kubectl get svc <name> -n <ns> -o jsonpath='{.spec.ports}'
9Walk the kube-proxy dataplane: iptables-save | grep <svc-cluster-ip> (iptables mode) or check IPVS rules (IPVS mode) for the Service
10Apply the smallest fix that resolves the cause: fix the selector, repair the Pod, restart kube-proxy if the dataplane is stale

4 · Verification

Confirm the procedure actually fixed the problem.

✓kubectl get endpointslices -n <ns> -l kubernetes.io/service-name=<name> shows addresses
✓kubectl get svc <name> -n <ns> shows CLUSTER-IP and the right ports
✓From inside the cluster, the Service is reachable: kubectl exec <client> -- curl -fsS http://<name>.<ns>.svc:<port>/ returns the expected response
✓From outside the cluster (if NodePort/LoadBalancer/Ingress), the public path returns the expected response
✓kubectl get events -n <ns> --field-selector involvedObject.name=<name> shows no new warnings for 5 minutes

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶If a manifest change caused the issue, git revert and kubectl apply
↶If the kube-proxy dataplane is stale, restarting kube-proxy on every node recovers the iptables/IPVS state: kubectl -n kube-system rollout restart ds/kube-proxy (Cilium users do not have kube-proxy; restart the operator instead)
↶If the fix involved changing the Service selector, confirm that no production traffic relied on the previous selector before deletion
↶If the Service was deleted and recreated with the same name, an in-flight DNS cache may still point at the old ClusterIP; the cache expires within ndots/TTL seconds

6 · Escalation

When the runbook isn't enough, contact:

· Service has endpoints but every connection times out at the kube-proxy layer: CNI dataplane is broken; see kubernetes-rb-troubleshoot-cni
· Service resolves and has endpoints but traffic returns the wrong backend: kube-proxy is in a stale state; restart kube-proxy on every node
· Cross-node traffic fails but same-node traffic works: kube-proxy dataplane inconsistency on the affected nodes; isolate the node and restart kube-proxy
· Multiple Services in the same namespace lose endpoints simultaneously: the Endpoints controller is failing; check controller-manager logs
· Service worked and now intermittently 503s: backend Pods are crashing or readiness is failing; see kubernetes-rb-investigate-crashloopbackoff and readiness probe flow

Service connectivity has a precise path: client → DNS → ClusterIP → kube-proxy / CNI → EndpointSlice → Pod IP → Pod port. The runbook walks that path top-down and stops at the first broken link.

1. Walk the path

flowchart LR
    A["Client<br/>Pod"] -->|DNS| B["ClusterIP<br/>(Service)"]
    B -->|kube-proxy / CNI| C["EndpointSlice"]
    C -->|Pod IP| D["Backend<br/>Pod"]
    D -->|targetPort| E["Application"]

Stage	Symptom	Where to look
DNS	`nslookup` returns NXDOMAIN or wrong IP	`kubernetes-rb-troubleshoot-coredns`
ClusterIP	`ping <cluster-ip>` is fine but `curl` times out	CNI dataplane, NetworkPolicy
kube-proxy	`iptables-save` missing the Service’s nat rule	kube-proxy restart
EndpointSlice	Selector does not match Pod labels	Selector typo or Pod label drift
Pod IP	Connection refused	Application not listening on targetPort
targetPort	Port mapping mismatch	`spec.ports[].targetPort`

2. Confirm the Service exists and has endpoints

Read-only / SafeConfirm the Service exists and has endpoints


kubectl get endpointslices -n <ns> -l kubernetes.io/service-name=<name> -o yaml
kubectl get endpoints -n <ns> <name>  # legacy view, included for completeness

If the EndpointSlice is empty, the Service selector does not match any Pod labels. This is the single most common production failure.

Read-only / SafeConfirm the Service exists and has endpoints


kubectl get svc <name> -n <ns> -o jsonpath='{.spec.selector}' | jq
kubectl get pods -n <ns> -l <selector> -o wide
kubectl get pods -n <ns> --show-labels | head

A common pattern: the Deployment has app.kubernetes.io/name=<name> but the Service selects app=<name>. The fix is in the manifest, not the cluster.

3. Confirm the Pod is listening

Read-only / SafeConfirm the Pod is listening


kubectl exec "$POD" -n <ns> -- ss -tlnp
kubectl exec "$POD" -n <ns> -- curl -fsS http://localhost:<target-port>/ || true

If the application is not listening on targetPort, the Service is correct but the Pod is the problem. The Pod is Running and even Ready (probe passes) but the listener died or never started.

4. Walk the dataplane

Read-only / SafeWalk the dataplane


NODE=$(kubectl get pod -n <ns> -l <selector> -o jsonpath='{.items[0].spec.nodeName}')
ssh "$NODE" -- sudo iptables-save | grep <cluster-ip> || echo "no rule"

# IPVS mode
ssh "$NODE" -- sudo ipvsadm -Ln | grep <cluster-ip> || echo "no virtual server"

# Cilium (no kube-proxy)
kubectl -n kube-system logs ds/cilium --tail=50 | grep <svc-cluster-ip> || true

If the rule is missing, the kube-proxy dataplane is stale. Restart it.

5. Test connectivity layer by layer

Read-only / SafeTest connectivity layer by layer


CLIENT=<client-pod>
POD_IP=$(kubectl get pods -n <ns> -l <selector> -o jsonpath='{.items[0].status.podIP}')
CLUSTER_IP=$(kubectl get svc <name> -n <ns> -o jsonpath='{.spec.clusterIP}')
PORT=$(kubectl get svc <name> -n <ns> -o jsonpath='{.spec.ports[0].port}')

# 1. Pod IP, backend port (bypasses Service entirely)
kubectl exec "$CLIENT" -n <ns> -- curl -fsS "http://$POD_IP:<target-port>/" --max-time 5

# 2. ClusterIP, Service port
kubectl exec "$CLIENT" -n <ns> -- curl -fsS "http://$CLUSTER_IP:$PORT/" --max-time 5

# 3. Service DNS
kubectl exec "$CLIENT" -n <ns> -- curl -fsS "http://<name>.<ns>.svc:$PORT/" --max-time 5

The first failure localises the bug. If (1) fails, the backend Pod itself is the problem. If (1) succeeds and (2) fails, kube-proxy is the problem. If (2) succeeds and (3) fails, DNS is the problem.

6. Apply the fix

Read-only / SafeApply the fix


# Fix in Git: change the Service selector to match Pod labels
git commit -am "fix <svc> selector"
kubectl apply -k overlays/prod/workload --record
kubectl get endpointslices -n <ns> -l kubernetes.io/service-name=<name>

# B. kube-proxy stale dataplane
kubectl -n kube-system rollout restart ds/kube-proxy
kubectl -n kube-system rollout status ds/kube-proxy --timeout=5m
ssh "$NODE" -- sudo iptables-save | grep <cluster-ip>

# C. Backend application not listening
# Fix is in the application or its readiness/startup probes
# See kubernetes-rb-investigate-crashloopbackoff

Common pitfalls

Symptom	Cause	Action
Endpoints present, traffic fails from one client only	NetworkPolicy blocks that client	See `kubernetes-rb-troubleshoot-networkpolicy`
Endpoints present, traffic fails cluster-wide	kube-proxy stale	Restart kube-proxy on every node
Endpoints empty after Deployment rollout	New Pods have different labels (e.g. templating error)	Inspect `kubectl get pods --show-labels`
Service works for ClusterIP, fails for NodePort	NodePort firewall rule missing	Open 30000-32767/TCP on the node firewall
Service works cluster-internal, fails from Ingress	Backend port mismatch	Confirm `service.port` matches `ingress.backend.service.port.number`

A Service with no endpoints is not a network problem; it is a label problem. A Service with endpoints but no traffic is a dataplane problem. The runbook distinguishes the two by reading the EndpointSlice first, not by reading the firewall.