Service troubleshooting — diagnosing the Service dataplane

Service troubleshooting verifies the Service, the EndpointSlice, the kube-proxy, and the dataplane. The diagnostic tools are kubectl describe, kubectl get endpointslices, iptables-save, ipvsadm. This lesson walks the diagnostic flow, the failure modes, and the operational discipline.

The diagnostic flow

The diagnostic flow for Service traffic:

flowchart TD
    A[Service failed] --> B{Service exists?}
    B -->|No| C[Create Service]
    B -->|Yes| D{EndpointSlice has backends?}
    D -->|No| E[Check Pods]
    D -->|Yes| F{kube-proxy running?}
    F -->|No| G[Restart kube-proxy]
    F -->|Yes| H{dataplane rules present?}
    H -->|No| I[Check sync]
    H -->|Yes| J[Capture traffic]

The diagnostic flow is the operator’s guide. The cluster operator must follow the flow.

Step 1: Verify the Service

The first step is to verify the Service:

# Substitute your own value before running:
SVC=billing

kubectl describe svc "$SVC"

The Service must have the right selector, the right ports, and the right type. The output shows the events and the ClusterIP.

Step 2: Verify the EndpointSlice

The second step is to verify the EndpointSlice:

# Substitute your own value before running:
SVC=billing

kubectl get endpointslices -l kubernetes.io/service-name="$SVC"

The EndpointSlice must have backends. The backends must be Ready.

Step 3: Verify the kube-proxy

The third step is to verify the kube-proxy:

kubectl logs -n kube-system -l k8s-app=kube-proxy

The kube-proxy’s logs show the Service events and the sync events. The operator must verify the kube-proxy is running.

Step 4: Verify the dataplane

The fourth step is to verify the dataplane rules:

# ClusterIP from the `kubectl describe svc` output in step 1:
CLUSTER_IP=10.96.0.42

# iptables
iptables-save | grep KUBE

# IPVS
ipvsadm -Ln | grep "$CLUSTER_IP"

The dataplane rules must be present. The rules are programmed by the kube-proxy.

Step 5: Capture the traffic

The fifth step is to capture the traffic:

tcpdump -i any -n port 80

The capture shows the traffic. The operator can analyze the capture with wireshark or tcpdump.

The failure modes

The Service traffic’s failure modes:

• Service missing: the Service is not created. The fix is to create the Service.
• Selector mismatch: the selector does not match the Pods. The fix is to verify the labels.
• Empty EndpointSlice: the Pods are not Ready. The fix is to verify the readiness probe.
• kube-proxy down: the kube-proxy is not running. The fix is to restart the kube-proxy.
• Dataplane rules corrupted: the rules are corrupted. The fix is to restart the kube-proxy.
• Port mismatch: the Service’s port does not match the Pod’s port. The fix is to verify the ports.

The operational discipline

The Service troubleshooting’s operational discipline:

• Document the diagnostic flow. The flow is the cluster’s operational reference.
• Use the diagnostic tools. The tools are the operator’s primary tools.
• Follow the checklist. The checklist is the operator’s guide.
• Document the troubleshooting. The troubleshooting is the cluster’s operational reference.
• Train the operations team on the diagnostics. The diagnostics are the team’s tools.
• Set up alerts on the kube-proxy metrics. The alerts are the leading indicator of outages.

Quiz

Production discipline

• The diagnostic flow is systematic. The cluster operator must follow the flow.
• Document the diagnostic flow. The flow is the cluster’s operational reference.
• Use the diagnostic tools. The tools are the operator’s primary tools.
• Follow the checklist. The checklist is the operator’s guide.
• Document the troubleshooting. The troubleshooting is the cluster’s operational reference.
• Train the operations team on the diagnostics. The diagnostics are the team’s tools.
• Set up alerts on the kube-proxy metrics. The alerts are the leading indicator of outages.
• Set up alerts on the EndpointSlice. The EndpointSlice is the Service’s data plane.