kube-proxy troubleshooting — iptables, IPVS, and eBPF diagnostics

kube-proxy troubleshooting requires mode-specific diagnostics. The iptables mode uses iptables-save and the kube-proxy logs. The IPVS mode uses ipvsadm and the kernel modules. The eBPF mode uses cilium status and bpftool. This lesson walks the diagnostics for each mode and the operational discipline.

The diagnostic flow

The diagnostic flow for the kube-proxy:

flowchart TD
    A[Service degraded] --> B{Cluster mode?}
    B -->|iptables| C[Check iptables rules]
    B -->|IPVS| D[Check IPVS rules]
    B -->|eBPF| E[Check eBPF maps]
    C --> A1[Rule count?]
    C --> A2[Sync duration?]
    C --> A3[kube-proxy logs?]
    D --> D1[IPVS rules?]
    D --> D2[Kernel modules?]
    D --> D3[Scheduler?]
    E --> E1[Cilium status?]
    E --> E2[Kernel version?]
    E --> E3[BPF maps?]
    A1 --> F[Apply fix]
    D1 --> F
    E1 --> F

The diagnostic flow is mode-specific. The cluster operator must understand the mode’s diagnostics.

The iptables diagnostics

The iptables diagnostics:

# Substitute your own value before running:
NODE_IP=192.0.2.21

# The iptables rule count
iptables-save | wc -l

# The kube-proxy rules
iptables-save | grep KUBE

# The kube-proxy's logs
kubectl logs -n kube-system -l k8s-app=kube-proxy

# The kube-proxy's metrics
curl "http://$NODE_IP:10249/metrics"

The diagnostics show the rule count, the rule content, the kube-proxy’s logs, and the metrics. The cluster operator must understand the iptables diagnostics.

The IPVS diagnostics

The IPVS diagnostics:

# Substitute your own value before running:
NODE_IP=192.0.2.21

# The IPVS rules
ipvsadm -Ln

# The IPVS kernel modules
lsmod | grep ip_vs

# The kube-proxy's logs
kubectl logs -n kube-system -l k8s-app=kube-proxy

# The kube-proxy's metrics
curl "http://$NODE_IP:10249/metrics"

The IPVS diagnostics show the rules, the kernel modules, the kube-proxy’s logs, and the metrics. The cluster operator must understand the IPVS diagnostics.

The eBPF diagnostics

The eBPF diagnostics:

# The Cilium status
cilium status

# The BPF maps
bpftool map list

# The cilium-agent's logs
kubectl logs -n kube-system -l k8s-app=cilium

# The Cilium metrics
cilium monitor --type trace

The eBPF diagnostics show the Cilium status, the BPF maps, the agent’s logs, and the metrics. The cluster operator must understand the eBPF diagnostics.

The common failure modes

The common failure modes of kube-proxy:

• kube-proxy down: the kube-proxy Pod is not running. The fix is to restart the Pod.
• Watch loop broken: the kube-proxy is not receiving events. The fix is to check the API connectivity.
• Dataplane rule count mismatch: the rule count does not match the Service count. The fix is to restart the kube-proxy.
• Sync duration increasing: the sync duration is too high. The fix is to switch to a faster mode or to investigate the API server.
• EndpointSlice updates lost: the kube-proxy misses an update. The fix is to restart the kube-proxy.

The metrics

The kube-proxy’s metrics are the leading indicator:

# Substitute your own value before running:
NODE_IP=192.0.2.21

# The kube-proxy's metrics
curl "http://$NODE_IP:10249/metrics"

kubeproxy_sync_proxy_rules_duration_seconds_count 100
kubeproxy_networkprogramming_latency_seconds_bucket{le="0.1"} 250
iptables_rules_count 5000
kubeproxy_ipvs_healthcheck_failures_counter 0

The metrics show the sync duration, the network programming latency, the iptables rule count, and the IPVS health check failures. The cluster operator monitors these metrics.

The operational discipline

The kube-proxy troubleshooting’s operational discipline:

• Document the kube-proxy mode. The cluster operator must understand which mode is used.
• Run the diagnostics regularly. The diagnostics are the leading indicator.
• Monitor the metrics. The metrics are the leading indicator of degradation.
• Test the recovery in staging. The recovery must be tested before production.
• Document the diagnostics. The diagnostics are the cluster’s operational reference.
• Plan the mode’s evolution. The mode can be changed; the change is significant.
• Train the operations team on the diagnostics. The diagnostics are the team’s tools.

Quiz

Production discipline

• The diagnostic flow is mode-specific. The cluster operator must understand the mode’s diagnostics.
• Monitor the kube-proxy’s metrics. The metrics are the leading indicator of degradation.
• Run the diagnostics regularly. The diagnostics are the leading indicator.
• Test the recovery in staging. The recovery must be tested before production.
• Document the diagnostics. The diagnostics are the cluster’s operational reference.
• Plan the mode’s evolution. The mode can be changed; the change is significant.
• Train the operations team on the diagnostics. The diagnostics are the team’s tools.
• Document the diagnostics in the runbook. The runbook is the cluster’s operational reference.
• Set up alerts on the metrics. The alerts are the leading indicator of outages.