kube-proxy and iptables/IPVS — the routing engine

Every ClusterIP in the cluster is a fiction until kube-proxy writes the rule that rewrites it to a Pod IP. When kube-proxy stops reconciling nothing looks broken — the Service keeps its address and DNS still resolves it — but connections land on endpoints that were deleted minutes ago, or hang against a chain that was never written. This lesson covers the two rule engines kube-proxy can use, how to read what each one actually programmed on a node, and how to tell a stale rule set from a routing failure that lies elsewhere.

The kube-proxy modes

The kube-proxy has two modes:

1. iptables. The kube-proxy programs iptables rules on every node. Traffic is matched against the rules and DNAT’d to the Pod IP.
2. IPVS. The kube-proxy programs IPVS rules on every node. Traffic is matched against the IPVS virtual servers and forwarded to the Pod IP.

flowchart LR
    A[Service ClusterIP] --> B[kube-proxy]
    B --> C{iptables mode?}
    C -->|Yes| D[iptables rules]
    C -->|No| E[IPVS rules]
    D --> F[Pod IP]
    E --> F[Pod IP]

The mode is configured in the kube-proxy’s ConfigMap:

kubectl get configmap kube-proxy -n kube-system -o yaml | grep mode

The output:

mode: iptables

or:

mode: ipvs

The diagnostic

The canonical diagnostic:

# Substitute your own value before running (the kube-proxy Pod name
# comes from step 2 below):
KUBE_PROXY_POD=kube-proxy-8x2vn

# 1. Check the kube-proxy mode
kubectl get configmap kube-proxy -n kube-system -o yaml | grep mode

# 2. Check the kube-proxy Pods
kubectl get pods -n kube-system -l k8s-app=kube-proxy -o wide

# 3. Check the kube-proxy logs
kubectl logs -n kube-system -l k8s-app=kube-proxy --tail=200

# 4. Check the iptables rules on a node
kubectl exec -it "$KUBE_PROXY_POD" -- iptables -t nat -L KUBE-SERVICES | head -20

# 5. Check the IPVS rules on a node
kubectl exec -it "$KUBE_PROXY_POD" -- ipvsadm -L -n | head -20

The iptables rules

The iptables rules are organized in chains:

Chain KUBE-SERVICES (2 references)
target     prot opt source               destination
KUBE-SVC-XXX  tcp  --  0.0.0.0/0            10.96.45.123        /* default/billing cluster IP */ tcp dpt:80

The chain KUBE-SERVICES matches the ClusterIP. The chain KUBE-SVC-XXX is the per-Service rule. The chain KUBE-SEP-XXX is the per-Pod rule.

A missing KUBE-SVC-XXX chain is a kube-proxy failure to reconcile.

The IPVS rules

The IPVS rules are organized as virtual servers:

IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.96.45.123:80 wlc
  -> 10.244.5.23:8080             Masq    1      0          0
  -> 10.244.5.24:8080             Masq    1      0          0

A missing virtual server is a kube-proxy failure to reconcile.

The kube-proxy reconciliation

The kube-proxy watches the API server for Service and EndpointSlice changes. When a change is detected, the kube-proxy reconciles the node’s rules.

sequenceDiagram
    participant API
    participant KP as kube-proxy
    participant Node
    API->>KP: Service added
    KP->>Node: Add iptables/IPVS rule
    API->>KP: EndpointSlice updated
    KP->>Node: Update iptables/IPVS rule
    API->>KP: Service deleted
    KP->>Node: Remove iptables/IPVS rule

A failed reconciliation is the kube-proxy’s failure mode.

The remediation

The remediation depends on the cause:

# Option 1: Restart the kube-proxy
kubectl rollout restart daemonset/kube-proxy -n kube-system

# Option 2: Reload the kube-proxy's ConfigMap
kubectl edit configmap kube-proxy -n kube-system

# Option 3: Switch the mode
kubectl edit configmap kube-proxy -n kube-system
# Change mode: iptables to mode: ipvs (or vice versa)
# Then restart the kube-proxy

The remediation is the routing recovery.

Production discipline

The kube-proxy is the cluster’s hypothesis. The discipline is to walk the canonical flow, identify the routing layer failure, apply the remediation. The routing engine is the cluster’s network; the remediation is the routing recovery.

• Check the kube-proxy mode. The mode is in the ConfigMap.
• Check the iptables/IPVS rules. The rules are the node’s routing.
• Restart the kube-proxy if needed. The restart is the remediation.

Quiz