ClusterIP unreachable — the kube-proxy and routing failure

A ClusterIP unreachable is a Service whose endpoints are populated but the Service does not route traffic. The kube-proxy, iptables, and IPVS rules are the diagnostic. The remediation is to fix the routing. The discipline is to walk the canonical flow and identify the failing layer.

The kube-proxy

The kube-proxy is the cluster’s Service routing engine. The kube-proxy watches the API server for Service and EndpointSlice changes, and programs the node’s iptables (or IPVS) rules to route the ClusterIP to the Pod IPs.

flowchart LR
    A[API server] --> B[kube-proxy]
    B --> C[iptables rules]
    C --> D[Node network]
    D --> E[Pod IP]

A ClusterIP unreachable is one where the kube-proxy’s rules are missing or wrong. The diagnostic is the kube-proxy’s state and the node’s iptables rules.

The diagnostic

The canonical diagnostic:

# Substitute your own values before running:
NODE=worker-01
CLUSTER_IP=10.96.45.123
SOURCE_POD=web-5f9c7d8b6c-2xk9p
SVC=billing
NS=prod

# 1. Check the kube-proxy Pods
kubectl get pods -n kube-system -l k8s-app=kube-proxy -o wide

# 2. Check the kube-proxy logs
kubectl logs -n kube-system -l k8s-app=kube-proxy --tail=200

# 3. Check the iptables rules on a node
kubectl exec -it "$NODE" -- iptables -t nat -L KUBE-SERVICES | grep "$CLUSTER_IP"

# 4. Check the IPVS rules on a node (if using IPVS)
kubectl exec -it "$NODE" -- ipvsadm -L -n | grep "$CLUSTER_IP"

# 5. Test the Service from inside a Pod
kubectl exec -it "$SOURCE_POD" -- curl -v "http://$SVC.$NS.svc.cluster.local"

The diagnostic is the kube-proxy’s state and the node’s rules.

Common causes

The most common causes of a ClusterIP unreachable:

1. kube-proxy is failing. The kube-proxy Pod is in CrashLoopBackOff or has crashed.
2. iptables rules are incorrect. The iptables rules on one or more nodes are missing or wrong.
3. IPVS rules are incorrect. The IPVS rules on one or more nodes are missing or wrong.
4. Network partition. The node cannot reach the API server.

flowchart TD
    A[ClusterIP unreachable] --> B{kube-proxy failing?}
    B -->|Yes| C[Restart kube-proxy]
    B -->|No| D{iptables rules wrong?}
    D -->|Yes| E[Fix iptables rules]
    D -->|No| F{IPVS rules wrong?}
    F -->|Yes| G[Fix IPVS rules]
    F -->|No| H[Network partition]
    H --> I[Fix the network]

The remediation

The remediation depends on the cause:

# Substitute your own value before running:
NODE=worker-01

# Option 1: Restart the kube-proxy
kubectl rollout restart daemonset/kube-proxy -n kube-system

# Option 2: Reload the iptables rules
kubectl exec -it "$NODE" -- iptables -t nat -F KUBE-SERVICES

# Option 3: Switch from iptables to IPVS (or vice versa)
kubectl edit configmap kube-proxy -n kube-system

The remediation is the routing recovery.

The IPVS mode

When the kube-proxy is in IPVS mode, the diagnostic is the IPVS rules:

# Substitute your own value before running:
NODE=worker-01

kubectl exec -it "$NODE" -- ipvsadm -L -n

The output shows the IPVS virtual servers and the real servers:

IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.96.45.123:80 wlc
  -> 10.244.5.23:8080             Masq    1      0          0
  -> 10.244.5.24:8080             Masq    1      0          0

The IPVS rules are the cluster’s routing. A missing rule is a routing failure.

Production discipline

A ClusterIP unreachable is the cluster’s hypothesis. The discipline is to walk the canonical flow, identify the failing layer, apply the remediation. The flow is the diagnostic; the routing is the recovery.

• Check the kube-proxy. The kube-proxy is the cluster’s routing engine.
• Check the iptables/IPVS rules. The rules are the node’s routing.
• Restart the kube-proxy if needed. The restart is the remediation.

Quiz