high riskcluster affecting~30 min
Runbook: Troubleshoot CNI
1 · Prerequisites
Confirm every item is in place before any state change.
2 · Pre-checks
Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.
- · Confirm the CNI DaemonSet is healthy:
kubectl -n kube-system get ds -o wide - · Confirm every node has a CNI Pod Running:
kubectl -n kube-system get pods -o wide -l k8s-app=<cni> - · Capture the kubelet CNI configuration:
ls /etc/cni/net.d/andcat /etc/cni/net.d/*.conflist - · Confirm the CNI version is the expected one (no recent upgrade in progress)
- · Capture the test flow: client Pod in
<ns1>to target Pod in<ns2>or to a Service ClusterIP - · Capture the failure mode: timeout, connection refused, or
No route to host
3 · Procedure
Execute each step in order. Verify the expected output of a step before moving to the next.
- 1Confirm the CNI agent is Running on the source and destination nodes:
kubectl -n kube-system get pods -o wide -l k8s-app=<cni> - 2Confirm the Pod has a Pod IP and the interface exists in the Pod network namespace:
kubectl exec <pod> -- ip addr show - 3Confirm the node has a route to the Pod CIDR:
ip routeon the source and destination nodes - 4Confirm the bridge or veth pair exists on the node:
ip linkandbridge link - 5Confirm the CNI agent logs for the affected node:
kubectl -n kube-system logs <cni-pod> --tail=200 - 6Confirm the kube-proxy dataplane (or CNI replacement, e.g. Cilium) has rules for the target Service:
iptables-save | grep <cluster-ip>or Cilium equivalent - 7For connection refused: target Pod not listening on the expected port (workload problem, not CNI)
- 8For timeout: dataplane or routing problem (CNI problem)
- 9For
No route to host: route table problem on the node (CNI configuration) - 10Apply the smallest fix: restart the CNI agent, restore the route, fix the bridge, restart the Pod to re-trigger CNI ADD
4 · Verification
Confirm the procedure actually fixed the problem.
- ✓Every node has a CNI Pod Running and Ready
- ✓
ip routeon the source node shows the Pod CIDR with the expected next-hop (CNI bridge or tunnel) - ✓
ip routeon the destination node shows the Pod CIDR with the expected next-hop - ✓
kubectl exec <pod> -- ip addr showshows the expected Pod IP and veth interface - ✓
kubectl exec <client> -- nc -vz <pod-ip> <port>returnssucceeded - ✓
kubectl exec <client> -- curl http://<service>:<port>/returns the expected response - ✓No
Warningevents in the last 5 minutes - ✓CNI agent logs do not show recent errors
5 · Rollback
If verification fails, undo the procedure in reverse order.
- ↶If restarting the CNI agent broke new Pod scheduling, restart kubelet on the affected node to re-trigger CNI ADD for stuck Pods
- ↶If a route table change broke traffic, restore the route from the CNI configuration
- ↶If the CNI was reinstalled, validate that kube-proxy (or CNI replacement) dataplane is consistent across every node
- ↶Capture the failing CNI agent logs to off-cluster storage before any restart
- ↶For Cilium: rolling restart the operator (
kubectl -n kube-system rollout restart deploy/cilium-operator) preserves state better than killing the DaemonSet
6 · Escalation
When the runbook isn't enough, contact:
- · CNI agent CrashLoopBackOff: CNI configuration is invalid or incompatible with the node OS/kernel; escalate to platform
- · CNI dataplane inconsistent across nodes: Cilium/ Calico cluster is partitioned; restart operator and verify the cluster connectivity
- · Pod cannot get an IP from the IPAM: cluster CIDR exhausted; escalate to capacity planning
- · No route to host after a recent cluster network change: the CNI was not re-installed after the change; escalate to platform
- · All Pod traffic fails: the CNI DaemonSet is not deploying to new nodes; escalate to platform — every new node needs the CNI before workloads can run
A CNI failure looks like a Pod-to-Pod connectivity failure: the Pod is Running, the kubelet is happy, but traffic times out. The cause is in the CNI dataplane, not in the workload.
1. Confirm the CNI agent is present on every node
Read-only / SafeConfirm the CNI agent is present on every node
kubectl get ds -A -o wide | grep -E 'cilium|calico|flannel|weave'
kubectl -n kube-system get pods -o wide -l k8s-app=<cni> | head -50
# Identify nodes without a CNI agent
kubectl get nodes -o json | jq -r '.items[] | .metadata.name' > /tmp/nodes.txt
kubectl -n kube-system get pods -o wide -l k8s-app=<cni> -o json | jq -r '.items[].spec.nodeName' | sort -u > /tmp/cni-nodes.txt
comm -23 /tmp/nodes.txt /tmp/cni-nodes.txt
A node without a CNI agent cannot host Pods. If only one or two nodes are missing the CNI agent, the workload scheduling is preferentially on those nodes — which is why the failure looks consistent.
2. Confirm the Pod has a network namespace and IP
Read-only / SafeConfirm the Pod has a network namespace and IP
kubectl exec <pod> -n <ns> -- ip route
kubectl exec <pod> -n <ns> -- cat /etc/resolv.conf
If the Pod has no IP, the kubelet’s CNI ADD call failed. Read the kubelet journal on the node.
Read-only / SafeConfirm the Pod has a network namespace and IP
NODE=$(kubectl get pod <name> -n <ns> -o jsonpath='{.spec.nodeName}')
ssh "$NODE" -- sudo journalctl -u kubelet --since "10 min ago" | grep -E 'cni|CNI|network' | tail
3. Walk the path
flowchart LR
A["Client Pod"] -->|veth| B["Node bridge/veth"]
B -->|route / overlay| C["Destination node"]
C -->|bridge / veth| D["Target Pod"]
D -->|port| E["Application"]| Stage | Symptom | Where to look |
|---|---|---|
| Pod → bridge | No veth in ip link output | CNI ADD failed; kubelet journal |
| Bridge → route | No route to host | ip route on the node |
| Route → overlay/underlay | Encapsulation errors | CNI agent logs; node-to-node ping |
| Destination node bridge | ARP failure | bridge/ARP table on the destination node |
| Target Pod → app | Connection refused | Application not listening |
4. Inspect the dataplane
Read-only / SafeInspect the dataplane
kubectl -n kube-system exec ds/cilium -- cilium status
kubectl -n kube-system exec ds/cilium -- cilium bpf nat list | grep <svc-cluster-ip>
kubectl -n kube-system exec ds/cilium -- cilium endpoint list | grep <pod-ip>
# Calico
kubectl -n kube-system exec ds/calico-node -- calicoctl node status
kubectl -n kube-system exec ds/calico-node -- calicoctl get workloads
# Generic
ip route show
bridge link show
arp -n
If the dataplane is missing rules for the Service, restart the CNI agent on the affected node.
5. Restart the CNI agent
Read-only / SafeRestart the CNI agent
kubectl -n kube-system rollout restart ds/cilium
kubectl -n kube-system rollout status ds/cilium --timeout=5m
# Calico
kubectl -n kube-system rollout restart ds/calico-node
kubectl -n kube-system rollout status ds/calico-node --timeout=5m
# Validate after restart
kubectl -n kube-system exec ds/cilium -- cilium status
kubectl -n kube-system exec ds/cilium -- cilium endpoint list | wc -l
kubectl -n kube-system exec ds/cilium -- cilium bpf nat list | grep <svc-cluster-ip>
6. Re-create the failing Pod
If the dataplane is healthy but the Pod is in a broken state (stale routes, missing veth), deleting the Pod forces kubelet to re-call CNI ADD.
Read-only / SafeRe-create the failing Pod
# For Deployment-managed Pods, the controller recreates; for one-offs, re-apply
kubectl wait --for=condition=Ready pod -l app=<name> -n <ns> --timeout=120s
kubectl get pod -l app=<name> -n <ns> -o jsonpath='{.items[*].status.podIP}'
7. Verify end-to-end
Read-only / SafeVerify end-to-end
CLIENT=<client-pod>
TARGET=$(kubectl get pod -l <target-label> -n <ns2> -o jsonpath='{.items[0].status.podIP}')
kubectl exec "$CLIENT" -n <ns1> -- nc -vz "$TARGET" <port> --wait=3
kubectl exec "$CLIENT" -n <ns1> -- curl -fsS http://<service>.<ns>.svc:<port>/ --max-time 5
Common pitfalls
| Symptom | Cause | Action |
|---|---|---|
| Connection times out, no CNI rule | CNI agent not running on a node | Restart CNI DaemonSet; check node |
No route to host from a Pod | Node route table missing Pod CIDR | Restart kubelet to re-add CNI routes |
| Dataplane stale after upgrade | Upgrade did not include dataplane restart | Restart operator and DaemonSet |
| Pod has no IP | IPAM exhausted or CNI ADD failed | Check kubelet journal; check IPAM |
| Intermittent failures, overlay encapsulation | MTU mismatch (see kubernetes-cxlvii-03-mtu-issues) | Reduce the overlay MTU |
A CNI failure is rarely a workload problem. The runbook verifies the dataplane, the routes, and the CNI agent before touching the workload.