CNI broken on one node

Reported symptoms

Nothing about this arrived as an incident. It arrived as four separate tickets over two days, each closed as a flake.

• Tuesday 10:40, the payments team: one Pod of a twelve-replica Deployment sat in ContainerCreating for half an hour. They deleted it. The replacement started in four seconds. Closed.
• Tuesday 16:15, the search team: same shape, different Deployment. Deleted the Pod, it came back immediately. Closed as “transient registry problem”, although nobody checked the registry.
• Wednesday 09:05, a batch job that had run every morning for a year did not start. Deleted, re-ran, fine.
• Wednesday 11:30, the platform team: a DaemonSet rollout stuck at 17 of 18. This one could not be fixed by deleting the Pod, because a DaemonSet keeps putting it back on the same node.

That last ticket is the one that could not be closed, and it is also the one that names the node. All four stuck Pods were on node-07.

What made this hard to see for two days:

• node-07 is Ready. Every condition on it is healthy, and no node alert has fired at any point in the two days.
• The capacity dashboard shows node-07 running 43 Pods against a maxPods of 110. It looks like the emptiest node in the cluster, which is exactly why the scheduler keeps choosing it.
• Every Pod already running on node-07 is perfectly healthy and has been serving traffic throughout.
• Nothing was changed on the node. No upgrade, no reboot, no CNI version change, no config management run.

The only thing anybody can remember happening to node-07 was two weeks ago, when it was used for a load test that created and destroyed several thousand short-lived Jobs.

Evidence provided

$ kubectl describe pod -n platform node-exporter-hd8wq | sed -n '/Events/,$p'

Warning  FailedCreatePodSandBox  2m (x14 over 41m)  kubelet
Failed to create pod sandbox: plugin type "..." failed (add):
failed to allocate for range 0: no IP addresses available in range set:
10.244.7.1-10.244.7.254

$ kubectl get node node-07 -o jsonpath='{range .status.conditions[*]}{.type}={.status} {end}'

MemoryPressure=False DiskPressure=False PIDPressure=False Ready=True

$ sudo ls /var/lib/cni/networks/k8s-pod-network | wc -l

254

$ sudo crictl pods -q | wc -l

44

$ sudo head -c 64 /var/lib/cni/networks/k8s-pod-network/10.244.7.113; echo; sudo crictl inspect 4d9f2ab7c1e05 2>&1 | head -1

$ for n in node-04 node-07 node-11; do echo $n $(ssh $n 'sudo ls /var/lib/cni/networks/k8s-pod-network | wc -l') $(ssh $n 'sudo crictl pods -q | wc -l'); done

node-04 39 37
node-07 254 44
node-11 52 50

$ sudo journalctl -u kubelet --since '15 days ago' --until '13 days ago' | grep -ci 'cni.*del'

Work the evidence before reading on

Five facts, and only one explanation fits all five.

1. The node is Ready and the plugin is answering. It returns a specific, well-formed error rather than failing to run. Is this a broken CNI, or a working one being asked for something it cannot give?
2. 254 allocations, 44 sandboxes. Which of those two numbers is the truth about the node, and what does the difference between them represent?
3. The count is 254 and the range is a /24. That is not a coincidence — what is the largest value that number could ever have been?
4. Deleting the Pod fixed it, four times. What does deleting a Pod actually change about where it runs, and what does that tell you about where the fault lives?
5. The node was not changed in two weeks, but something did happen to it two weeks ago. What is the connection between thousands of short-lived Pods and a number that only ever goes up?

Before continuing: the count has been climbing for two weeks and nothing anywhere reported it. Which two numbers, compared, would have made it obvious on day one?

Root cause

1. The node has run out of Pod IPs

The CNI on this cluster uses host-local IPAM. Each node gets a /24 — here 10.244.7.0/24 — and the plugin hands out addresses from it with no cross-node coordination. A /24 is 254 usable addresses, which is comfortably more than the maxPods limit of 110, and that headroom is why nobody has ever thought about it.

host-local keeps its state as files on the node. One file per allocated address, named for the address, containing the container ID that holds it:

sudo ls /var/lib/cni/networks/k8s-pod-network | head

The plugin returns an address by deleting that file. That happens during the CNI DEL that follows sandbox teardown. No DEL, no deletion, no returned address.

2. A failed DEL is nearly silent

This is the part that lets a leak run for two weeks.

When a DEL fails, the kubelet logs a warning and carries on. It does not block the Pod from being removed, it does not retry indefinitely, and it does not surface anything to the API. From the operator side the Pod deletion looks completely clean: the object disappears, the container is gone, kubectl get pods is tidy. The only trace is one line in one node journal, and the address is gone from the pool for good.

3. Why the DELs failed

libcni caches the result of every successful ADD under /var/lib/cni/results, and that cache is the plugin record of what a given container was given. DEL is called with little more than the container ID and the network name; the plugin reads the cached result to learn what to tear down. If the cache entry is gone, the plugin cannot complete the DEL.

During the load test the node runtime was restarted and that cache was cleared. Every sandbox that was live at that moment lost its record, so the DELs that followed had nothing to work from. The journal band two weeks ago is exactly that window, and the fact that there have been no DEL failures since is consistent: the damage was done in one window and has been permanent ever since.

4. Why it looked intermittent

It never was. Every Pod scheduled onto node-07 after the pool emptied failed, and every Pod scheduled anywhere else succeeded. Deleting a stuck Pod creates a replacement that the scheduler places fresh — usually somewhere else, because node-07 is one of eighteen — and it works. The ticket closes, the evidence goes with it, and the node stays broken.

The DaemonSet was the one workload that could not be fixed this way, because a DaemonSet is pinned to the node by definition. It is the only reason this was ever found.

Resolution

1. Cordon the node: kubectl cordon node-07. That stops any further Pod landing on a node that cannot give it an address, and it takes effect immediately. If the cluster can carry the loss of one node of capacity, holding here until a maintenance window is a legitimate end state rather than a failure to fix it.
2. Decide between the two repairs before starting either. Draining the node and clearing the store with nothing running is slower and disruptive to the 44 Pods on it, and it has no way to go wrong. Reconciling in place is fast and non-disruptive, and it can assign a live Pod address to a second Pod if the comparison is done carelessly.
3. For the in-place path, build the list of live sandbox IDs first with sudo crictl pods -q, then read the container ID recorded inside each allocation file, and remove only the files whose ID is not in that list. Work from a list you have written down, not from a one-liner improvised at the prompt.
4. For the drain path, kubectl drain node-07 --ignore-daemonsets --delete-emptydir-data, confirm with sudo crictl pods -q that nothing is left, then remove the address files. Respect PodDisruptionBudgets and expect the drain to be the slow part.
5. Leave the orphaned host-side veth interfaces alone for now. They are a real leak and worth cleaning, but they consume no addresses, they are not what is blocking Pod creation, and removing interfaces on a live node is a separate change with its own risk.
6. Do not restart the kubelet expecting it to help. It does not reconcile the IPAM store, and a kubelet restart is one of the events that can lose the result cache and make the underlying problem worse.
7. Uncordon and prove an ADD succeeds before declaring the node healthy: kubectl uncordon node-07, then schedule a throwaway Pod pinned to that node and watch it reach Running with an address in the node subnet.
8. Run the same allocation-versus-sandbox comparison on every other node. The load test may not have been confined to one, and a node sitting at 200 of 254 is the next outage rather than a healthy node.

Verification

1. The counts agree on node-07: the number of address files under the network directory equals the number of sandboxes sudo crictl pods -q reports, allowing for the plugin bookkeeping entries. Lower is not the test; equal is the test.
2. A new Pod pinned to node-07 reaches Running with an IP inside 10.244.7.0/24, and kubectl describe pod shows no FailedCreatePodSandBox event.
3. Deleting that test Pod returns its address. Re-count immediately: if the count does not drop by one, the leak is still active and the cleanup only bought time. This is the check that can still fail, and it is the one that distinguishes a repaired node from a tidied one.
4. The kubelet journal on node-07 shows CNI ADD succeeding for new Pods and no new DEL failures: sudo journalctl -u kubelet --since "30 minutes ago" | grep -i cni.
5. The stuck DaemonSet reports 18 of 18 desired and ready, which is the workload that could not route around the fault and is therefore the honest end-to-end check.
6. Every node in the cluster has been measured, not just node-07, and the allocation and sandbox counts track on all of them.
7. A re-count 24 hours later still tracks. A leak that has been cleaned up and a leak that has stopped look identical on the day, and only the second one is fixed.

Prevention

• Alert on the gap, not on the level. Utilisation crossing a threshold cannot distinguish a busy node from a leaking one. The difference between allocated addresses and running sandboxes can, and it opens on the day the leak starts rather than on the day the pool empties.
• Alert on time spent in ContainerCreating. This failure never produces a Pending Pod and never produces a node condition, so the two alerts most clusters already have are both silent for it.
• Read Ready for what it says. It means the kubelet and the network plugin are initialised. It does not mean a Pod can be created on the node, and those two came apart here for a fortnight.
• Reconcile the IPAM store after any runtime or kubelet restart on a node. That is precisely when the result cache DEL depends on is most likely to be lost, and it is cheap to check while the node is already in a maintenance state.
• Size the per-node CIDR against churn, not against maxPods. A /24 behind a 110-Pod limit looks generous. All that headroom decides is how many weeks the leak runs before it becomes an outage.
• Never close a ticket because deleting the Pod fixed it. A workload that recovers when it is moved is telling you the fault belongs to the place it was. Rescheduling is the single action most likely to resolve the symptom and erase the evidence, and it did that four times here.