CNI lifecycle — ADD, DEL, CHECK, and the kubelet as the runtime

The CNI lifecycle has three operations: ADD, DEL, and CHECK. The kubelet invokes each at a precise moment in the Pod’s life. The plugin is responsible for the mutation; the runtime is responsible for the call. This lesson walks the lifecycle, the on-disk state that survives a kubelet restart, and the failure modes that production operators must recognise.

The ADD operation

ADD is invoked when the kubelet creates the Pod’s sandbox. The sandbox is a paused container whose only job is to hold the network namespace shared by every container in the Pod. The kubelet calls libcni with the following parameters:

CNI_COMMAND=ADD
CNI_CONTAINERID=<sandbox-id>
CNI_NETNS=/var/run/netns/<sandbox-id>
CNI_IFNAME=eth0
CNI_PATH=/opt/cni/bin

The plugin receives the same parameters as JSON over stdin. The plugin must:

1. Create the veth pair, with one end in the sandbox and the other on the host.
2. Assign the IP from the IPAM.
3. Configure the routes inside the namespace.
4. Return the result JSON on stdout.

libcni caches the result at /var/lib/cni/results/<network-name>_<container-id>. The cache allows the kubelet to call DEL with only the container ID and the network name and let the plugin read the cached result to know what to remove.

sequenceDiagram
    autonumber
    participant K as kubelet
    participant L as libcni
    participant P as CNI plugin
    participant R as Result cache
    K->>L: ADD (sandbox-id, netns)
    L->>P: stdin JSON
    P->>P: create veth, assign IP, configure routes
    P-->>L: stdout JSON
    L->>R: write result
    L-->>K: result
    K->>K: write podIP to API server

The DEL operation

DEL is invoked when the sandbox is destroyed. The kubelet calls libcni with:

CNI_COMMAND=DEL
CNI_CONTAINERID=<sandbox-id>
CNI_NETNS=/var/run/netns/<sandbox-id>
CNI_IFNAME=eth0

The plugin reads the cached result, removes the veth pair, returns the IP to the IPAM pool, and removes the routes. The plugin must be idempotent: a redundant DEL on a container that has already been removed must return success.

A DEL failure is reported as a kubelet warning but does not block the Pod’s removal. The IP is leaked from the IPAM pool; the operator must observe the IPAM utilisation and the result cache for orphans.

The CHECK operation

CHECK is invoked when the kubelet queries the container’s network status. The plugin must verify that the configuration is still correct:

• The veth pair still exists.
• The IP is still assigned.
• The routes are still present.

If CHECK returns an error, the kubelet logs the failure but takes no automatic action. The plugin can return a hint in the result, such as {"action": "reconcile"}, to indicate that the kubelet should call ADD again. That pattern is rare in production; the typical response to a CHECK failure is an operator inspection.

The result cache

The result cache is on disk at:

ls /var/lib/cni/results/

k8s-pod-network_a1b2c3d4e5f6
k8s-pod-network_b2c3d4e5f6a7
k8s-pod-network_c3d4e5f6a7b8

Each file is the JSON result for one container. The kubelet reads the cache on every DEL to learn the IP and interface name to remove. The cache is also the fallback when the kubelet restarts: the runtime can ask the plugin to verify the configuration against the cache.

The lifecycle on a running node

The lifecycle leaves traces you can audit:

# Recent ADD calls
journalctl -u kubelet --since "5 minutes ago" | grep -i "cni"

kubelet[1234]: (pod) Successfully added network ... plugin="calico"
kubelet[1234]: (pod) Deleted network ... plugin="calico"

The kubelet log shows every ADD and DEL. Plugin errors appear as AddPod: failed to ... lines.

The failure modes

The lifecycle has the following failure modes:

Each failure mode is per-node. A bug in the plugin on node-1 affects Pods scheduled to node-1 only.

The operational discipline

The lifecycle’s operational discipline:

• Audit the result cache. The cache is the source of truth for DEL. A corrupted cache blocks every future DEL.
• Audit the binlog of ADD/DEL. The kubelet log shows the lifecycle.
• Monitor IPAM utilisation. A leak shows up as growing utilisation without a corresponding Pod count.
• Monitor orphan veths. A leak shows up as a growing count of cali* interfaces on the host.
• Test the lifecycle in staging. ADD, scale, delete, and restart the kubelet; verify the IPAM is consistent.

Quiz

Production discipline

• The CNI lifecycle is ADD, DEL, CHECK. Each is invoked by the runtime; each is a synchronous blocking call.
• The result cache is the source of truth for DEL. The cache must be on durable storage.
• DEL must be idempotent. A redundant DEL on a already-removed container must return success.
• CHECK is for verification, not repair. The plugin can report a drift; the runtime does not take automatic action.
• Audit the lifecycle on every node. The kubelet log is the source of truth.
• Monitor IPAM utilisation and orphan veths. Both are the leading indicators of lifecycle failure.
• Test the lifecycle in staging. Restart the kubelet; verify the IPAM is consistent.
• Document the lifecycle. The CNI is the cluster’s networking implementation; the documentation is the cluster’s networking reference.