Pod CIDR allocation — how the cluster splits the IP space across nodes

The cluster CIDR is the IP space the cluster uses for Pods. The CNI plugin splits the cluster CIDR into per-node CIDRs; the kubelet allocates IPs from the per-node CIDR. The cluster CIDR must be sized for the maximum Pod count. This lesson walks the allocation, the failure modes of exhaustion, and the operational discipline of sizing.

The cluster CIDR

The cluster CIDR is the IP space the cluster uses for Pods. The kubeadm default is 10.244.0.0/16. The cluster operator can choose a different CIDR at init time:

kubeadm init --pod-network-cidr=10.244.0.0/16

The CIDR is stored in the cluster’s configuration (kubectl-cluster-info or the kubelet’s --cluster-cidr flag). The CNI plugin reads the CIDR to allocate per-node CIDRs.

flowchart LR
    A[Cluster CIDR 10.244.0.0/16] --> B[Per-node CIDR 10.244.1.0/24]
    A --> C[Per-node CIDR 10.244.2.0/24]
    A --> D[Per-node CIDR 10.244.3.0/24]
    B --> E[Pod 1: 10.244.1.5]
    B --> F[Pod 2: 10.244.1.6]
    C --> G[Pod 3: 10.244.2.5]
    C --> H[Pod 4: 10.244.2.6]

The per-node CIDR is the maximum number of Pods a node can host. A /24 gives 254 addresses per node. A busy production node routinely needs /23 or /22.

The per-node CIDR

The per-node CIDR is allocated by the controller-manager (in the routed mode) or by the CNI plugin (in the host-local mode). The kubelet reports its per-node CIDR via the Node object:

# Substitute your own value before running:
NODE=worker-01

kubectl get node "$NODE" -o jsonpath='{.spec.podCIDR}'

10.244.1.0/24

The kubelet’s --node-cidr-mask flag controls the per-node CIDR size. The default is /24 for IPv4. The cluster operator can change it to /23 or /22 for larger nodes.

The controller-manager’s role

The kube-controller-manager allocates the per-node CIDRs in the routed mode. The --allocate-node-cidrs and --cluster-cidr flags control the allocation:

kube-controller-manager \
  --allocate-node-cidrs=true \
  --cluster-cidr=10.244.0.0/16 \
  --node-cidr-mask-size=24

The controller-manager watches the Node objects and assigns a CIDR to each new node. The allocation is persistent: the controller-manager remembers the assignment and does not re-allocate on restart.

The cluster CIDR allocation is recorded in the Node’s spec.podCIDR field. The cluster operator can audit the allocation:

kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.podCIDR}{"\n"}{end}'

node-1   10.244.1.0/24
node-2   10.244.2.0/24
node-3   10.244.3.0/24

The CNI plugin’s IPAM

The CNI plugin uses the per-node CIDR to allocate Pod IPs. The two patterns are:

• Host-local IPAM: the CNI plugin uses the per-node CIDR directly. The IPAM has no central state.
• Calico IPAM: the CNI plugin uses the cluster CIDR and allocates blocks to nodes. The block size is configurable.

In host-local IPAM, the per-node CIDR is the source of truth. The cluster CIDR is the cluster’s view; the IPAM is the node’s view.

In Calico IPAM, the CNI plugin allocates blocks from the cluster CIDR. The block size is set by the IPPool resource. The per-node CIDR is the block the CNI allocated to the node.

The kubelet’s view

The kubelet has its own view of the per-node CIDR:

# The kubelet's config
cat /var/lib/kubelet/config.yaml

node-ip: 10.0.0.1
node-cidr-mask: 24

The kubelet uses the per-node CIDR to validate the Pod’s IP. A Pod that is assigned an IP outside the per-node CIDR is rejected by the kubelet.

The CNI plugin is the source of truth for the IP. The kubelet’s view is a cache that the CNI plugin updates.

The failure modes

The CIDR allocation’s failure modes:

The operational discipline

The CIDR allocation’s operational discipline:

• Size the cluster CIDR for the maximum node count. The formula is (maximum node count) << (per-node mask size).
• Size the per-node CIDR for the maximum Pod count. The formula is (maximum Pods per node) + headroom.
• Audit the CIDR allocation at every node addition. The controller-manager’s allocation is the source of truth.
• Monitor the IPAM utilisation. A growing utilisation is the leading indicator of exhaustion.
• Plan the cluster’s IPv6 migration. The IPv4-only cluster is a finite resource.
• Document the CIDR choice. The CIDR is the cluster’s IP space; the documentation is the reference.

Quiz

Production discipline

• The cluster CIDR is the cluster’s IP space. The CNI plugin divides it into per-node CIDRs.
• The per-node CIDR is the maximum Pod count per node. The kubelet —node-cidr-mask flag controls the size.
• The cluster CIDR must be sized for the maximum node count. The formula is (maximum node count) << (per-node mask size).
• The per-node CIDR must be sized for the maximum Pod count. The formula is `(maximum Pods per node)
  • headroom`.
• Audit the CIDR allocation at every node addition. The controller-manager’s allocation is the source of truth.
• Monitor the IPAM utilisation. A growing utilisation is the leading indicator of exhaustion.
• Plan the cluster’s IPv6 migration. The IPv4-only cluster is a finite resource.
• Document the CIDR choice. The CIDR is the cluster’s IP space; the documentation is the reference.