CNI selection — Cilium, Calico, Flannel, and choosing

The CNI plugin is the cluster’s network fabric. It provides Pod IPs, routing, and (in most cases) NetworkPolicy enforcement. Production clusters choose between Cilium, Calico, and Flannel, with Cilium leading for modern workloads. This lesson walks the comparison and the production discipline of choice.

The CNI role

flowchart LR
    K[kubelet] -->|CNI| P[Pod network]
    P -->|routing| K2[Other pods]
    P -->|Service IP| SVC[Service]
    P -->|external| EXT[Outside cluster]

The CNI:

• Assigns Pod IPs at Pod creation.
• Sets up the Pod’s network interface.
• Configures routing for Pod-to-Pod traffic.
• Implements NetworkPolicy (optional).
• Provides observability (advanced CNIs).

The CNI choice

Three dominant CNIs in production:

Cilium

Cilium’s architecture:

flowchart LR
    P[Pod] -->|CNI| A[cilium-agent]
    A -->|eBPF| K[Kernel]
    K -->|map| M[eBPF maps]
    A -->|cilium-operator| ETCD
    A -->|Hubble| OBS[Observability]

• cilium-agent. Runs on every node; configures eBPF programs.
• cilium-operator. Cluster-wide policy management.
• Hubble. Observability layer (flow logs, metrics).
• eBPF kernel programs. The data plane: fast packet processing in kernel space.

# Install via Helm
helm repo add cilium https://helm.cilium.io
helm install cilium cilium/cilium \
  --version 1.16.x \
  --namespace kube-system \
  --set kubeProxyReplacement=true \
  --set hubble.enabled=true \
  --set hubble.metrics.enabled="{dns,drop,tcp,flow,port-distribution,icmp}"

Calico

Calico’s architecture:

flowchart LR
    P[Pod] -->|CNI| CD[calico-cni]
    CD -->|iptables or eBPF| K[Kernel]
    K -->|routing| BGP[BGP or VXLAN]
    BGP -->|peers| RP[Calico route reflector]
    CD -->|Felix| K2[Felix policy daemon]
    K2 -->|NetworkPolicy| K

• calico-cni. The CNI plugin at Pod creation.
• Felix. The per-node daemon that programs iptables (or eBPF) routes and policy.
• Typha. Cluster-wide state caching.

# Install via operator
kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml

$ kubectl get pods -n kube-system -l k8s-app=cilium

...

Flannel

Flannel’s architecture:

flowchart LR
    P1[Pod] --> VETH[CN plugin]
    P2[Pod] --> VETH
    VETH -->|VXLAN| FNET[flannel.1]
    FNET -->|UDP 8472| NODE[Node]

• flanneld. Per-node agent; assigns subnet leases; configures VXLAN.
• VXLAN. Encapsulation for cross-node Pod traffic.

Flannel is minimal. It does not implement NetworkPolicy (use Calico or Cilium for that).

# Install flannel
kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml

Flannel’s role: a starting CNI for small clusters. For NetworkPolicy, switch to Calico.

The CNI alternatives

Multus is useful when a Pod needs multiple network interfaces (e.g., primary + secondary data plane). Antrea is eBPF-based; production-ready.

The choice

The choice depends on:

For most modern production clusters, Cilium is the right answer. Calico is the right answer when BGP is needed (e.g., for on-prem routing integration).

The MTU consideration

Each CNI’s overlay (VXLAN, GENEVE) consumes part of the MTU:

If your network has a lower MTU (e.g., 1400), the CNI must be configured for it. Mismatch causes fragmentation-related failures.

# Configure Calico for lower MTU
calicoctl set felixconfig ... IpMtu=1400

The NetworkPolicy verification

After CNI install, test NetworkPolicy:

# Apply a default-deny policy
kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: default
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress
EOF

# Verify isolation: pods in different namespaces should still talk; same-namespace pods should not
kubectl exec -n default pod-a -- wget -O- http://pod-b.default
# Should fail with timeout (denied)

The test confirms NetworkPolicy is enforcing.

The CNI verification

# Pod network
kubectl exec pod-a -- ip addr show
# Expected: eth0 with the Pod IP, lo with 127.0.0.1

# Ping across nodes
kubectl exec pod-a -- ping -c 1 pod-b
# Expected: receives response

# DNS resolution
kubectl exec pod-a -- nslookup kubernetes.default
# Expected: returns 10.96.0.1 (the Service IP)

$ kubectl get pods -n kube-system -l k8s-app=calico-node

...

The CNI upgrades

For upgrades:

# Cilium
helm upgrade cilium cilium/cilium \
  --version 1.16.x \
  --namespace kube-system

# Calico
kubectl apply -f calico-updated.yaml

Cilium upgrades can include cilium-agent version bumps that require node restarts (rolling).

The discipline

• Choose the CNI deliberately. Document the choice and its rationale.
• Install CNI immediately after kubeadm init. No Pod network without it.
• Test NetworkPolicy. A cluster without NetworkPolicy testing is one that doesn’t enforce policies.
• Monitor CNI health. Cilium / Calico provide health metrics; alert on regression.
• Re-evaluate for new clusters. Cilium is the modern default.

Quiz

Production discipline

• Choose the CNI deliberately. Document it.
• Install immediately after kubeadm init. No CNI = no Pod network.
• Test NetworkPolicy. Default-deny verification matters.
• Monitor CNI health. CNI is part of the cluster’s heartbeat.
• Re-evaluate. Cilium is the modern default; old clusters may have Flannel.

The CNI is the cluster’s network fabric. Operating it well is making a deliberate choice and validating it.