Networking restoration — CNI, CoreDNS, and cluster DNS verification

Phase 3 of complete cluster loss recovery is restoring cluster networking — the CNI, CoreDNS, and the data plane that makes pods addressable. This lesson walks the CNI install, the pod CIDR alignment, CoreDNS verification, kube-proxy verification, and the operational discipline.

The CNI install

flowchart LR
    A["kubeadm init: pod CIDR"] --> B[Apply CNI manifest]
    B --> C["CNI agent: DaemonSet on every node"]
    C --> D[Pod networking up]
    D --> E[CoreDNS scheduled]
    E --> F[Cluster DNS functional]

The CNI is installed as a DaemonSet that runs on every node, plus a conflist on every node’s /etc/cni/net.d/. The DaemonSet manages the host networking; the conflist tells kubelet how to invoke the CNI plugin when adding a Pod to a network namespace.

Common CNIs:

# Calico
kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml

# Cilium
kubectl apply -f https://github.com/cilium/cilium-cli/raw/master/k8s-install-quickstart.yaml

# Flannel
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

Verifying the CNI

kubectl get pods -n kube-system -l k8s-app=calico-node

NAME                READY   STATUS    RESTARTS   AGE
calico-node-xxxxx   1/1     Running   0          2m
calico-node-yyyyy   1/1     Running   0          2m
calico-node-zzzzz   1/1     Running   0          2m

The CNI Pods must be Running on every node. If a Pod is Pending, the kubelet on that node is not Ready or the CNI’s DaemonSet is misconfigured.

# A calico-node Pod name from the listing above:
CALICO_POD=calico-node-8k4wv

kubectl exec -n kube-system "$CALICO_POD" -- calicoctl node status

IPv4 BGP status: Up
IPv4 IPPools:
  CIDR: 10.244.0.0/16
  Mode: VXLAN

The CNI’s view of the cluster should show all nodes with BGP or VXLAN up, depending on the configuration.

CoreDNS verification

CoreDNS is the cluster DNS service. Without it, Pods cannot resolve service names.

kubectl get pods -n kube-system -l k8s-app=kube-dns

NAME                       READY   STATUS    RESTARTS   AGE
coredns-xxxxx-yyyyy        1/1     Running   0          2m
coredns-zzzzz-aaaaa        1/1     Running   0          2m

Both CoreDNS Pods (or however many replicas) should be Running. CoreDNS is a Deployment with 2 replicas by default.

# A CoreDNS Pod name from the listing above:
COREDNS_POD=coredns-668d6bf9bc-4nzt7

kubectl exec -n kube-system "$COREDNS_POD" -- \
  nslookup kubernetes.default.svc.cluster.local

Server:     10.96.0.10
Address:    10.96.0.10#53

Name:   kubernetes.default.svc.cluster.local
Address: 10.96.0.1

CoreDNS must resolve kubernetes.default.svc.cluster.local to the kubernetes Service’s ClusterIP. If it does not, the cluster DNS is not functional.

kube-proxy verification

kube-proxy programs iptables (or IPVS) rules on every node to route Service traffic to Pod backends. Without it, Services do not work.

kubectl get pods -n kube-system -l k8s-app=kube-proxy

NAME                READY   STATUS    RESTARTS   AGE
kube-proxy-xxxxx    1/1     Running   0          2m
kube-proxy-yyyyy    1/1     Running   0          2m
kube-proxy-zzzzz    1/1     Running   0          2m

kube-proxy is a DaemonSet; one Pod per node.

# On a node, check iptables rules for kube-proxy
iptables-save | grep -c KUBE

42

The iptables rules count indicates kube-proxy has programmed the data plane. A zero count means kube-proxy is not programming rules.

The end-to-end network test

Before applying application workloads, run a network test:

kubectl run nettest --rm -it --image=nicolaka/netshoot -- \
  bash -c "nslookup kubernetes.default && ping -c 1 10.96.0.1"

The test verifies:

• The Pod can be scheduled on a worker node.
• The Pod’s network namespace is functional.
• DNS resolution works (nslookup).
• Service ClusterIP is reachable (ping).

If any of these fail, the network is not ready; workloads will not function.

The operational failure modes

Phase 3 fails for predictable reasons:

• CNI pod CIDR mismatch. kubeadm init’s --pod-network-cidr does not match the CNI’s IPAM configuration. Pods stay Pending.
• CNI not installed on every node. The DaemonSet is restricted to a node label that workers do not have. Workers’ Pods are Pending.
• CoreDNS pending. CoreDNS Pods are Pending because the CNI is not ready. Service discovery is broken.
• kube-proxy not programming. kube-proxy Pods are Running but iptables rules are empty. The CNI plugin is not the cause — kube-proxy cannot reach the API server.
• MTU mismatch. The CNI’s overlay MTU does not match the underlying network’s MTU. Packets are dropped silently.

Quiz

Production discipline

Phase 3 (networking restoration) in production rests on five non-negotiable elements:

• Align pod CIDR with kubeadm init. The —pod-network-cidr flag and the CNI’s IPAM must match. Save the kubeadm init flags in the runbook.
• Verify the CNI on every node. The DaemonSet must be Running on every worker. A node without the CNI cannot run Pods.
• Verify CoreDNS before workloads. DNS resolution is a prerequisite for service discovery. Verify with nslookup.
• Verify kube-proxy rules. The iptables/IPVS rules count must be >0; otherwise Services do not route.
• Run an end-to-end test. A debug Pod that nslookups and pings confirms the network is functional. Apply workloads only after the test passes.

Networking is the prerequisite for every workload. A network that has not been verified is a network whose recovery is unknown. The discipline is to verify before applying workloads.