critical riskcluster affecting~90 min

Runbook: Build a New Kubernetes Cluster with kubeadm

1 · Prerequisites

Confirm every item is in place before any state change.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Confirm hostnames, IPs and the load-balancer VIP are stable and DNS-resolvable both ways
· Confirm NTP is synchronised across every node: chronyc tracking reports Leap status: Normal
· Confirm kernel modules are loaded: lsmod | grep -E br_netfilter|overlay shows both
· Confirm sysctls are persisted in /etc/sysctl.d/99-kubernetes.conf: sysctl net.ipv4.ip_forward net.bridge.bridge-nf-call-iptables
· Confirm swap is disabled on every node: swapon --show returns empty
· Confirm the container runtime is installed and running: crictl info reports a non-empty status with RuntimeReady: true
· Confirm kubeadm, kubelet and kubectl are pinned to the target minor version (1.34.x)
· Confirm an etcd snapshot destination exists and is reachable from the first control-plane node

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1On the first control-plane node, run sudo kubeadm init --config kubeadm-config.yaml --upload-certs after capturing the rendered plan
2Record the join commands that kubeadm prints; they include the discovery token CA cert hash and, for control-plane joins, a certificate key
3Configure kubectl for the admin user: mkdir -p $HOME/.kube && sudo cp /etc/kubernetes/admin.conf $HOME/.kube/config && sudo chown $(id -u):$(id -g) $HOME/.kube/config
4Install the chosen CNI (Cilium, Calico, etc.) using its official Helm chart or operator; do not mix providers
5Wait for all node Ready and all system Pods Running: kubectl get nodes and kubectl -n kube-system get pods
6Remove the scheduling taint on the first control-plane if it is a single-node cluster: kubectl taint nodes --all node-role.kubernetes.io/control-plane-
7For HA, join additional control-plane nodes with kubeadm join --control-plane --certificate-key ...
8Join worker nodes with the worker join command captured earlier
9Install cluster add-ons (CoreDNS was installed by kubeadm; add ingress, metrics-server, snapshot-controller as required)
10Take an etcd snapshot now: sudo ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key snapshot save /var/backups/etcd/initial.db
11Validate the cluster end-to-end: deploy a test workload, hit it through the Service, then delete it

4 · Verification

Confirm the procedure actually fixed the problem.

✓kubectl get nodes reports every node Ready with the correct role labels
✓kubectl -n kube-system get pods reports every Pod Running or Completed
✓kubectl cluster-info returns both the control-plane and CoreDNS URLs without errors
✓kubectl -n kube-system get configmap kubeadm-config -o yaml shows the rendered init configuration
✓etcdctl endpoint health --cluster reports every member healthy and the leader elected
✓etcdctl endpoint status --cluster -w table shows consistent DB sizes across members (within 5%)
✓kubectl get --raw=/healthz returns ok
✓A test workload (e.g. kubectl run nginx --image=nginx --restart=Never --port=80 followed by kubectl expose pod nginx --port=80) is reachable from inside the cluster
✓The post-build etcd snapshot is restorable in a sandbox (see the kubernetes-rb-restore-etcd runbook)

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶If init fails after certificates have been written, run sudo kubeadm reset --force on the failed node before retrying with corrected configuration
↶If a CNI misconfiguration is suspected, kubectl -n kube-system get pods -o wide shows the CNI DaemonSet status; uninstall via the CNI install instructions before re-applying
↶If the cluster must be abandoned, sudo kubeadm reset --force on every node and remove /etc/kubernetes, /var/lib/kubelet and /var/lib/etcd
↶A working post-init etcd snapshot must be retained even on a rollback path - it is the baseline of the new cluster identity
↶If a worker node fails to join, run sudo kubeadm reset on it; do not retry without reading journalctl -u kubelet

6 · Escalation

When the runbook isn't enough, contact:

· kubeadm init fails with cri-socket errors: verify the container runtime socket path and that crictl info succeeds on every node
· CNI Pods stuck in CrashLoopBackOff after install: the CNI does not match the pod CIDR or the network plugin name expected by kubelet; switch CNI cleanly or fix the config before adding workloads
· etcd fails to elect a leader or members report divergent revision: never proceed; restore from snapshot before re-trying init
· Token or certificate key lost: a new token can be created with kubeadm token create --print-join-command; a new certificate key with kubeadm init phase upload-certs --upload-certs --certificate-key (rotate, do not reuse)
· Time skew across control-plane nodes >30s: stop and fix NTP; do not proceed with a skewed cluster

Building a cluster is the only operation that creates a long-lived identity the cluster will depend on for the rest of its life. Every later runbook in this course assumes this one succeeded: a stable control plane, a working CNI, a known etcd topology, and a baseline etcd snapshot.

Before you begin

The pre-checks above are not formality. kubeadm init will succeed on a cluster with wrong time, swap enabled or a missing kernel module, and the failure will show up at 02:00 three weeks later when the kubelet clock drifts past the API server certificate expiry. Every pre-check is something the next incident will check for you.

1. Capture the kubeadm configuration

kubeadm reads a kubeadm-config.yaml. Put it under version control before you pass it to init, so the cluster’s intent is reproducible.

# kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta4
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 10.0.0.10
  bindPort: 6443
nodeRegistration:
  criSocket: unix:///run/containerd/containerd.sock
---
kubernetesVersion: v1.34.0
controlPlaneEndpoint: 'lb.kube.internal:6443"
networking:
  podSubnet: '10.244.0.0/16"
  serviceSubnet: '10.96.0.0/16"
etcd:
  local:
    dataDir: '/var/lib/etcd"

Read-only / SafeCapture the kubeadm configuration


sudo kubeadm config validate --config kubeadm-config.yaml

2. Initialise the first control-plane

Read-only / SafeInitialise the first control-plane


sudo kubeadm init  \
--config kubeadm-config.yaml \
--upload-certs \
--skip-phases=addon/kube-proxy \
| tee /var/log/kubeadm-init.log

# Capture the worker join command
grep -A1 'kubeadm join' /var/log/kubeadm-init.log | tail -2

# Capture the control-plane join command and certificate key
grep -E 'kubeadm join|--certificate-key' /var/log/kubeadm-init.log | head -4

--upload-certs lets you add more control-plane nodes without sharing the CA key out of band. The certificate key is shown once; capture it to a secret manager, not a chat window.

Read-only / SafeInitialise the first control-plane


kubectl --kubeconfig=/etc/kubernetes/admin.conf -n kube-system get pods
kubectl --kubeconfig=/etc/kubernetes/admin.conf get --raw=/healthz

3. Install the CNI

The CNI must match podSubnet in kubeadm-config.yaml. Cilium is the default for this course; Calico and Flannel work with the same discipline.

Read-only / SafeInstall the CNI


helm repo update

helm install cilium cilium/cilium \
--namespace kube-system \
--set kubeProxyReplacement=true \
--set k8sServiceHost=lb.kube.internal \
--set k8sServicePort=6443 \
--set ipam.mode=kubernetes \
--set ipv4NativeRoutingCIDR=10.244.0.0/16

# Wait until every Cilium Pod is Running
kubectl -n kube-system rollout status ds/cilium --timeout=5m
kubectl -n kube-system rollout status deploy/cilium-operator --timeout=5m

If kubeProxyReplacement=true is set, the kube-proxy DaemonSet is not installed (kubeadm was told to skip it). Without this, the cluster ends up with two competing dataplanes and unpredictable Service behaviour. Verify:

Read-only / SafeInstall the CNI


kubectl -n kube-system logs -l k8s-app=cilium --tail=50 | grep -i 'ready' || true

4. Validate the cluster end-to-end

A “Running” kubeadm init is not a working cluster. The smoke test must exercise a Pod, a Service, and DNS resolution.

Read-only / SafeValidate the cluster end-to-end


kubectl run smoketest --image=registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7 --restart=Never --command -- sleep 3600

# Wait for it to be Ready and get a Pod IP
kubectl wait --for=condition=Ready pod/smoketest --timeout=120s
POD_IP=$(kubectl get pod smoketest -o jsonpath='{.status.podIP}')
echo "Pod IP: $POD_IP"

# Pod-to-Pod (cluster-internal ping)
kubectl exec smoketest -- ping -c2 "$POD_IP"

# DNS resolution
kubectl exec smoketest -- nslookup kubernetes.default

# Tear down
kubectl delete pod smoketest --wait=false

If any step fails, do not proceed to “Validation passed”. The next runbook will assume every one of those commands worked.

5. Take the baseline etcd snapshot

The cluster has no history. The first snapshot is the baseline every later restore will be measured against.

Read-only / SafeTake the baseline etcd snapshot


--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
snapshot save /var/backups/etcd/post-build.db

sudo etcdutl snapshot status /var/backups/etcd/post-build.db -w table

# Push off-host immediately - local disk is not backup
sudo rsync -a /var/backups/etcd/post-build.db backup@kube-backup.internal:/srv/etcd-snapshots/
sha256sum /var/backups/etcd/post-build.db | tee /var/backups/etcd/post-build.db.sha256

6. Bring up HA control-plane and workers

For HA clusters, repeat the kubeadm join --control-plane command on each additional control-plane node. For workers, use the worker join command.

Cluster-wide riskBring up HA control-plane and workers


sudo kubeadm join lb.kube.internal:6443 \
--token <token> \
--discovery-token-ca-cert-hash sha256:<hash> \
--control-plane \
--certificate-key <key> \
--cri-socket unix:///run/containerd/containerd.sock

Read-only / SafeBring up HA control-plane and workers


sudo kubeadm join lb.kube.internal:6443 \
--token <token> \
--discovery-token-ca-cert-hash sha256:<hash> \
--cri-socket unix:///run/containerd/containerd.sock

7. Final validation

Read-only / SafeFinal validation


kubectl get pods -A -o wide | grep -v 'Running|Completed' || echo "all pods healthy"
kubectl get --raw=/healthz; echo
kubectl get --raw=/readyz; echo
etcdctl --endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
endpoint health --cluster -w table

The cluster is now in a known state. Every later change is a controlled delta from here.

Common pitfalls

Symptom	Cause	Action
Nodes `Ready` but Pods stuck in `ContainerCreating`	CNI not installed or pod CIDR mismatch	Re-install CNI; reconcile `podSubnet`
`kubectl get nodes` shows only the init node	Workers never joined; CA hash or token wrong	Re-create the token; verify hash with `openssl x509 -pubkey`
`etcdctl endpoint health` reports two members, not three	Join never completed; check `journalctl -u kubelet` on the missing node	Run `kubeadm reset` and re-join
`dial tcp 10.96.0.1:443` timeouts on every Pod	Service CIDR / kube-proxy / CNI dataplane broken	Verify CNI logs and `iptables-save \| grep KUBE` (if iptables mode)
API server repeatedly restarts	Certificate SAN mismatch (advertised address not in cert)	Re-init with corrected `apiServer.certSANs`

A cluster that fails any of these does not have a working Service dataplane. Treat it as “not built” and rebuild from this runbook rather than patching around the gap.