Lab 1: Build a kubeadm cluster from bare hosts

Objective

By the end of this lab you will have a three-node Kubernetes 1.34 cluster that you built yourself, from three hosts that had nothing on them, and you will be able to point at the file the cluster was built from. More usefully, you will have watched the cluster pass through the two states that confuse everyone the first time: a control plane whose own node is NotReady, and a worker that has registered but has no Pods on it yet. Neither is a fault. Both are explainable from node conditions, and you will explain them from node conditions rather than from memory.

Architecture

Three hosts, one control-plane node and two workers, on a flat layer-2 segment. etcd is stacked — it runs as a static Pod on the control-plane host, which is the kubeadm default described in the HA topology lesson. There is no load balancer, because there is one API server.

flowchart TB
    subgraph CP["k8s-cp-1 · 192.0.2.11"]
        API[kube-apiserver]
        ETCD[etcd member]
        SCH[kube-scheduler]
        CM[kube-controller-manager]
        K1[kubelet + containerd]
    end
    subgraph W1["k8s-w-1 · 192.0.2.12"]
        K2[kubelet + containerd]
        P1[Pods]
    end
    subgraph W2["k8s-w-2 · 192.0.2.13"]
        K3[kubelet + containerd]
        P2[Pods]
    end
    K2 -->|:6443| API
    K3 -->|:6443| API
    API <--> ETCD

Cluster-wide: Pod subnet 10.244.0.0/16, Service subnet 10.96.0.0/12, DNS domain cluster.local.

Requirements

• Three hosts, each with 2 vCPU, 4 GiB RAM and 30 GiB disk. Two vCPUs is not a suggestion: kubeadm’s preflight check fails a control-plane node with one.
• Debian 12 (bookworm), freshly installed, reachable by SSH, with sudo. Ubuntu 24.04 works identically; the package repository lines are the only difference and both are given below.
• Static addresses on all three hosts, and forward and reverse name resolution between them (/etc/hosts is sufficient for a lab).
• Outbound HTTPS to pkgs.k8s.io, registry.k8s.io, download.docker.com and github.com. An air-gapped variant of this lab needs a mirror and is out of scope here.
• No out-of-band access requirement. This lab does not reconfigure the hosts’ primary interfaces, SSH, or the default route. It does load kernel modules, set sysctls, and — in Cleanup — flush iptables rules that kubeadm and the CNI created. Read the Cleanup warning before you run it on a host that has a firewall you care about.
• Roughly 3 GiB of disk consumed per node by images and etcd.

Scenario

You have been handed three VMs and a sentence: “we need a cluster to test the new ingress controller on, same version as production.” Production is on 1.34. Nobody has written down how the production cluster was built, so this is also the first time anyone will produce a kubeadm-config.yaml for this estate. That second job is the one that outlasts the lab: a cluster built from flags on somebody’s shell history cannot be rebuilt, and a cluster that cannot be rebuilt is a cluster you are afraid to change.

Tasks

Task 1 — Record the starting state on all three hosts

Do this on every host before anything is installed. It costs a minute and it is the only reference you will have during Cleanup.

mkdir -p "$HOME/kubeadm-lab"
cd "$HOME/kubeadm-lab"

{
  echo "HOST: $(hostname -f)"
  echo "DATE: $(date -Is)"
  echo "--- kernel"
  uname -r
  echo "--- swap"
  swapon --show
  echo "--- existing kubernetes state"
  ls -la /etc/kubernetes /var/lib/etcd 2>&1
  echo "--- loaded modules of interest"
  lsmod | grep -E 'br_netfilter|^overlay ' || echo none
  echo "--- iptables rule counts"
  sudo iptables-save | wc -l
} > pre-lab-state.txt

cat pre-lab-state.txt

Two lines matter. If ls /etc/kubernetes lists anything other than “No such file or directory”, this host is not clean and kubeadm init or kubeadm join will fail on a preflight check about existing files. If swapon --show prints a table, Task 2 is going to change something you must undo later.

Task 2 — Node preparation: modules, sysctls, swap

Run this block on all three hosts. It is the same on every node; there is nothing control-plane-specific about it.

sudo tee /etc/modules-load.d/k8s.conf >/dev/null <<'EOF'
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

sudo tee /etc/sysctl.d/99-kubernetes.conf >/dev/null <<'EOF'
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

sudo sysctl --system >/dev/null

Then confirm, rather than assuming the file was read:

$ sysctl net.ipv4.ip_forward net.bridge.bridge-nf-call-iptables

net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1

The order in that block is deliberate. net.bridge.bridge-nf-call-iptables does not exist as a sysctl until br_netfilter is loaded, so writing the file first and loading the module second produces a sysctl --system that reports “No such file or directory” and moves on. That is the single most common reason a freshly built cluster has Pods that can reach the outside world and cannot reach a Service IP.

Now swap. kubeadm’s preflight refuses to run with swap active unless you tell it otherwise, and the reason is scheduling arithmetic: the kubelet’s eviction thresholds are expressed against physical memory, and a node that can page will sail past the point where the scheduler believed it was full.

sudo swapoff -a
sudo sed -i.pre-lab '/\sswap\s/ s/^/#/' /etc/fstab

swapon --show
grep -n swap /etc/fstab

swapon --show should now print nothing at all. The sed -i.pre-lab leaves /etc/fstab.pre-lab behind — that is your restore point, and Cleanup uses it.

Task 3 — Install and configure containerd

Still on all three hosts. Debian’s own containerd package is older than the line this course targets, so take it from the Docker repository, which is what the upstream container-runtimes documentation points at.

sudo apt-get update
sudo apt-get install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings

curl -fsSL https://download.docker.com/linux/debian/gpg \
  | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg

CODENAME="$(. /etc/os-release && echo "$VERSION_CODENAME")"
ARCH="$(dpkg --print-architecture)"

echo "deb [arch=$ARCH signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian $CODENAME stable" \
  | sudo tee /etc/apt/sources.list.d/docker.list >/dev/null

sudo apt-get update
sudo apt-get install -y containerd.io

On Ubuntu 24.04, substitute ubuntu for debian in both the key URL and the repository line; everything else is identical.

containerd ships a config that is deliberately minimal, so generate the full default and change the one setting that matters:

sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml >/dev/null
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml

sudo systemctl restart containerd
sudo systemctl enable containerd

grep -n 'SystemdCgroup' /etc/containerd/config.toml
sudo ctr version

SystemdCgroup = true is the whole point of this task. systemd owns the cgroup hierarchy on a Debian host; if containerd creates its own cgroups under a different driver, you get two managers writing to the same tree. The failure is not immediate — the node registers, Pods start — and then under memory pressure the kubelet’s accounting and the actual cgroup limits disagree, and Pods are killed or not killed at the wrong times. The KubeletConfiguration in Task 5 sets the kubelet to the same driver; the two settings are a pair and are never changed independently.

Task 4 — Install the Kubernetes packages, pinned

All three hosts. The repository URL carries the minor version, so this is where you choose 1.34 — not later, in a flag.

sudo install -m 0755 -d /etc/apt/keyrings

curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.34/deb/Release.key \
  | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
sudo chmod a+r /etc/apt/keyrings/kubernetes-apt-keyring.gpg

echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.34/deb/ /' \
  | sudo tee /etc/apt/sources.list.d/kubernetes.list >/dev/null

sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

kubeadm version -o short
kubectl version --client -o yaml | head -5

apt-mark hold is not caution, it is correctness. Kubernetes has a supported version-skew window between the control plane and the kubelet, and an unattended upgrade that walks the kubelet forward past it will take the node out on a Tuesday morning with no change ticket attached. Record the exact version kubeadm version -o short printed — Task 5 needs it.

Task 5 — Write the cluster’s configuration file

On k8s-cp-1 only. This file is the deliverable that outlives the lab.

# kubeadm-config.yaml
# Substitute: advertiseAddress, nodeRegistration.name, and kubernetesVersion
# (use exactly what `kubeadm version -o short` printed in Task 4).
apiVersion: kubeadm.k8s.io/v1beta4
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 192.0.2.11
  bindPort: 6443
nodeRegistration:
  name: k8s-cp-1
  criSocket: unix:///run/containerd/containerd.sock
---
apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
kubernetesVersion: v1.34.0
networking:
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.96.0.0/12
  dnsDomain: cluster.local
etcd:
  local:
    dataDir: /var/lib/etcd
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd

Validate it before it touches anything:

$ sudo kubeadm config validate --config kubeadm-config.yaml

ok

Pull the images before init, so that a slow registry shows up as a slow docker-style pull rather than as a mysterious timeout inside a kubeadm phase:

sudo kubeadm config images pull --config kubeadm-config.yaml
sudo crictl images

Task 6 — Initialise the control plane

Dry-run first. --dry-run walks every phase against a temporary directory and prints what it would write, without touching /etc/kubernetes.

cd "$HOME/kubeadm-lab"
sudo kubeadm init --config kubeadm-config.yaml --dry-run 2>&1 | tee init-dryrun.txt
tail -40 init-dryrun.txt

Then the real thing:

$ sudo kubeadm init --config kubeadm-config.yaml 2>&1 | tee init.txt

[init] Using Kubernetes version: v1.34.0
[preflight] Running pre-flight checks
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[kubeconfig] Writing "admin.conf" kubeconfig file
[control-plane] Creating static Pod manifest for "kube-apiserver"
[etcd] Creating static Pod manifest for local etcd
[wait-control-plane] Waiting for the kubelet to boot up the control plane
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config"
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

Read init.txt as a list of phases, in order, each of which produced files you can go and look at. The certs phase filled /etc/kubernetes/pki. The control-plane and etcd phases wrote four manifests into /etc/kubernetes/manifests, and the kubelet — which was already running and failing — noticed them and started four static Pods. Confirm both:

ls -1 /etc/kubernetes/manifests/
sudo crictl ps --name 'kube-apiserver|etcd|kube-scheduler|kube-controller-manager'

Now capture the join command immediately, because the token in the init output has a 24-hour life and the output scrolls away:

cd "$HOME/kubeadm-lab"
sudo kubeadm token create --print-join-command | tee join-command.txt

Configure kubectl for your user:

mkdir -p "$HOME/.kube"
sudo cp -i /etc/kubernetes/admin.conf "$HOME/.kube/config"
sudo chown "$(id -u):$(id -g)" "$HOME/.kube/config"

kubectl cluster-info

Task 7 — Understand the NotReady node before you fix it

This is the step people skip. Look at the node now, before the CNI is installed.

$ kubectl get nodes

NAME       STATUS     ROLES           AGE   VERSION
k8s-cp-1   NotReady   control-plane   2m    v1.34.0

Do not install anything yet. Ask the node why:

kubectl describe node k8s-cp-1 | sed -n '/^Conditions:/,/^Addresses:/p'
kubectl -n kube-system get pods

The Ready condition carries a reason, and on a cluster with no CNI it names the container runtime network as uninitialised. The corroborating evidence is in the Pod list: CoreDNS is Pending, because CoreDNS is an ordinary Deployment that needs a Pod IP, and nothing can hand out Pod IPs yet. kube-proxy and the four static Pods are Running, because they use the host network and never needed one.

This matters beyond the lab. “Node NotReady” is one symptom with several causes — no CNI, kubelet stopped, disk pressure, certificate expiry — and the condition message distinguishes them in one command. Reaching for a CNI reinstall because a node says NotReady, when the real cause was an expired kubelet certificate, is a genuinely expensive mistake.

Task 8 — Install the CNI

The Pod subnet you gave kubeadm and the one the CNI uses must be the same value. flannel’s default network configuration is 10.244.0.0/16, which is why Task 5 used it.

Pick a release tag from the flannel releases page in this lab’s references — this course targets the 0.25.x line — and pin it. Do not apply a manifest from a branch name; a branch moves and your cluster silently changes.

# Substitute the 0.25.x tag you chose from the releases page:
FLANNEL_VERSION=v0.25.7

curl -fsSLO "https://github.com/flannel-io/flannel/releases/download/$FLANNEL_VERSION/kube-flannel.yml"
grep -n 'Network' kube-flannel.yml | head

Confirm with your own eyes that the Network value in the ConfigMap inside that file reads 10.244.0.0/16 before applying it. Then:

kubectl apply -f kube-flannel.yml

kubectl -n kube-flannel rollout status daemonset/kube-flannel-ds --timeout=180s
kubectl wait node/k8s-cp-1 --for=condition=Ready --timeout=180s
kubectl -n kube-system rollout status deployment/coredns --timeout=180s

Watch what that sequence proves, in order: the CNI DaemonSet is running on the node; the node’s Ready condition flipped as a direct consequence; and CoreDNS — which was Pending for want of a Pod IP one command ago — is now able to schedule. Three commands, one causal chain.

Task 9 — Join the two workers

On k8s-w-1 and k8s-w-2, run the command from join-command.txt. It looks like this, with your own token and hash:

$ sudo kubeadm join 192.0.2.11:6443 --token TOKEN --discovery-token-ca-cert-hash sha256:HASH

[preflight] Running pre-flight checks
[preflight] Reading configuration from the "kubeadm-config" ConfigMap
[kubelet-start] Starting the kubelet
[kubelet-check] The kubelet is healthy after 1.002s

This node has joined the cluster.

Now verify the join from the control-plane host, not from the worker. A worker that prints “This node has joined the cluster” has succeeded at its half of the handshake; whether the cluster agrees is a separate question, and it is the one that matters.

kubectl get nodes -o wide
kubectl wait node/k8s-w-1 node/k8s-w-2 --for=condition=Ready --timeout=180s
kubectl -n kube-flannel get pods -o wide

Each worker gets a flannel Pod within seconds of registering, because the CNI is a DaemonSet and a DaemonSet’s whole job is “one Pod per node, including nodes that did not exist when I was created”. You did not have to install anything on the workers to make the network work, and that is worth noticing.

Task 10 — Prove the cluster actually works

A cluster that lists three Ready nodes has proved that three kubelets can talk to an API server. It has not proved that a Pod on one node can reach a Pod on another, which is the thing everything else depends on. Write this manifest on k8s-cp-1:

# netcheck.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: kubeadm-lab
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: netcheck
  namespace: kubeadm-lab
spec:
  replicas: 2
  selector:
    matchLabels:
      app: netcheck
  template:
    metadata:
      labels:
        app: netcheck
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: netcheck
      containers:
        - name: web
          image: nginx:1.27-alpine
          ports:
            - containerPort: 80
          resources:
            requests:
              cpu: 50m
              memory: 32Mi
            limits:
              memory: 64Mi

The topologySpreadConstraints block is what forces the two replicas onto different nodes. Without it the scheduler is free to put both on one worker, and the test proves nothing.

kubectl apply -f netcheck.yaml
kubectl -n kube-lab-placeholder version >/dev/null 2>&1 || true
kubectl -n kubeadm-lab rollout status deployment/netcheck --timeout=120s
kubectl -n kubeadm-lab get pods -o wide

Read the NODE column: the two Pods must be on different hosts. Now send traffic from one to the other, by Pod IP, across the node boundary:

POD_A="$(kubectl -n kubeadm-lab get pods -l app=netcheck \
  -o jsonpath='{.items[0].metadata.name}')"
IP_B="$(kubectl -n kubeadm-lab get pods -l app=netcheck \
  -o jsonpath='{.items[1].status.podIP}')"

echo "from $POD_A to $IP_B"
kubectl -n kubeadm-lab exec "$POD_A" -- wget -qO- --timeout=5 "http://$IP_B" | head -4

A response means the CNI is routing between nodes. A hang means it is not, and the first thing to check is the sysctl pair from Task 2 on the node that hosts the destination Pod.

Finally prove cluster DNS, which is the other thing everything depends on:

kubectl -n kubeadm-lab expose deployment netcheck --port=80
kubectl -n kubeadm-lab exec "$POD_A" -- wget -qO- --timeout=5 http://netcheck | head -4
kubectl -n kubeadm-lab exec "$POD_A" -- nslookup netcheck.kubeadm-lab.svc.cluster.local

Task 11 — Take the baseline etcd snapshot

The cluster now has an identity — a CA, a set of certificates, a member list — that exists in exactly one place. Back it up before you have anything worth losing, so that the procedure is rehearsed rather than improvised.

etcdctl is not installed on the host; it is inside the etcd image, so run it there. The etcd static Pod mounts the host’s /var/lib/etcd, so a snapshot written to that path lands on the host filesystem.

kubectl -n kube-system exec etcd-k8s-cp-1 -- \
  etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /var/lib/etcd/etcd-baseline.db

sudo mv /var/lib/etcd/etcd-baseline.db "$HOME/kubeadm-lab/etcd-baseline.db"
sudo chown "$(id -u):$(id -g)" "$HOME/kubeadm-lab/etcd-baseline.db"
ls -l "$HOME/kubeadm-lab/etcd-baseline.db"

Move it off the data directory, as above, and then off the host. A snapshot that lives only on the machine whose disk you are protecting against is not a backup.

Validation

Work through each of these and confirm the stated result. Collect the output into cluster-baseline.txt, which is a deliverable.

• kubectl get nodes -o wide lists three nodes, all Ready, all at the same VERSION, with the roles column showing control-plane on exactly one.
• kubectl -n kube-system get pods shows every Pod Running: one each of etcd-, kube-apiserver-, kube-controller-manager-, kube-scheduler-, three kube-proxy-, and two coredns-.
• kubectl -n kube-flannel get pods -o wide shows three Pods, one per node.
• kubectl get --raw='/readyz?verbose' returns a list of checks each followed by ok, ending in readyz check passed.
• kubectl -n kube-system get configmap kubeadm-config -o yaml contains the ClusterConfiguration you wrote in Task 5. If it does not match your file, the cluster was built from something else and your deliverable is wrong.
• The cross-node wget in Task 10 returns nginx’s welcome HTML, and the NODE column of kubectl -n kubeadm-lab get pods -o wide shows the two replicas on different hosts.
• nslookup netcheck.kubeadm-lab.svc.cluster.local from inside a Pod resolves to an address in 10.96.0.0/12.
• sudo kubeadm certs check-expiration on k8s-cp-1 lists every certificate with roughly 364 days remaining and the CA with roughly 9 years. Save this output — it is the only cheap record of when this cluster’s clock starts ticking.
• ls -l etcd-baseline.db shows a file of a few megabytes. Zero bytes means the snapshot command failed and the shell swallowed it.

Expected Outcome

A three-node cluster, and on k8s-cp-1 a working directory:

kubeadm-lab/
├── cluster-baseline.txt
├── etcd-baseline.db
├── init-dryrun.txt
├── init.txt
├── join-command.txt
├── kube-flannel.yml
├── kubeadm-config.yaml
├── netcheck.yaml
└── pre-lab-state.txt

You can rebuild this cluster from kubeadm-config.yaml and kube-flannel.yml alone. You can state what version every component is, why the node was NotReady for four minutes, and which single line in the config file you would have to plan a change window around.

Troubleshooting

kubeadm init fails at preflight with [ERROR NumCPU] or [ERROR Mem]. The control-plane host needs 2 CPUs and roughly 1.7 GiB of RAM. Resize the VM; do not pass --ignore-preflight-errors to make the message go away, because the resulting control plane will be slow in ways you will misdiagnose later.

kubeadm init fails at wait-control-plane. The manifests were written but no static Pod came up. kubectl does not work yet, so ask the two components that do: sudo journalctl -u kubelet -n 100 --no-pager and sudo crictl ps -a. The commonest cause on a fresh host is a cgroup-driver mismatch — check that Task 3’s SystemdCgroup = true really landed, with grep SystemdCgroup /etc/containerd/config.toml.

Node stays NotReady after the CNI is applied. Check the flannel Pod on that specific node with kubectl -n kube-flannel get pods -o wide and then kubectl -n kube-flannel logs. If flannel is CrashLoopBackOff, the usual cause is a Pod CIDR mismatch: the value in kube-flannel.yml is not the podSubnet you gave kubeadm.

kubeadm join fails with couldn't validate the identity of the API Server. The CA hash in your join command does not match the cluster’s CA. That happens when the token was created against a previous init on the same host. Re-issue with sudo kubeadm token create --print-join-command on k8s-cp-1 and use the new output.

kubeadm join fails on FileAvailable--etc-kubernetes-kubelet.conf. The worker has state from a previous cluster. Run the Cleanup block for a worker on that host, then retry.

Pods on different nodes cannot reach each other, but same-node traffic works. This is the sysctl pair from Task 2, on the destination node. Check sysctl net.bridge.bridge-nf-call-iptables there, and check that lsmod | grep br_netfilter returns a row — if the module was loaded by hand and /etc/modules-load.d/k8s.conf was never written, a reboot undoes it.

kubectl says connection refused on port 6443 after a reboot. The kubelet starts the static Pods; give it a minute. If it persists, sudo systemctl status kubelet and sudo crictl ps -a are the two commands, in that order.

Cleanup

Cleanup returns each host to the state Task 1 recorded. Run the worker steps on k8s-w-1 and k8s-w-2 first, then the control-plane steps on k8s-cp-1 — draining a node whose API server you have already destroyed does not work.

Step 1, on k8s-cp-1, remove the workload and the nodes from the cluster while the cluster still exists:

kubectl delete -f "$HOME/kubeadm-lab/netcheck.yaml" --ignore-not-found
kubectl drain k8s-w-1 --ignore-daemonsets --delete-emptydir-data --force
kubectl drain k8s-w-2 --ignore-daemonsets --delete-emptydir-data --force
kubectl delete node k8s-w-1 k8s-w-2

Step 2, keep the deliverables. They are small text files and they are the evidence that you did the lab.

mkdir -p "$HOME/kubeadm-lab-deliverables"
cp -a "$HOME/kubeadm-lab/kubeadm-config.yaml" \
      "$HOME/kubeadm-lab/join-command.txt" \
      "$HOME/kubeadm-lab/cluster-baseline.txt" \
      "$HOME/kubeadm-lab/etcd-baseline.db" \
      "$HOME/kubeadm-lab-deliverables/"

Step 3, on each worker, then on the control plane:

sudo kubeadm reset --force

sudo iptables -F
sudo iptables -t nat -F
sudo iptables -t mangle -F
sudo iptables -X

sudo rm -rf /etc/kubernetes /var/lib/etcd /var/lib/cni /etc/cni/net.d
sudo rm -rf "$HOME/.kube"

Step 4, on all three hosts, undo the host changes from Tasks 2 through 4:

sudo rm -f /etc/modules-load.d/k8s.conf /etc/sysctl.d/99-kubernetes.conf
sudo rm -f /etc/apt/sources.list.d/kubernetes.list /etc/apt/sources.list.d/docker.list
sudo apt-mark unhold kubelet kubeadm kubectl
sudo apt-get purge -y kubeadm kubectl kubelet containerd.io
sudo apt-get autoremove -y

sudo mv /etc/fstab.pre-lab /etc/fstab
sudo swapon -a

swapon --show
sudo sysctl --system >/dev/null

Step 5, confirm you are back where Task 1 found you:

cd "$HOME/kubeadm-lab"
{
  echo "--- swap"; swapon --show
  echo "--- existing kubernetes state"; ls -la /etc/kubernetes 2>&1
  echo "--- iptables rule counts"; sudo iptables-save | wc -l
} > post-lab-state.txt

diff -u pre-lab-state.txt post-lab-state.txt || true

The diff will show the hostname and date lines differing, and the swap and /etc/kubernetes lines matching. If /etc/kubernetes still exists, the reset did not complete and the host is not clean for a future lab.

What You Learned

• A cluster is built from a file, not from a command line. You produced kubeadm-config.yaml, validated it before it touched a host, and then found the same content in the kubeadm-config ConfigMap afterwards. That round trip is what makes the cluster reproducible.
• NotReady is a condition with a message, not a verdict. You read the reason off the node before installing anything, and the CoreDNS Pending Pods corroborated it. The same command distinguishes a missing CNI from an expired certificate at 03:00.
• The cgroup driver is set in two places and must agree. containerd’s SystemdCgroup = true and the KubeletConfiguration’s cgroupDriver: systemd are one decision written twice, and the failure they cause is delayed and looks like a memory problem.
• The sysctl and the module are ordered. br_netfilter must be loaded before net.bridge.bridge-nf-call-iptables exists to be set, and the symptom of getting it backwards is Service traffic failing while everything else looks fine.
• Ready nodes are not a working network. You proved Pod-to-Pod across a node boundary and cluster DNS from inside a Pod, which are the two facts every later workload silently assumes.
• The first snapshot is part of the build. You took it before the cluster had anything worth losing, which is the only time the procedure is cheap to rehearse.

Production notes

This lab is a cluster build compressed into an afternoon. In production the same work is a scheduled change with three distinct gates.

Before the window. Node preparation — Tasks 2 through 4 — is not part of the change; it is done ahead of time by configuration management and verified independently, so the window opens with hosts that already pass preflight. The kubeadm-config.yaml goes through review in version control, and the review question is the one from Task 5’s callout: is controlPlaneEndpoint right, and do the Pod and Service subnets collide with anything else routed on this network? Both are effectively permanent.

During the window. kubeadm init on a production control plane is a one-way operation from the moment the certs phase completes. The rollback is kubeadm reset and start again, which is cheap on a new cluster and impossible on one with workloads — so the window’s real content is the validation section above, run in full, before anything is allowed to schedule on the cluster.

After the window. Two artefacts leave the window or it did not succeed: the etcd snapshot, stored somewhere that survives the loss of the control-plane host, and the kubeadm certs check-expiration output, which tells you the date this cluster’s certificates need renewing. On a cluster that is rebuilt every few months nobody ever hits that date. On a cluster that runs for two years, that date is an incident with a year of warning that nobody read.