medium riskservice affecting~45 min

Runbook: Join a Worker Node to the Cluster

1 · Prerequisites

Confirm every item is in place before any state change.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Confirm the cluster exists and is healthy: kubectl get nodes returns Ready and kubectl -n kube-system get pods shows every component Running
· Confirm the target node meets the kubeadm prerequisites: kernel modules, sysctls, swap disabled, NTP synchronised (see the kubernetes-rb-build-cluster pre-checks)
· Confirm the kubelet and kubeadm packages on the new node match the cluster minor version: kubeadm version reports the same minor as the control plane
· Confirm the container runtime on the new node is the same one the cluster uses: crictl info shows the same runtime name and version as a known-good worker
· Confirm the join token has not expired (default TTL 24h): kubeadm token list reports a token with a future expiry, or generate a fresh one
· Confirm the discovery-token-ca-cert-hash matches the cluster CA: openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex
· Confirm the node hostname is unique and DNS-resolvable to its IP (both ways) from the control plane
· Confirm the certificate signing requests will not pile up: kubectl get csr shows zero Pending for unrelated nodes

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1On the new node, configure /etc/hosts, swap off, kernel modules, sysctls, container runtime, kubelet and kubectl to the target version
2On a control-plane node, generate a fresh join command if needed: sudo kubeadm token create --print-join-command
3On the new node, run the join command printed by kubeadm: sudo kubeadm join <endpoint>:6443 --token <token> --discovery-token-ca-cert-hash sha256:<hash> --cri-socket unix:///run/containerd/containerd.sock
4On a control-plane node, approve the node certificate if RBAC requires it: kubectl get csr then kubectl certificate approve <csr-name>
5On a control-plane node, label the node for its intended role (e.g. kubectl label node <node> node-role.kubernetes.io/work= --overwrite)
6Taint the node for staged rollout: kubectl taint nodes <node> node.kubernetes.io/unschedulable=:NoSchedule until you are ready to admit workloads
7Wait for the CNI DaemonSet to schedule a Pod on the node: kubectl -n kube-system get pods -o wide | grep <node>
8Confirm the node can run a test Pod by deploying a small workload (kubectl run ... --dry-run=server -o yaml) and forcing the scheduler to place it via nodeName
9If scheduling is desired, untaint: kubectl taint nodes <node> node.kubernetes.io/unschedulable-
10Verify capacity was reported to the scheduler: kubectl describe node <node> | grep -A20 "Allocatable"

4 · Verification

Confirm the procedure actually fixed the problem.

✓kubectl get node <node> reports Ready with the correct role labels and taints
✓kubectl get node <node> -o jsonpath='{.status.conditions}' shows Ready=True,MemoryPressure=False,DiskPressure=False,PIDPressure=False
✓kubectl -n kube-system get pods -o wide --field-selector spec.nodeName=<node> shows the CNI Pod (Cilium/Calico) Running
✓A test Pod with nodeName: <node> reaches Running and Ready and can ping its own Pod IP
✓kubectl describe node <node> shows non-zero Allocatable resources and a kubelet version matching the cluster minor
✓kubectl get csr shows no Pending requests for the new node

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶If the join fails partway through, run sudo kubeadm reset --force on the new node and remove /etc/kubernetes, /var/lib/kubelet before retrying
↶If the new node has certificate or RBAC issues, delete the CSR (kubectl delete csr <name>) and re-create the join
↶If a workload landed on the new node and the node must be removed: kubectl drain <node> --ignore-daemonsets --delete-emptydir-data, then kubectl delete node <node>, then kubeadm reset --force on the node
↶If the cluster rejects the join because of certificate hash mismatch, do not edit /etc/kubernetes/pki/ca.crt on the new node; verify the hash on the control plane and re-generate the join command
↶A bad label or taint is reverted with kubectl label node <node> <key>- and kubectl taint nodes <node> <key>-

6 · Escalation

When the runbook isn't enough, contact:

· Node joins but never goes Ready: kubelet is failing its TLS handshake with the API server; check journalctl -u kubelet -n 200 for x509 errors
· CNI Pod does not start on the new node: the CNI DaemonSet tolerations do not include the node taints, or the pod CIDR has no free address for the node
· Join succeeds but kubectl get nodes never lists the new node: clock skew on the new node past the certificate validity window; fix NTP before retrying
· Multiple Pending CSRs appear for the new node: a previous half-completed join left orphan CSRs; clean them up to avoid confusing later joins
· Node appears with OutOfDisk or MemoryPressure=True immediately after joining: the new node has different Allocatable reservations than the rest of the fleet; fix kubelet flags, not the node

A node that joins with wrong prerequisites will appear Ready until something schedules onto it, then fail in a way that costs an investigation. Every pre-check above is a known production failure mode for joins.

1. Prepare the node

The kubeadm prerequisites apply here exactly as they do for control planes. The new node must:

Have NTP synchronised
Have swap disabled (swapoff -a plus removal from /etc/fstab)
Load overlay and br_netfilter modules
Have net.ipv4.ip_forward=1, net.bridge.bridge-nf-call-iptables=1, net.bridge.bridge-nf-call-ip6tables=1
Run the same container runtime version as the cluster

Read-only / SafePrepare the node


timedatectl status | grep -E 'System clock|NTP'
lsmod | grep -E 'br_netfilter|^overlay '
sysctl net.ipv4.ip_forward net.bridge.bridge-nf-call-iptables net.bridge.bridge-nf-call-ip6tables
swapon --show
crictl info | jq -r '.status.runtimeReady,.status.images.length'
kubeadm version -o short
kubelet --version

2. Generate the join command

A join token lives 24 hours by default. Do not re-use one from a stale terminal session — its discovery token may have expired.

Read-only / SafeGenerate the join command


sudo kubeadm token create --print-join-command --ttl=24h

# Capture the discovery-token-ca-cert-hash explicitly for cross-check
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt \
| openssl rsa -pubin -outform der 2>/dev/null \
| openssl dgst -sha256 -hex \
| awk '{print $2}'

3. Join

Read-only / SafeJoin


sudo kubeadm join lb.kube.internal:6443 \
--token <token> \
--discovery-token-ca-cert-hash sha256:<hash> \
--cri-socket unix:///run/containerd/containerd.sock \
| tee /var/log/kubeadm-join.log

The command prints bootstrap token lifecycle messages and a final line like This node has joined the cluster. If the final line is absent the join did not complete and the node is not a member; the rollback section covers how to clean up.

4. Approve and label

TLS bootstrap creates a CSR that kubeadm does not auto-approve unless RBAC grants it. With a sane cluster, approval is automatic; in a hardened cluster it is not.

Read-only / SafeApprove and label


kubectl get csr
kubectl get csr -o json | jq -r '.items[] | select(.spec.username | startswith("system:bootstrap:") ) | .metadata.name'

Read-only / SafeApprove and label


kubectl certificate approve <csr-name> || true

kubectl label node <node> node-role.kubernetes.io/worker= --overwrite
kubectl label node <node> workload.example.com/role=general --overwrite || true

# Stage the node for validation - keep it unschedulable until verified
kubectl taint nodes <node> node.kubernetes.io/joining=NoSchedule --overwrite
kubectl taint nodes <node> node.kubernetes.io/unschedulable=:NoSchedule --overwrite

5. Verify the node can actually run workloads

Joining is not the same as running a Pod. Validate the dataplane end-to-end on the new node before admitting workloads.

Read-only / SafeVerify the node can actually run workloads


kubectl -n kube-system get pods -o wide --field-selector spec.nodeName=<node>
# Expect the CNI DaemonSet Pod to be Running

kubectl describe node <node> | sed -n '/Conditions/,/Capacity/p'
# Expect Ready=True and pressures all False

# Force a test Pod onto the new node
cat <<'YAML' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: nodetest
spec:
nodeName: <node>
containers:
- name: pause
  image: registry.k8s.io/pause:3.10
YAML

kubectl wait --for=condition=Ready pod/nodetest --timeout=120s
kubectl exec nodetest -- ip addr show | grep -E 'eth0|inet '

kubectl delete pod nodetest --wait=false

If nodetest does not reach Ready, the join succeeded at the API layer but failed at the dataplane layer. Do not admit workloads.

6. Open the node to the scheduler

Read-only / SafeOpen the node to the scheduler


kubectl taint nodes <node> node.kubernetes.io/joining-
kubectl taint nodes <node> node.kubernetes.io/unschedulable-

kubectl describe node <node> | grep -E 'Allocatable|Node-local'
kubectl top node <node>

The node is now part of the fleet. The capacity column is what the HPA and scheduler will count; verify it matches what the hardware provides.

Common pitfalls

Symptom	Cause	Action
`kubeadm join` hangs at `discovering cluster info`	Wrong endpoint or blocked 6443/tcp from the new node	`nc -vz lb.kube.internal 6443` from the node
`x509: certificate is valid for ... not for <ip>`	`apiServer.certSANs` does not include the load-balancer DNS or IP	Update `kubeadm-config.yaml` and re-init the control plane
Node shows `Ready` but no Pods ever schedule	kubelet cannot pull images from the registry	Check `kubectl describe node` for `OutOfDisk` and `journalctl -u kubelet` for pull errors
`kubectl get nodes` lists the new node, then it disappears	kubelet lost its client cert; clock skew on the node	`systemctl status kubelet` and `chronyc tracking`
`Pending` CSRs pile up	Bootstrap RBAC removed or changed	Restore the `system:node-bootstrapper` ClusterRoleBinding, then `kubectl certificate approve`

A node that joins but cannot run a Pod is a node that pretends to add capacity. Treat the nodetest step as mandatory, not optional.