Node replacement and rollback — the recovery path

The expensive decision is not how to replace a node but when to stop repairing one. An expired certificate, a dead containerd, or a full disk is bounded work; a kernel panic, a failing disk, or a host that will not boot is not, and hours disappear into a node that was never coming back. This lesson sets out where the line falls, and the sequence that follows once it is crossed — drain, delete the Node object, re-provision, rejoin with a fresh token.

Repairable vs replaceable

A node is repairable if the issue can be fixed without replacing the host. Examples:

• Expired certificate.
• Failed containerd.
• Disk full.

A node is replaceable if the issue cannot be fixed without replacing the host. Examples:

• Hardware failure.
• Kernel panic.
• Boot failure.

flowchart TD
    A[Node failing] --> B{Repairable?}
    B -->|Yes| C[Fix the node]
    B -->|No| D[Replace the node]

The diagnostic is the node’s health.

The diagnostic

The canonical diagnostic:

# Substitute your own values before running:
NODE=worker-03                    # name of the Node object
NODE_HOST=worker-03.example.com   # SSH-reachable address of the same host

# 1. Check the node's conditions
kubectl describe node "$NODE"

# 2. Check the kubelet's logs
ssh "$NODE_HOST" "journalctl -u kubelet -n 200"

# 3. Check the runtime's logs
ssh "$NODE_HOST" "journalctl -u containerd -n 200"

# 4. Check the kernel's logs
ssh "$NODE_HOST" "dmesg | tail -200"

# 5. Check the hardware
ssh "$NODE_HOST" "sensors 2>/dev/null || echo 'no sensors'"
ssh "$NODE_HOST" "smartctl -a /dev/sda 2>/dev/null || echo 'no smartctl'"

# 6. Test the SSH connection
ssh "$NODE_HOST" "echo 'hello'"

The diagnostic is the node’s conditions, the kubelet’s logs, the runtime’s logs, the kernel’s logs, and the hardware.

The replacement

The replacement is the canonical recovery path:

# Substitute your own values before running:
NODE=worker-03                       # Node object being replaced
NEW_NODE_HOST=worker-04.example.com  # SSH-reachable address of the replacement host

# 1. Cordon the node
kubectl cordon "$NODE"

# 2. Drain the node
kubectl drain "$NODE" --ignore-daemonsets --delete-emptydir-data

# 3. Delete the node
kubectl delete node "$NODE"

# 4. Re-provision the node (out-of-band)
# - Re-image the node
# - Install the containerd
# - Install the kubelet, kubeadm, kubectl

# 5. Generate the join command on a control-plane node. It prints the
#    API server endpoint, the bootstrap token and the CA cert hash.
kubeadm token create --print-join-command

# 6. Join the new node to the cluster, using the values step 5 printed:
API_SERVER=192.0.2.10
API_PORT=6443
JOIN_TOKEN=abcdef.0123456789abcdef
CA_CERT_HASH=sha256:56b65c61aafa58096802e3926e7d7a04b78df0d3f1cc3d425d4dbcb2be3582d1

ssh "$NEW_NODE_HOST" \
  "sudo kubeadm join ${API_SERVER}:${API_PORT} --token ${JOIN_TOKEN} --discovery-token-ca-cert-hash ${CA_CERT_HASH}"

# 7. Verify the node is Ready
kubectl get nodes

The replacement is the node’s recovery.

The cluster-api alternative

The cluster-api is the production-grade alternative to manual replacement. The cluster-api controller watches the MachineDeployment and replaces the failed Machine with a new one.

flowchart TD
    A[MachineDeployment] --> B[MachineSet]
    B --> C[Machine]
    C --> D{Failed?}
    D -->|Yes| E[Replace]
    D -->|No| F[Continue]

The cluster-api is the cluster’s auto-replacement.

The remediation

The remediation depends on the cause:

# Substitute your own value before running:
NODE=worker-03

# Option 1: Manual replacement
kubectl drain "$NODE" --ignore-daemonsets --delete-emptydir-data
kubectl delete node "$NODE"
# Re-provision and join

# Option 2: cluster-api replacement
# Cluster-api automatically replaces the failed Machine

The remediation is the node replacement.

Production discipline

A node replacement is the cluster’s hypothesis. The discipline is to walk the 11-step methodology applied to the node, identify the cause, apply the replacement. The node is the cluster’s worker; the replacement is the worker’s recovery.

• Distinguish repairable from replaceable. The diagnostic is the node’s health.
• Drain before delete. The drain is the eviction of the Pods.
• Re-provision and join. The re-provision is the host’s recovery.

Quiz