high riskcluster affecting~30 min

Runbook: Troubleshoot a NotReady Node

1 · Prerequisites

Confirm every item is in place before any state change.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Capture the node status: kubectl get node <node> -o yaml | tee /tmp/node.yaml
· Capture the node conditions: kubectl describe node <node> | sed -n "/Conditions:/,/Allocated resources:/p"
· Confirm the API server can list all nodes: kubectl get nodes -o wide (a NotReady node still appears here)
· Capture the kubelet logs from the affected node: ssh <node> sudo journalctl -u kubelet -n 200 --no-pager, or kubectl debug node/<node> -it --image=busybox when SSH is unavailable
· Capture the node network reachability: ping -c2 <node-ip> and nc -vz <node-ip> 10250 (kubelet port)
· Confirm any Pods on the node are accounted for (so a drain does not surprise): kubectl get pods -A -o wide --field-selector spec.nodeName=<node>

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Classify the failure from the node conditions: NotReady vs Unknown vs pressure
2For Ready=False: kubelet is reporting unhealthy; read the kubelet journal on the node
3For Ready=Unknown: API server lost contact with the node; the node may be up but unreachable; read kubelet journal if reachable
4For pressure conditions: MemoryPressure, DiskPressure, PIDPressure; identify the cause from the journal and metrics
5For network unreachable: the node may be partitioned; check the switch / NIC / firewall before declaring the node dead
6For runtime failure: check the container runtime (containerd, CRI-O) on the node
7Apply the smallest fix: restart kubelet, free disk space, restart the runtime, repair the network, recover the node from BMC
8Cordon the node to stop new Pods landing on it: kubectl cordon <node>
9After the node is healthy, uncordon: kubectl uncordon <node>
10If Pods need to move, drain: kubectl drain <node> --ignore-daemonsets --delete-emptydir-data (see kubernetes-rb-drain-production-node)

4 · Verification

Confirm the procedure actually fixed the problem.

✓kubectl get node <node> reports Ready with all pressure conditions False
✓kubectl get pods -A -o wide --field-selector spec.nodeName=<node> shows Pods that should be there, Running
✓kubectl debug node/<node> -- cat /var/log/kubelet.log (or equivalent) reports kubelet healthy
✓From another node, nc -vz <node-ip> 10250 returns succeeded
✓No new Warning events on the node in the last 5 minutes
✓kubectl top node <node> reports CPU and memory within capacity

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶If the fix involved restarting kubelet and Pods were slow to come back, give the node 5 minutes before declaring it failed; kubelet restart is not a node restart
↶If the cordoned node needs to remain out of service while a deeper investigation runs, leave it cordoned
↶If a Pod on the node was killed by a fix and the workload cannot tolerate it, do not restart the node; re-create the Pod on a different node
↶Capture the kubelet journal and the node condition history before any reboot that may lose them
↶If the fix involved restoring network and the node needs a reboot, schedule the reboot in the next maintenance window

6 · Escalation

When the runbook isn't enough, contact:

· Node remains Ready=Unknown after kubelet restart: the API server cannot reach the kubelet port 10250; escalate to network/platform
· DiskPressure persists after cleanup: the node is too small for the workload; escalate to capacity planning
· MemoryPressure persists after Pod eviction: the node has too little memory for current demand; escalate to capacity planning
· Kubelet logs show repeated authentication errors to the API server: certificate rotation issue; see kubernetes-rb-renew-cluster-certs
· Node unreachable from both data and management networks: BMC/IPMI access is the next step; escalate to hardware/datacenter ownership

A NotReady node has either lost contact with the API server (Unknown) or is reporting unhealthy (NotReady with a reason). The cause is on the node itself; the API server is a messenger.

1. Read the conditions

Read-only / SafeRead the conditions


kubectl get node <node> -o jsonpath='{.status.conditions}' | jq

Condition	Value	Meaning
`Ready`	`True`	Node is healthy
`Ready`	`False`	Node reports unhealthy (kubelet can talk to API server)
`Ready`	`Unknown`	API server has not heard from the node in `node-monitor-grace-period` (default 40s)
`MemoryPressure`	`True`	Available memory < `eviction-hard` threshold
`DiskPressure`	`True`	Available disk < `eviction-hard` threshold
`PIDPressure`	`True`	PID availability < `eviction-hard` threshold

2. Get to the kubelet journal

The kubelet journal is on the node itself. Use BMC/IPMI if the node is unreachable; otherwise kubectl debug or direct SSH.

Read-only / SafeGet to the kubelet journal


kubectl debug node/<node> -it --image=registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7 -- chroot /host journalctl -u kubelet --since "30 min ago" --no-pager | tail -80

# Or SSH (if reachable)
ssh <node> -- sudo journalctl -u kubelet --since "30 min ago" --no-pager | tail -80

# Or BMC (if the node is unreachable from the network)
ipmitool -I lanplus -H <bmc-ip> -U admin -f /etc/k8s/bmc-pw sol activate

3. Classify the failure

Class	Evidence	Where to look next
Network partition	API server reports `Unknown`; node is up on BMC	NIC, switch, firewall, kubelet port 10250
Kubelet crash	kubelet logs show panic, OOM, or exit	Restart kubelet, capture crash log
Runtime failure	kubelet logs show CRI errors	`systemctl status containerd` or `crictl info`
Disk full	`DiskPressure=True`, kubelet logs show `no space left`	`df -h`, clean up
Memory pressure	`MemoryPressure=True`, kubelet logs show `evict`	`free -h`, identify the consuming Pod
Cert/auth failure	kubelet logs show `x509`, `Unauthorized`	See `kubernetes-rb-renew-cluster-certs`

4. Cordon and isolate

Read-only / SafeCordon and isolate


kubectl cordon <node>
kubectl get node <node> -o jsonpath='{.spec.unschedulable}'
# Expect: true

A cordoned node still runs its existing Pods. To move them off, see kubernetes-rb-drain-production-node.

5. Apply the fix

Read-only / SafeApply the fix


ssh <node> -- sudo systemctl restart kubelet
ssh <node> -- sudo journalctl -u kubelet --since "1 min ago" --no-pager | tail -20

# B. Free disk
ssh <node> -- sudo journalctl --vacuum-size=200M
ssh <node> -- sudo docker image prune -af || sudo crictl rmi --prune
ssh <node> -- sudo find /var/lib/containerd -size +1G -mtime +7 -delete || true

# C. Restart container runtime
ssh <node> -- sudo systemctl restart containerd
ssh <node> -- sudo crictl info | jq -r '.status.runtimeReady'

# D. Repair network (after BMC check)
ssh <node> -- sudo ip link set <iface> down && sudo ip link set <iface> up
ssh <node> -- sudo systemctl restart NetworkManager || sudo systemctl restart systemd-networkd

# E. Recover after partition (after the node is reachable)
# Once reachable, kubelet will re-register itself within lease-renew-interval seconds

6. Uncordon

Read-only / SafeUncordon


kubectl get node <node> -o jsonpath='{.status.conditions}' | jq
# Expect: Ready=True, all pressures False

kubectl uncordon <node>
kubectl get node <node> -o jsonpath='{.spec.unschedulable}'
# Expect: empty or false

7. Move workloads back (if drained)

If the node was drained, workloads will not return automatically. Either let the scheduler place them based on capacity, or re-deploy them.

Read-only / SafeMove workloads back (if drained)


kubectl scale deploy/<name> --replicas=<n> -n <ns>

# Watch
kubectl rollout status deploy/<name> -n <ns> --timeout=10m
kubectl get pods -n <ns> -o wide --field-selector spec.nodeName=<node>

Common pitfalls

Symptom	Cause	Action
Node recovered then NotReady again	Underlying fault (disk full, kernel panic) still present	Find the fault, do not retry the restart
`Unknown` for 30+ seconds then `NotReady`	Watchdog fired; node was unhealthy beyond `node-monitor-grace-period`	Read kubelet journal from the moment of recovery
`DiskPressure` recurs after cleanup	Workload generates more data than expected	Investigate the workload; do not delete more logs
Node is `Ready` but no Pods scheduled	Pods do not tolerate the node’s taints	Inspect taints; uncordon alone does not admit workloads
Recovery required a reboot	Reboot is destructive; treat as a maintenance window, not a hotfix

A NotReady node is not a workload incident; it is a node incident. The fix is on the node. The runbook confirms the cause before restarting anything, and never assumes “restart kubelet” fixes the problem.