high riskcluster affecting~30 min

Runbook: Troubleshoot kubelet

1 · Prerequisites

Confirm every item is in place before any state change.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Capture the kubelet journal from the affected node: kubectl debug node/<node> -- chroot /host journalctl -u kubelet --since "30 min ago" --no-pager | tee /tmp/kubelet.log
· Capture the kubelet configuration: cat /var/lib/kubelet/config.yaml
· Capture the kubelet serving certificate: ls /var/lib/kubelet/pki/
· Confirm the API server is reachable from the node: curl -k https://<api-vip>:6443/healthz
· Confirm the kubelet port (10250) is reachable from the API server: nc -vz <node-ip> 10250
· Capture the container runtime state from the node: crictl info

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Read the kubelet journal end-to-end and identify the failing stage: registration, certificate, runtime, or heartbeats
2For registration failure: confirm /etc/kubernetes/bootstrap-kubelet.conf (if used) or the kubelet flags match the cluster
3For certificate failure: identify whether the serving cert expired, was rotated, or is mismatched
4For runtime failure: systemctl status containerd and crictl info show whether the runtime is healthy
5For heartbeat failure: confirm the node can reach the API server and the kubelet --node-ip is correct
6Apply the smallest fix: rotate the serving cert, fix the runtime, repair the network, or restart kubelet after capturing the journal
7Verify the kubelet can register and serve: kubectl get node <node> -o wide reports Ready

4 · Verification

Confirm the procedure actually fixed the problem.

✓kubectl get node <node> reports Ready with the kubelet version matching the cluster minor
✓kubectl debug node/<node> -- chroot /host crictl info reports RuntimeReady: true
✓kubectl get pods -A -o wide --field-selector spec.nodeName=<node> shows Pods Running on the node
✓kubectl get csr | grep <node> shows no Pending CSRs (registration is complete)
✓No x509 or Unauthorized errors in the kubelet journal since the fix
✓The kubelet serving certificate is valid for at least 30 days: openssl x509 -in /var/lib/kubelet/pki/kubelet.crt -noout -dates

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶If the fix required a kubelet restart and Pods were slow to come back, give the node 5 minutes; kubelet restart is not a node restart
↶If the serving cert rotation broke the kubelet authentication to the API server, restore the previous cert from a backup and restart kubelet
↶If the runtime fix broke Pod scheduling, restart the runtime and confirm CNI is healthy on the node
↶Capture the kubelet journal and config before any change that may lose them
↶If the fix made things worse, the kubelet can be re-installed by re-running the node bootstrap (see the cluster build runbook)

6 · Escalation

When the runbook isn't enough, contact:

· Kubelet logs show repeated x509: certificate is valid for ... not for <ip> errors: certificate SAN mismatch; regenerate the kubelet serving cert
· Kubelet cannot reach the API server after a load-balancer change: the API server VIP moved; escalate to platform ownership
· Kubelet fails to start with failed to load kubelet config: the YAML is invalid; restore from a known-good config and re-run kubelet
· Kubelet starts but every Pod is stuck in ContainerCreating: runtime or CNI issue, not kubelet; see kubernetes-rb-troubleshoot-cni
· Kubelet repeatedly OOMKilled: the node is under-provisioned for the workload; escalate to capacity planning

The kubelet is the per-node agent that registers the node and runs Pods. A kubelet failure looks like a NotReady node or a Pod stuck in ContainerCreating. The cause is in the kubelet journal.

1. Get the kubelet journal

Read-only / SafeGet the kubelet journal


kubectl debug node/<node> -it --image=registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7 \
-- chroot /host journalctl -u kubelet --since "30 min ago" --no-pager | tee /tmp/kubelet.log

# Or SSH
ssh <node> -- sudo journalctl -u kubelet --since "30 min ago" --no-pager | tee /tmp/kubelet.log

# Or BMC (if the node is unreachable)
# Capture the SOL log via ipmitool

2. Common kubelet error patterns

Journal pattern	Class	Action
`x509: certificate is valid for 10.0.0.1, not 10.0.0.50`	Cert SAN mismatch	Regenerate kubelet serving cert
`x509: certificate has expired or is not yet valid`	Clock skew	Fix NTP; check cert validity
`Failed to list API server tokens`	API server unreachable	Test connectivity
`Unable to register node`	Registration conflict	Confirm `--node-ip` and `--register-withtaints`
`failed to load kubelet config`	Config YAML invalid	Restore config from backup
`kubelet cgroup driver: "cgroupfs" is different from docker cgroup driver: "systemd"`	Cgroup driver mismatch	Set kubelet `--cgroup-driver=systemd` and restart
`runtime network not ready`	CNI not running	See `kubernetes-rb-troubleshoot-cni`
`pods "..." is forbidden: ... exceeded quota`	ResourceQuota	Adjust the Pod spec or quota

3. Registration failure

Read-only / SafeRegistration failure


cat /etc/kubernetes/bootstrap-kubelet.conf 2>/dev/null || echo "no bootstrap kubeconfig"
cat /var/lib/kubelet/kubeconfig 2>/dev/null | head

# From the API server
kubectl get nodes -o wide
kubectl get csr | head

If csr shows a Pending request for the node, approve it (if RBAC requires manual approval) or restart kubelet to re-trigger the bootstrap.

4. Certificate failure

Read-only / SafeCertificate failure


openssl x509 -in /var/lib/kubelet/pki/kubelet.crt -noout -dates -subject -ext subjectAltName

# Check the client certificate (used to talk to API server)
openssl x509 -in /var/lib/kubelet/pki/kubelet-client.crt -noout -dates -issuer

# Check the CA
openssl x509 -in /etc/kubernetes/pki/ca.crt -noout -dates -subject

A kubelet serving cert is valid for one year by default in kubeadm clusters. If it expired, see kubernetes-rb-renew-cluster-certs.

5. Runtime failure

Read-only / SafeRuntime failure


systemctl status containerd
crictl info | jq -r '.status.runtimeReady, .status.images | length'

# From the kubelet journal, the CRI errors
journalctl -u kubelet --since "10 min ago" | grep -iE 'cri|containerd|runc' | tail

If the runtime is unhealthy, kubelet cannot start Pods. Restart the runtime first.

6. Apply the smallest fix

Read-only / SafeApply the smallest fix


sudo systemctl restart kubelet
sudo journalctl -u kubelet --since "1 min ago" --no-pager | tail -20

# B. Approve the bootstrap CSR
kubectl certificate approve <csr-name>

# C. Regenerate the kubelet serving cert (kubeadm)
sudo kubeadm certs renew kubelet-serving
# Or for the client cert:
sudo kubeadm certs renew client

# D. Restart the runtime
sudo systemctl restart containerd

# E. Fix the cgroup driver (must match containerd)
sudo sed -i 's/cgroupfs/systemd/' /etc/containerd/config.toml
sudo systemctl restart containerd
sudo systemctl restart kubelet

# F. Repair the API server reachability
sudo ip route
sudo cat /etc/resolv.conf
curl -k https://<api-vip>:6443/healthz

7. Verify

Read-only / SafeVerify


kubectl get node <node> -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}'
# Expect: True

# Pods running on the node
kubectl get pods -A -o wide --field-selector spec.nodeName=<node> | head -10

# No errors in the kubelet journal since the fix
journalctl -u kubelet --since "5 min ago" | grep -iE 'error|fail' | tail

Common pitfalls

Symptom	Cause	Action
Kubelet starts then immediately exits	Config YAML invalid or runtime path wrong	Check the journal for the specific error
Kubelet OOMKilled on start	kubelet has been running for too long without a restart, or has a memory leak	Investigate; restart, do not just bump the limit
Pods stuck in `ContainerCreating`	Runtime or CNI issue, not kubelet	See `kubernetes-rb-troubleshoot-cni`
`403 Forbidden` from kubelet to API server	RBAC or cert issue	Check the kubelet client cert; check `system:nodes` ClusterRoleBinding
`dial tcp 10.96.0.1:443` timeouts in kubelet	Cluster DNS unreachable from the kubelet host	Fix the host’s `/etc/resolv.conf`

A kubelet failure is rarely a kubelet bug. It is configuration, certificate, runtime, or network. The runbook distinguishes them by reading the journal before restarting.