kubelet logs and auth — the node's voice

When a node goes NotReady while the kubelet is plainly running, the answer is usually in the first twenty lines of journalctl -u kubelet, and it is usually authentication. The kubelet’s client certificate is good for a year and renews itself by submitting a CSR for kube-controller-manager to approve; when that loop breaks, the certificate expires and every call to the API server comes back 401. Note that kubeadm certs renew all does not cover it — the kubelet’s certificate is not one of the control plane’s — so recovering the node means bootstrapping it again.

The kubelet’s logs

The kubelet’s logs are the source of truth for the node’s health. The logs are in /var/log/kubelet.log or via journalctl -u kubelet.

# View the kubelet's logs
journalctl -u kubelet -n 200

# Filter by severity
journalctl -u kubelet -p err -n 100

# Filter by time
journalctl -u kubelet --since "1 hour ago"

A real kubelet’s log:

Aug 16 04:23:01 node-03 kubelet[1234]: I0816 04:23:01.123456    1234 server.go:440] "Kubelet: syncing loop"
Aug 16 04:23:01 node-03 kubelet[1234]: E0816 04:23:01.234567    1234 kubelet.go:1341] "Image garbage collection failed" err="rpc error: code = Unavailable desc = connection error"
Aug 16 04:23:02 node-03 kubelet[1234]: E0816 04:23:02.345678    1234 kubelet.go:2231] "Failed to start ContainerManager" err="failed to initialize cgroup driver"

The logs tell the operator what the kubelet is doing.

The kubelet’s authentication

The kubelet authenticates to the API server using a client certificate. The certificate is at /var/lib/kubelet/pki/kubelet-client-current.pem.

# Check the kubelet's certificate
ls -la /var/lib/kubelet/pki/

# Check the certificate's expiration
openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -dates

A real certificate:

notBefore=Aug 16 03:00:00 2026 GMT
notAfter=Aug 16 04:00:00 2026 GMT

If the certificate is expired, the kubelet cannot authenticate to the API server.

The diagnostic

The canonical diagnostic:

# 1. Check the kubelet's logs
journalctl -u kubelet -n 200

# 2. Check the kubelet's certificate
ls -la /var/lib/kubelet/pki/
openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -dates

# 3. Check the kubelet's configuration
cat /var/lib/kubelet/config.yaml

# 4. Check the kubelet's flags
systemctl show kubelet --property=ExecStart

# 5. Check the API server's logs
kubectl logs -n kube-system -l component=kube-apiserver --tail=200

The diagnostic is the kubelet’s logs, the certificate, the configuration, and the API server’s logs.

Common failures

• Certificate expired. The kubelet’s certificate is expired. The remediation is to renew the certificate.
• Token expired. The kubelet’s bootstrap token is expired. The remediation is to regenerate the token.
• RBAC failure. The kubelet’s system:node ClusterRole is missing. The remediation is to restore the RBAC.
• API server unreachable. The kubelet cannot reach the API server. The remediation is to fix the network.

flowchart TD
    A[kubelet failing] --> B{Certificate valid?}
    B -->|No| C[Renew the certificate]
    B -->|Yes| D{Token valid?}
    D -->|No| E[Regenerate the token]
    D---|Yes| F{RBAC OK?}
    F -->|No| G[Restore the RBAC]
    F -->|Yes| H{API server reachable?}
    H -->|No| I[Fix the network]
    H -->|Yes| J[Unknown]

The remediation

The remediation depends on the cause:

# Option 1: Restart the kubelet
systemctl restart kubelet

# Option 2: Renew the certificate
kubeadm certs renew kubelet-client

# Option 3: Rotate the bootstrap token
kubeadm token create --print-join-command

# Option 4: Restore the RBAC
kubectl apply -f https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/gce/gci/configure-helper.sh

The remediation is the kubelet recovery.

Production discipline

A kubelet failure is the cluster’s hypothesis. The discipline is to walk the 11-step methodology applied to the kubelet, identify the cause, apply the remediation. The kubelet is the node’s voice; the remediation is the kubelet recovery.

• Check the kubelet’s logs. The logs are the node’s voice.
• Check the certificate. The certificate is the kubelet’s authentication.

Quiz