containerd and runtime recovery — the container engine

When containerd stops answering, containers that are already running keep running, but the kubelet can no longer create, inspect, or delete anything — so the node goes NotReady while its workload is still serving traffic. Nothing in kubectl explains that: the evidence is the containerd journal, the service status, and whether crictl can reach the socket at all. This lesson covers working that path on the node and separating a dead runtime from a full disk, an unreachable registry, and a configuration that will not parse.

The containerd runtime

The containerd runtime is the cluster’s default container engine. The runtime is configured via /etc/containerd/config.toml and managed by systemctl.

flowchart LR
    A[kubelet] --> B[CRI]
    B --> C[containerd]
    C --> D[runc]
    D --> E[Container]

The runtime is the node’s container engine.

The diagnostic

The canonical diagnostic:

# 1. Check the runtime's logs
journalctl -u containerd -n 200

# 2. Check the runtime's status
systemctl status containerd

# 3. Check the runtime's images
crictl images

# 4. Check the runtime's containers
crictl ps

# 5. Check the runtime's pods
crictl pods

# 6. Check the kubelet's runtime logs
journalctl -u kubelet | grep -i containerd

The diagnostic is the runtime’s logs, the runtime’s status, and the kubelet’s runtime logs.

Common failures

• Runtime is failing. The containerd is in CrashLoopBackOff. The remediation is to restart the containerd.
• Image registry unreachable. The containerd cannot pull the image. The remediation is to fix the registry.
• Disk full. The containerd’s storage is full. The remediation is to clean up the storage.
• Configuration error. The containerd’s config is invalid. The remediation is to fix the config.

flowchart TD
    A[Runtime failing] --> B{containerd running?}
    B -->|No| C[Restart containerd]
    B -->|Yes| D{Images pullable?}
    D -->|No| E[Fix the registry]
    D---|Yes| F{Disk full?}
    F -->|Yes| G[Clean up the disk]
    F -->|No| H{Config valid?}
    H -->|No| I[Fix the config]
    H -->|Yes| J[Unknown]

The remediation

The remediation depends on the cause:

# Option 1: Restart the containerd
systemctl restart containerd

# Option 2: Clean up unused images
crictl rmi --prune

# Option 3: Clean up unused containers
crictl rm --prune

# Option 4: Fix the config
# Edit /etc/containerd/config.toml
systemctl restart containerd

The remediation is the runtime recovery.

The kubelet-runtime integration

The kubelet and the runtime communicate via the CRI (Container Runtime Interface). The CRI is the protocol; the runtime is the implementation.

# Check the kubelet's runtime configuration
cat /var/lib/kubelet/config.yaml | grep -A5 runtime

# Output:
# runtimeRequestTimeout: 2m
# containerRuntimeEndpoint: unix:///run/containerd/containerd.sock
# containerRuntime: containerd

The runtime endpoint is the kubelet’s connection to the runtime.

The diagnostic for a runtime failure

# 1. Read the containerd journal
journalctl -u containerd -n 200

# 2. Test the runtime
crictl info

# 3. Check the kubelet's CRI connection
crictl --runtime-endpoint unix:///run/containerd/containerd.sock ps

# 4. Check the runtime's storage
du -sh /var/lib/containerd/

The diagnostic is the runtime’s logs, the runtime’s status, and the kubelet’s CRI connection.

Production discipline

A runtime failure is the cluster’s hypothesis. The discipline is to walk the 11-step methodology applied to the runtime, identify the cause, apply the remediation. The runtime is the node’s container engine; the remediation is the runtime recovery.

• Check the runtime’s logs. The logs are the runtime’s voice.
• Check the runtime’s status. The status is the runtime’s health.
• Restart the runtime. The restart is the runtime’s recovery.

Quiz