KubernetesXXX · Container Runtime and CRIContainer runtime
containerd — the production CRI implementation
What you'll learn
- Trace containerd's architecture and primary subsystems
- Identify the four namespaces and the snapshotter
- Configure the runtime modes (runc, kata)
- Diagnose containerd's failure modes from the logs
Prerequisites
Verified against Kubernetes 1.34.x · kubeadm 1.34.x · kubectl 1.34.x · etcd 3.6.x · CoreDNS 1.11.x · containerd 1.7.x / 2.x · 2026-08-16
containerd is the cluster’s default CRI implementation. It
is a CNCF graduated project; the runtime is a single
binary (containerd) that runs as a systemd service. The
kubelet connects to containerd over a Unix socket. This
lesson walks containerd’s architecture, the namespaces,
the snapshotter, and the operational patterns.
containerd’s architecture
containerd is a multi-subsystem binary. The primary subsystems:
flowchart LR
A[containerd] --> B[API server<br/>CRI / containerd]
B --> C[Supervisor]
C --> D[Execution service]
C --> E[Local store]
C --> F[Snapshotter]
C --> G[Network]
C --> H[Metadata]
D -->|spawns| I[runc / kata]
F -->|overlayfs| J[Image layers]
H -->|boltdb| K[Container database]
- API server: the gRPC server that implements the
CRI and the containerd APIs. The kubelet calls the
CRI; the containerd CLI (
ctr) calls the containerd API. - Supervisor: the lifecycle manager that creates and deletes containers.
- Execution service: the runtime that spawns the
container processes (via
runcorkata). - Local store: the image blob store.
- Snapshotter: the filesystem layer manager (overlayfs, btrfs, devicemapper).
- Network: the CNI integration.
- Metadata: the bolt database that stores the container’s metadata.
The subsystems run as goroutines in the single containerd process. The process is supervised by systemd.
The containerd namespaces
containerd uses namespaces to isolate state. The default namespaces:
default: the namespace for containerd’s internal state.k8s.io: the namespace for Kubernetes’ Pods and containers. The kubelet calls the CRI with thek8s.ionamespace.
The kubelet’s CRI calls specify the Pod’s
metadata.namespace. containerd translates the Pod’s
namespace into the containerd namespace.
ctr -n k8s.io containers list
The command lists the containers in the k8s.io
namespace. The operator uses ctr to debug
containerd’s state.
The snapshotter
The snapshotter is containerd’s filesystem layer manager. The snapshotter creates the container’s filesystem from the image’s layers.
The supported snapshotters:
- overlayfs: the default. Uses the kernel’s overlayfs to layer the image’s layers.
- btrfs: uses btrfs subvolumes. Faster on btrfs filesystems.
- devicemapper: uses the device-mapper. Suitable for some enterprise storage configurations.
- native: uses simple directory copies. Slow but portable.
The snapshotter is configured in containerd’s
configuration file (/etc/containerd/config.toml):
[plugins."io.containerd.snapshotter.v1.overlayfs"]
no_sync = true
The default is overlayfs. The overlayfs snapshotter creates a snapshotted directory for each container; the container’s writes are stored in the snapshot’s upperdir.
The runtime modes
containerd supports multiple runtime modes for the container processes:
- runc: the default. The Open Container Initiative (OCI) reference implementation. Runs the container as a regular process on the host.
- kata: the runtime for hardware-isolated containers. Runs the container in a lightweight VM.
- gVisor: a user-space kernel that runs the container in a sandbox. Slower but more secure.
- runsc: gVisor’s runtime binary.
The runtime is selected per-Pod via the Pod’s
spec.runtimeClassName. The RuntimeClass references the
runtime.
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
name: kata
handler: kata
The kubelet reads the RuntimeClass and selects the appropriate runtime when calling the CRI.
The CRI plugin
The CRI plugin is containerd’s implementation of the CRI. The plugin is a separate binary that runs as a gRPC proxy between the kubelet and containerd.
flowchart LR
A[kubelet] -->|gRPC| B[containerd CRI plugin]
B -->|containerd API| C[containerd]
C -->|spawns| D[runc / kata]
The CRI plugin was historically a separate binary
(cri-containerd); in current versions it is built
into containerd and runs as a built-in plugin. The
CRI plugin implements the CRI’s gRPC server.
The CRI plugin’s configuration:
[plugins."io.containerd.grpc.v1.cri"]
sandbox_image = "registry.k8s.io/pause:3.9"
containerd_snapshotter = "overlayfs"
disable_apparmor = false
disable_cgroup = false
disable_hugetlb_controller = true
The sandbox_image is the pause container’s image. The
containerd_snapshotter is the snapshotter for the
container’s filesystem.
The image cache
containerd’s image cache is the local store. The cache
is configured under /var/lib/containerd:
ls /var/lib/containerd
io.containerd.content.v1.content/ # image blobs
io.containerd.snapshotter.v1.overlayfs/ # layer snapshots
io.containerd.metadata.v1.bolt/ # metadata database
The io.containerd.content.v1.content directory holds the
image blobs. The io.containerd.snapshotter.v1.overlayfs
directory holds the layer snapshots. The metadata
database holds the container’s state.
The kubelet’s image GC uses the CRI’s ListImages and
RemoveImage APIs to manage the cache. The default
threshold is 85% of the filesystem.
The containerd’s logs
containerd’s logs are the primary diagnostic. The logs include the CRI requests, the snapshotter’s actions, and the runtime’s spawns.
journalctl -u containerd | tail -50
The typical log entries:
containerd\ntime="2026-08-16T10:00:00Z" level=info msg="Start container" id=...
containerd\ntime="2026-08-16T10:00:00Z" level=error msg="Failed to create container" id=...
The kubelet’s logs include the corresponding CRI requests. The operator should read both logs to diagnose a containerd failure.
containerd’s failure modes
The failure modes:
| Failure | Symptom | Root cause |
|---|---|---|
| containerd crash | Pods stuck in ContainerCreating | binary crashed, OOMKill, configuration error |
| Image pull failure | Container’s ErrImagePull | registry unreachable, image not found, authentication failed |
| Snapshotter failure | Container’s CreateContainer fails | filesystem full, snapshotter misconfigured |
| Runtime failure | Container’s StartContainer fails | runc binary missing, kernel feature missing |
| CRI plugin failure | kubelet logs failed to call CRI | CRI plugin stalled, gRPC backlog |
The diagnostic:
systemctl status containerd
journalctl -u containerd -n 100
ctr -n k8s.io containers list
The systemctl status shows the process’s health. The
journalctl shows the runtime’s logs. The ctr listing
shows the containers in the k8s.io namespace.
The containerd configuration
The containerd configuration is in
/etc/containerd/config.toml. The file is a TOML file
that containerd reads on startup.
version = 2
[plugins."io.containerd.grpc.v1.cri"]
sandbox_image = "registry.k8s.io/pause:3.9"
containerd_snapshotter = "overlayfs"
[plugins."io.containerd.snapshotter.v1.overlayfs"]
no_sync = true
The default configuration is /etc/containerd/config.toml;
the containerd config default command generates the
default configuration.
The containerd upgrade
The containerd upgrade is a node-level operation. The process:
- Drain the node.
- Stop the kubelet (so it stops calling the CRI).
- Stop containerd.
- Install the new containerd binary.
- Update the configuration.
- Start containerd.
- Verify the runtime is healthy.
- Start the kubelet.
- Uncordon the node.
The upgrade is disruptive to the node’s Pods. The drain ensures the Pods are evicted before the upgrade. The containerd upgrade is a maintenance operation.
Quiz
Knowledge check · 4 questions
Q1. Which containerd namespace holds the containers a Kubernetes kubelet created?
Q2. `ctr containers list` with no arguments shows the Pods running on a Kubernetes node.
Q3. Recover a freshly repaved node whose containerd configuration was regenerated with upstream defaults in an air-gapped cluster.
The cluster pulls exclusively from an internal mirror at `registry.internal:5000`. A node image rebuild replaced `/etc/containerd/config.toml` with the output of `containerd config default`. `node-22` joins and reports `Ready`, but every Pod scheduled to it stays in `ContainerCreating` with the event `Failed to create pod sandbox: failed to get sandbox image "registry.k8s.io/pause:3.9": failed to pull image`. Application images pull fine on the same node.
Q4. Which file and configuration key set the pause image containerd uses for every Pod sandbox, and which containerd namespace holds the containers the kubelet created?
Passing score: 75%. Answers are checked in this browser.
Production discipline
- containerd is the default runtime. The kubelet connects to containerd over a Unix socket. The production rule is to use containerd.
- The snapshotter is the source of the container’s filesystem. The default is overlayfs. Other snapshotters may be appropriate for specific storage configurations.
- The runtime mode is configurable per Pod. Use
RuntimeClassto select the runtime. The default is runc; kata and gVisor are for hardware-isolated or secure workloads. - Audit the configuration at every node repave. A new node that joins the cluster with the wrong containerd configuration is a node that is failing silently. Validate the configuration at bootstrap.
- Monitor the containerd’s metrics. The
containerd_*metrics expose the runtime’s health. The operator should alert on the error rate and the resource usage. - The containerd upgrade is a maintenance operation. The node must be drained before the upgrade. The upgrade is not a rolling upgrade; the node is offline for the duration.