Skip to main content
RunBook Academy

KubernetesXXX · Container Runtime and CRIContainer runtime

containerd — the production CRI implementation

Advanced⏱ ~18 minkubectl

What you'll learn

  • Trace containerd's architecture and primary subsystems
  • Identify the four namespaces and the snapshotter
  • Configure the runtime modes (runc, kata)
  • Diagnose containerd's failure modes from the logs

Prerequisites

Verified against Kubernetes 1.34.x · kubeadm 1.34.x · kubectl 1.34.x · etcd 3.6.x · CoreDNS 1.11.x · containerd 1.7.x / 2.x · 2026-08-16

Not yet marked complete on this device.

containerd is the cluster’s default CRI implementation. It is a CNCF graduated project; the runtime is a single binary (containerd) that runs as a systemd service. The kubelet connects to containerd over a Unix socket. This lesson walks containerd’s architecture, the namespaces, the snapshotter, and the operational patterns.

containerd’s architecture

containerd is a multi-subsystem binary. The primary subsystems:

flowchart LR
    A[containerd] --> B[API server<br/>CRI / containerd]
    B --> C[Supervisor]
    C --> D[Execution service]
    C --> E[Local store]
    C --> F[Snapshotter]
    C --> G[Network]
    C --> H[Metadata]
    D -->|spawns| I[runc / kata]
    F -->|overlayfs| J[Image layers]
    H -->|boltdb| K[Container database]
  • API server: the gRPC server that implements the CRI and the containerd APIs. The kubelet calls the CRI; the containerd CLI (ctr) calls the containerd API.
  • Supervisor: the lifecycle manager that creates and deletes containers.
  • Execution service: the runtime that spawns the container processes (via runc or kata).
  • Local store: the image blob store.
  • Snapshotter: the filesystem layer manager (overlayfs, btrfs, devicemapper).
  • Network: the CNI integration.
  • Metadata: the bolt database that stores the container’s metadata.

The subsystems run as goroutines in the single containerd process. The process is supervised by systemd.

The containerd namespaces

containerd uses namespaces to isolate state. The default namespaces:

  • default: the namespace for containerd’s internal state.
  • k8s.io: the namespace for Kubernetes’ Pods and containers. The kubelet calls the CRI with the k8s.io namespace.

The kubelet’s CRI calls specify the Pod’s metadata.namespace. containerd translates the Pod’s namespace into the containerd namespace.

ctr -n k8s.io containers list

The command lists the containers in the k8s.io namespace. The operator uses ctr to debug containerd’s state.

The snapshotter

The snapshotter is containerd’s filesystem layer manager. The snapshotter creates the container’s filesystem from the image’s layers.

The supported snapshotters:

  • overlayfs: the default. Uses the kernel’s overlayfs to layer the image’s layers.
  • btrfs: uses btrfs subvolumes. Faster on btrfs filesystems.
  • devicemapper: uses the device-mapper. Suitable for some enterprise storage configurations.
  • native: uses simple directory copies. Slow but portable.

The snapshotter is configured in containerd’s configuration file (/etc/containerd/config.toml):

[plugins."io.containerd.snapshotter.v1.overlayfs"]
  no_sync = true

The default is overlayfs. The overlayfs snapshotter creates a snapshotted directory for each container; the container’s writes are stored in the snapshot’s upperdir.

The runtime modes

containerd supports multiple runtime modes for the container processes:

  • runc: the default. The Open Container Initiative (OCI) reference implementation. Runs the container as a regular process on the host.
  • kata: the runtime for hardware-isolated containers. Runs the container in a lightweight VM.
  • gVisor: a user-space kernel that runs the container in a sandbox. Slower but more secure.
  • runsc: gVisor’s runtime binary.

The runtime is selected per-Pod via the Pod’s spec.runtimeClassName. The RuntimeClass references the runtime.

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: kata
handler: kata

The kubelet reads the RuntimeClass and selects the appropriate runtime when calling the CRI.

The CRI plugin

The CRI plugin is containerd’s implementation of the CRI. The plugin is a separate binary that runs as a gRPC proxy between the kubelet and containerd.

flowchart LR
    A[kubelet] -->|gRPC| B[containerd CRI plugin]
    B -->|containerd API| C[containerd]
    C -->|spawns| D[runc / kata]

The CRI plugin was historically a separate binary (cri-containerd); in current versions it is built into containerd and runs as a built-in plugin. The CRI plugin implements the CRI’s gRPC server.

The CRI plugin’s configuration:

[plugins."io.containerd.grpc.v1.cri"]
  sandbox_image = "registry.k8s.io/pause:3.9"
  containerd_snapshotter = "overlayfs"
  disable_apparmor = false
  disable_cgroup = false
  disable_hugetlb_controller = true

The sandbox_image is the pause container’s image. The containerd_snapshotter is the snapshotter for the container’s filesystem.

The image cache

containerd’s image cache is the local store. The cache is configured under /var/lib/containerd:

ls /var/lib/containerd
io.containerd.content.v1.content/  # image blobs
io.containerd.snapshotter.v1.overlayfs/  # layer snapshots
io.containerd.metadata.v1.bolt/  # metadata database

The io.containerd.content.v1.content directory holds the image blobs. The io.containerd.snapshotter.v1.overlayfs directory holds the layer snapshots. The metadata database holds the container’s state.

The kubelet’s image GC uses the CRI’s ListImages and RemoveImage APIs to manage the cache. The default threshold is 85% of the filesystem.

The containerd’s logs

containerd’s logs are the primary diagnostic. The logs include the CRI requests, the snapshotter’s actions, and the runtime’s spawns.

journalctl -u containerd | tail -50

The typical log entries:

containerd\ntime="2026-08-16T10:00:00Z" level=info msg="Start container" id=...
containerd\ntime="2026-08-16T10:00:00Z" level=error msg="Failed to create container" id=...

The kubelet’s logs include the corresponding CRI requests. The operator should read both logs to diagnose a containerd failure.

containerd’s failure modes

The failure modes:

FailureSymptomRoot cause
containerd crashPods stuck in ContainerCreatingbinary crashed, OOMKill, configuration error
Image pull failureContainer’s ErrImagePullregistry unreachable, image not found, authentication failed
Snapshotter failureContainer’s CreateContainer failsfilesystem full, snapshotter misconfigured
Runtime failureContainer’s StartContainer failsrunc binary missing, kernel feature missing
CRI plugin failurekubelet logs failed to call CRICRI plugin stalled, gRPC backlog

The diagnostic:

systemctl status containerd
journalctl -u containerd -n 100
ctr -n k8s.io containers list

The systemctl status shows the process’s health. The journalctl shows the runtime’s logs. The ctr listing shows the containers in the k8s.io namespace.

The containerd configuration

The containerd configuration is in /etc/containerd/config.toml. The file is a TOML file that containerd reads on startup.

version = 2

[plugins."io.containerd.grpc.v1.cri"]
  sandbox_image = "registry.k8s.io/pause:3.9"
  containerd_snapshotter = "overlayfs"

[plugins."io.containerd.snapshotter.v1.overlayfs"]
  no_sync = true

The default configuration is /etc/containerd/config.toml; the containerd config default command generates the default configuration.

The containerd upgrade

The containerd upgrade is a node-level operation. The process:

  1. Drain the node.
  2. Stop the kubelet (so it stops calling the CRI).
  3. Stop containerd.
  4. Install the new containerd binary.
  5. Update the configuration.
  6. Start containerd.
  7. Verify the runtime is healthy.
  8. Start the kubelet.
  9. Uncordon the node.

The upgrade is disruptive to the node’s Pods. The drain ensures the Pods are evicted before the upgrade. The containerd upgrade is a maintenance operation.

Quiz

Knowledge check · 4 questions

  1. Q1. Which containerd namespace holds the containers a Kubernetes kubelet created?

  2. Q2. `ctr containers list` with no arguments shows the Pods running on a Kubernetes node.

  3. Q3. Recover a freshly repaved node whose containerd configuration was regenerated with upstream defaults in an air-gapped cluster.

    The cluster pulls exclusively from an internal mirror at `registry.internal:5000`. A node image rebuild replaced `/etc/containerd/config.toml` with the output of `containerd config default`. `node-22` joins and reports `Ready`, but every Pod scheduled to it stays in `ContainerCreating` with the event `Failed to create pod sandbox: failed to get sandbox image "registry.k8s.io/pause:3.9": failed to pull image`. Application images pull fine on the same node.

  4. Q4. Which file and configuration key set the pause image containerd uses for every Pod sandbox, and which containerd namespace holds the containers the kubelet created?

Passing score: 75%. Answers are checked in this browser.

Production discipline

  • containerd is the default runtime. The kubelet connects to containerd over a Unix socket. The production rule is to use containerd.
  • The snapshotter is the source of the container’s filesystem. The default is overlayfs. Other snapshotters may be appropriate for specific storage configurations.
  • The runtime mode is configurable per Pod. Use RuntimeClass to select the runtime. The default is runc; kata and gVisor are for hardware-isolated or secure workloads.
  • Audit the configuration at every node repave. A new node that joins the cluster with the wrong containerd configuration is a node that is failing silently. Validate the configuration at bootstrap.
  • Monitor the containerd’s metrics. The containerd_* metrics expose the runtime’s health. The operator should alert on the error rate and the resource usage.
  • The containerd upgrade is a maintenance operation. The node must be drained before the upgrade. The upgrade is not a rolling upgrade; the node is offline for the duration.