Container runtimes — runc, containerd, CRI and the kubelet boundary

Kubelet does not start containers. It asks a runtime to start containers, via the Container Runtime Interface (CRI). The runtime layer is where image pulls, layer unpacking, OCI runtime spec construction, and fork/exec happen. This lesson walks the stack from kubelet down to the kernel and identifies where each production failure mode lives.

The layered stack

A Kubernetes node has four logical layers between the kubelet process and the container process:

flowchart TB
    K[kubelet] -->|CRI gRPC| SH[CRI shim<br/>containerd CRI plugin / CRI-O]
    SH -->|OCI bundle| HL[High-level runtime<br/>containerd]
    HL -->|runc create/start| LL[Low-level runtime<br/>runc / crun]
    LL -->|clone / unshare / cgroup| KRN[Linux kernel<br/>namespaces + cgroups]

• kubelet — schedules the Pod onto the node (already done by the API server and scheduler), pulls the image via the runtime, applies resource limits via cgroups, reports status. Kubelet does not touch namespaces.
• CRI shim — translates kubelet’s CRI gRPC calls into high-level runtime operations. For containerd, this is the built-in CRI plugin; for CRI-O, it is the entire daemon.
• High-level runtime — manages the local content store, image pulls, snapshotter (layer composition), and produces an OCI runtime spec bundle for the low-level runtime.
• Low-level runtime — runc or crun. Receives an OCI bundle, performs clone(2) with the right namespace flags, writes the cgroup configuration, and execs the container’s entrypoint.

runc and crun

runc is the reference OCI low-level runtime, originally from Docker, now maintained by the OCI community. It is written in Go and produces a single container per invocation.

crun is an alternative written in C, with a smaller memory footprint and faster start times (typically 10-30% faster than runc for cold starts). It implements the same OCI runtime spec and is a drop-in replacement.

runc --version
# runc version 1.2.x
# spec: 1.2.0

crun --version
# crun version 1.x
# spec: 1.2.0

Production trade-off:

• runc is the default; well-tested, ubiquitous, supported by every CRI implementation.
• crun is faster and lighter; supported by containerd and CRI-O. Worth measuring on dense clusters (many short-lived Pods).

The choice is configured at the runtime level, not in the Pod spec. Kubelet asks the CRI shim “create a container with this OCI spec”; the shim chooses which low-level runtime to call.

containerd vs CRI-O

Both are valid high-level runtimes. Both implement CRI. They differ in operational surface:

In production, both are equivalent in capability. The choice is usually operational familiarity:

• containerd is the upstream Kubernetes default and what kubeadm installs. Larger community, more third-party tooling, same binary used by Docker (Docker Engine 23+ runs containerd underneath).
• CRI-O is Red Hat’s lighter-weight implementation, optimised for OpenShift and RHEL environments.

The wrong choice is rare. The right choice is the one your platform team can operate.

The snapshotter: how layers become a root filesystem

A container’s root filesystem is composed from image layers and any writable upper layer. The snapshotter is the component that manages this composition. Two are common in production:

flowchart LR
    subgraph OverlayFS["overlayfs snapshotter"]
        L0[Layer 0<br/>read-only] --> MNT[Mount]
        L1[Layer 1<br/>read-only] --> MNT
        L2[Layer 2<br/>read-only] --> MNT
        UPPER[Upper dir<br/>per-container writable] --> MNT
    end
    MNT --> C[Container rootfs]

• overlayfs — the default. Uses Linux OverlayFS to compose layers at mount time. Low disk overhead (layers are shared hard-links), low overhead per container. Requires kernel support (any kernel that ships with containerd 1.7+).
• native — copies layers into a per-container directory. Slower to start, much higher disk cost; useful for kernels without OverlayFS or for very small layers where copy is cheaper than mount.
• devmapper — block-level snapshots via the device-mapper. Used historically for SELinux-heavy environments; less common in 2026.
• btrfs / zfs — copy-on-write filesystems. Useful on nodes that already run btrfs; rare elsewhere.

The choice of snapshotter affects:

• Disk usage. OverlayFS shares lower layers across containers; a 1 GB image with 100 replicas uses ~1 GB, not 100 GB. The native snapshotter uses ~100 GB.
• Cold-start latency. OverlayFS mounts in milliseconds; native copies can take seconds per container.
• Kernel requirements. Some kernels (older CentOS, certain container-optimised distros) ship with OverlayFS disabled or broken.

CRI versions and shims

CRI is a gRPC API. As of Kubernetes 1.34, the supported CRI version is v1 (with v1alpha2 and v1beta1 removed in 1.26). The shim layer translates CRI into runtime operations:

• containerd CRI plugin — built into containerd; enabled by default when containerd is configured as the runtime for kubelet.
• CRI-O — its own daemon, dedicated to running Kubernetes Pods.
• Docker Engine shim — dockershim was removed in Kubernetes 1.24. Clusters running kubelet 1.34 must use a CRI-conformant runtime (containerd or CRI-O).

When kubelet starts, it opens a gRPC connection to the runtime’s CRI socket. The socket is usually /run/containerd/containerd.sock or /var/run/crio/crio.sock. Kubelet logs the connection at startup; a kubelet that cannot reach the socket will not start any Pods.

cat /var/lib/kubelet/kubeadm-flags.env
# KUBELET_KUBEADM_ARGS="--container-runtime-endpoint=unix:///run/containerd/containerd.sock ..."

Containerd’s CLI tools

containerd ships three CLI tools, and the choice of which to use trips up many operators:

• ctr — direct containerd client. Talks to containerd’s core API, not CRI. Used for image pulls and snapshot manipulation outside the kubelet path. Not for inspecting Kubernetes Pods.
• crictl — CRI client. Speaks the same protocol kubelet uses. This is the tool for inspecting Pods, containers, and images as kubelet sees them.
• nerdctl — Docker-compatible CLI for containerd. Useful for running containers locally on a node outside Kubernetes, but not the tool for kubelet-side debugging.

Production troubleshooting always uses crictl:

crictl ps                       # list running containers (runtime's view)
crictl pods                     # list Pods (sandbox view)
crictl images                   # list images in the cache

# Take a container ID from the `crictl ps` output above:
CONTAINER_ID=9c2f4b8e1a7d3

crictl inspect "$CONTAINER_ID"   # full runtime spec for a container
crictl logs "$CONTAINER_ID"      # container stdout/stderr
crictl exec -it "$CONTAINER_ID" sh

CRI failures that show up as Pod symptoms

Pod status messages like ContainerCreating, CrashLoopBackOff, RunContainerError, CreateContainerError originate from CRI errors. The CRI shim reports them back to kubelet, kubelet emits a Pod event, and the Pod remains stuck until the underlying condition resolves.

Common CRI-layer failures:

• Image pull failure — registry unreachable, auth failure, missing manifest for the node architecture. Diagnose with crictl pull <image>.
• RunC OOM — the low-level runtime failed to allocate memory for the container process. Often correlated with the node being under memory pressure.
• Cannot join sandbox — the container process cannot setns into the sandbox’s namespaces. The most common cause is that the sandbox (pause) container is missing or died.
• Missing device — the runtime cannot apply a device configuration from the Pod spec (e.g., a CSI volume mount failed).
• AppArmor / seccomp / capability denial — the runtime refuses to start a container that violates the Pod’s security context. The Pod event names the policy that denied.

Runtime configuration the operator must own

Production clusters pin and document:

• Runtime version — both containerd and runc have shipped CVEs. Pin to a known-good version, follow the upstream CVE feed, and upgrade in the maintenance window.
• Storage driver / snapshotter — set explicitly in /etc/containerd/config.toml ([plugins."io.containerd.snapshotter.v1"]). Default is overlayfs.
• Registry mirrors and credentials — configured in containerd’s hosts.toml for each registry. Used for air-gapped clusters and for proxying through a private registry.
• Default ulimits and seccomp — the runtime applies defaults if the Pod spec does not specify. Audit these.
• Log driver and rotation — default is json-file with rotation at 10 MB / 3 files. Confirm disk headroom and monitoring.
• Sandbox image — registry.k8s.io/pause:3.x is the default. Confirm it is pullable from every node.

# /etc/containerd/config.toml — production-relevant excerpt
version = 2

[plugins."io.containerd.snapshotter.v1.overlayfs"]
  no_sync = false

[plugins."io.containerd.grpc.v1.cri"]
  sandbox_image = "registry.k8s.io/pause:3.10"
  max_container_log_line_size = 16384

[plugins."io.containerd.grpc.v1.cri".containerd]
  disable_snapshot_annotations = true

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
    runtime_type = "io.containerd.runc.v2"
    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
      SystemdCgroup = true
      BinaryName = "/usr/local/sbin/runc"

Cross-course references

• The Docker course parts XXVII-Docker-Install and XXXIX-Docker-Resources cover the runtime and resource controls that Kubernetes inherits through CRI.
• The Linux course part LXXVIII-Linux-Containers covers the kernel primitives (namespaces, cgroups, capabilities, seccomp) that the runtime composes.
• The Docker course part LVI-Docker-Internals covers the containerd/runc split from Docker’s perspective — the same layering Kubernetes uses.
• The Observability course part IX-Observability-Exporters covers the metrics that surface runtime cache size, container count, and image pull times.

Quiz

Production discipline

• Pin runtime versions (containerd, runc) and treat them as a deployable artifact with its own CVE feed.
• Set the cgroup driver explicitly (systemd in 2026) and ensure kubelet, containerd, and the kernel all agree.
• Use crictl for runtime-level debugging on kubelet nodes, never docker.
• Configure imagePullPolicy and registry credentials at the runtime layer for nodes that need cluster-wide image sources beyond the Pod’s imagePullSecrets.
• Monitor runtime disk usage (container_image_size_bytes, container_fs_inodes_free) and alert before the kubelet image GC triggers.