containerd and shims — the supervisor layer

containerd is the container supervisor. The shim is its parent-of- container process. This lesson explains why.

The shim is easy to dismiss as plumbing. It is a small process, one per container, that appears to do nothing. What it actually does is hold two pieces of state that no other component can hold, and both of them are load-bearing for the daemon-restart story that makes Docker operable in production.

Why the shim exists

sequenceDiagram
  participant D as dockerd
  participant C as containerd
  participant S as containerd-shim-runc-v2
  participant R as runc
  participant K as kernel
  D->>C: gRPC CreateTask
  C->>S: exec shim binary, hand it the bundle path
  S-->>C: listening socket address
  C->>S: ttrpc Create
  S->>R: runc create - namespaces, cgroups, rootfs, no user process yet
  C->>S: ttrpc Start
  S->>R: runc start
  R->>K: execve the entrypoint
  Note over R: runc exits
  Note over S: shim is now the parent of the container process
  Note over S: shim reparents itself to init

The shim’s two jobs, stated precisely:

1. It is the parent that reaps. The container’s PID 1 has to be a child of something that calls wait4(2), or it becomes a zombie on exit. runc is gone by then. If the parent were containerd, containerd could never restart. The shim volunteers to be that parent and nothing else.
2. It holds the exit status. When the container process dies, the kernel gives the status to its parent — the shim — and nobody else can have it. The shim caches it and serves it to containerd whenever containerd next asks, which may be after a containerd restart. Without this, a container that exited while containerd was down would have an unknowable exit code.

The shim then re-parents itself to init (PID 1 on the host), so that neither dockerd nor containerd is in the container’s ancestry. Kill either daemon and nothing in the tree above the container changes.

Inspecting containerd directly

Everything Docker creates lives in containerd’s moby namespace. This is the view that still works when dockerd does not.

# Containers Docker has defined
sudo ctr -n moby containers list

# Tasks: the ones that are actually running, with host PIDs
sudo ctr -n moby tasks list

# Full metadata for one, including the OCI spec
CID=$(docker inspect --format '{{.Id}}' web)
sudo ctr -n moby containers info "$CID" | head -40

# The shims themselves
ps -eo pid,ppid,etimes,rss,args | grep 'containerd-shim' | grep -v grep

# Which shim belongs to which container: the id is in its argv
pgrep -a containerd-shim-runc-v2 | grep "$(echo "$CID" | cut -c1-12)"

Illustrative ctr -n moby tasks list:

TASK                                                                PID      STATUS
3f2a9c1e8b74d05f6a1c2e9d8b7a6f5e4d3c2b1a0f9e8d7c6b5a4938271605f4    14822    RUNNING
9c1e8b74d05f6a1c2e9d8b7a6f5e4d3c2b1a0f9e8d7c6b5a4938271605f43f2a    15104    RUNNING

ctr is a debugging client, not a management tool — it deliberately has no safety rails and it does not know about anything Docker layers on top (names, restart policies, log drivers). Use it to observe. Using it to start or stop things behind dockerd’s back leaves the daemon’s view and containerd’s view disagreeing, which is a worse problem than the one you were solving.

Reading the shim’s own state on disk

# The runtime v2 task state root for Docker's namespace
sudo ls -1 /run/containerd/io.containerd.runtime.v2.task/moby/ | head

CID=$(docker inspect --format '{{.Id}}' web)
sudo ls -la /run/containerd/io.containerd.runtime.v2.task/moby/"$CID"/

# The OCI runtime spec runc was handed for this container
sudo python3 -m json.tool \
/run/containerd/io.containerd.runtime.v2.task/moby/"$CID"/config.json | head -60

That config.json is the single most informative file in the whole stack. It is the OCI runtime specification for the container as runc received it: every namespace, every mount with its options, the complete capability sets, the seccomp profile, the cgroup path and limits, the rlimits, the AppArmor profile name, and the exact process.args that were exec’d.

When docker inspect and observed behaviour disagree — a mount that is read-only when you did not ask for it, a capability you thought you dropped, an environment variable that is not what the Compose file says — config.json is the authority. It is what the runtime was actually told.

Diagnosing shim-level failures

# One shim per running container is the expected ratio
running=$(docker ps -q | wc -l)
shims=$(pgrep -c -f containerd-shim-runc-v2 || echo 0)
echo "running containers: $running   shims: $shims"

# Shims with no corresponding running container are the interesting ones
for pid in $(pgrep -f containerd-shim-runc-v2); do
id=$(tr '\0' ' ' < /proc/"$pid"/cmdline | grep -o '[0-9a-f]\{64\}' | head -1)
short=$(echo "$id" | cut -c1-12)
if ! docker ps -q --no-trunc | grep -q "$id"; then
  echo "ORPHAN shim pid=$pid container=$short"
fi
done

# Resource usage: a shim should be a few MB and near-zero CPU
ps -eo pid,pcpu,rss,etimes,comm -p "$(pgrep -d, -f containerd-shim-runc-v2)"

# containerd's own log, which is where shim errors land
sudo journalctl -u containerd.service --since '30 min ago' --no-pager | tail -40

A shim consuming meaningful CPU or hundreds of megabytes of RSS is abnormal and usually means it is relaying a very high volume of container output — the shim sits in the path of the container’s stdout and stderr, and a container logging in a tight loop makes its shim work hard. That is a logging problem presenting as a shim problem.

Note that shim errors go to containerd’s journal, not dockerd’s. This catches people out: journalctl -u docker.service is the reflex and it will not have them.

Verification

#!/usr/bin/env bash
set -euo pipefail

running=$(docker ps -q | wc -l)
shims=$(pgrep -c -f containerd-shim-runc-v2 || echo 0)

echo "containers=$running shims=$shims"
[ "$running" -eq "$shims" ] \
|| { echo "FAIL: shim count does not match container count" >&2; exit 1; }

# Every running container has a task containerd knows about
for id in $(docker ps -q --no-trunc); do
sudo ctr -n moby tasks list | grep -q "$id" \
  || { echo "FAIL: no containerd task for $id" >&2; exit 1; }
done

# Every shim is parented to init, not to a daemon
for pid in $(pgrep -f containerd-shim-runc-v2); do
ppid=$(ps -o ppid= -p "$pid" | tr -d ' ')
[ "$ppid" = '1' ] \
  || { echo "FAIL: shim $pid has parent $ppid, expected 1" >&2; exit 1; }
done

echo OK

The last check is the one that proves the property this whole lesson is about. If a shim’s parent is not init, the re-parenting did not happen and a daemon restart will take that container with it.

Knowledge check