runc and the OCI runtime

runc is the lowest layer of the Docker stack. It reads the OCI runtime spec and turns it into a Linux process.

The property that explains most of its behaviour: runc is not resident. It is invoked, it does its work, it execves your entrypoint over itself, and the process that was runc becomes your process. There is no runc left. ps on a host with fifty running containers shows zero runc processes, and that is correct, not a symptom.

Everything follows from that. runc errors are always start-time errors. If a container has been running for six hours and something breaks, runc is not involved and cannot be — it stopped existing six hours ago.

What runc does

flowchart TB
  Bundle["OCI bundle<br/>config.json + rootfs/"] --> Create["runc create"]
  Create --> NS["clone/unshare: new namespaces"]
  Create --> CG["write cgroup v2 limits"]
  Create --> FS["pivot_root into the rootfs"]
  Create --> Init["init process waits<br/>on a FIFO"]
  Init --> Start["runc start"]
  Start --> Sec["apply capabilities, seccomp, LSM label"]
  Sec --> Exec["execve the entrypoint"]
  Exec --> Gone["runc has exited"]

The OCI specification splits this into two commands and the split is not cosmetic. create “MUST create a new container” and apply “All of the properties configured in config.json except for process” — while explicitly requiring that “the user-specified program (from process) MUST NOT be run at this time”. start is the one that “MUST run the user-specified program”.

That two-phase design is what lets the shim get everything ready — namespaces, cgroups, mounts, network attachment — and then release the process at a moment of its choosing. It is also why a container can sit in created state indefinitely with a fully-built environment and no running program, which is a state you will see in docker inspect when a start fails.

The full ordering runc performs:

1. Read the OCI config. Capabilities, namespaces, cgroups, mounts, environment, rlimits.
2. Create the cgroup and write its limits. memory.max, cpu.max, pids.max. This happens before the process exists, so the limits apply to its very first allocation.
3. Create the namespaces. CLONE_NEWNS, CLONE_NEWPID, CLONE_NEWNET, CLONE_NEWUTS, CLONE_NEWIPC, and CLONE_NEWUSER when user namespaces are in use.
4. Set up the filesystem. Mount the overlay rootfs, then the mounts from config.json — /proc, /sys, /dev, tmpfs, and your bind mounts — inside the new mount namespace.
5. pivot_root into the container’s root so the host filesystem is no longer reachable through any path.
6. Drop capabilities to the bounding, permitted, effective and inheritable sets from the config.
7. Apply the LSM label — the AppArmor profile or SELinux context.
8. Load the seccomp filter. This is nearly last on purpose: the filter would block the syscalls runc itself needs for the earlier steps.
9. execve the entrypoint. runc is now your process.

Reading the OCI config

The bundle for a live container is on disk and is the authority on what the container was actually given — as distinct from what you asked for.

CONTAINER=web
CID=$(docker inspect --format '{{.Id}}' "$CONTAINER")
SPEC=/run/containerd/io.containerd.runtime.v2.task/moby/"$CID"/config.json

# The command that was actually exec'd
sudo python3 -c "import json,sys; d=json.load(open(sys.argv[1])); print(d['process']['args'])" "$SPEC"

# Capabilities the process really has
sudo python3 -c "import json,sys; d=json.load(open(sys.argv[1])); print(json.dumps(d['process']['capabilities'], indent=2))" "$SPEC"

# Namespaces: an entry with a 'path' is a SHARED namespace, not a new one
sudo python3 -c "import json,sys; d=json.load(open(sys.argv[1])); print(json.dumps(d['linux']['namespaces'], indent=2))" "$SPEC"

# Every mount with its real options
sudo python3 -c "import json,sys; d=json.load(open(sys.argv[1])); [print(m['destination'], m['type'], m.get('options')) for m in d['mounts']]" "$SPEC"

# The cgroup path, which is where the limits live
sudo python3 -c "import json,sys; d=json.load(open(sys.argv[1])); print(d['linux']['cgroupsPath'])" "$SPEC"

The namespaces query is the one that repays learning. Each entry has a type and optionally a path. No path means runc created a fresh namespace of that type. A path means the container was joined to an existing namespace — which is what --network container:other, --pid host and --ipc host produce.

So a container that “has its own network” but shows {"type": "network", "path": "/proc/1234/ns/net"} is sharing another process’s network stack, and every networking symptom you are chasing belongs to that other process. docker inspect shows this too, in HostConfig.NetworkMode, but the spec shows it unambiguously and shows it for every namespace type at once.

Running runc directly

Doing this once makes the layering concrete. Build a bundle and run it with no daemon anywhere in the picture.

WORK=/var/tmp/runc-demo
mkdir -p "$WORK"/rootfs
cd "$WORK"

# A root filesystem, exported from an image
docker export "$(docker create --name runcdemo alpine:3.20 sh)" \
| tar -C "$WORK"/rootfs -xf -
docker rm runcdemo

# A default OCI spec, then adjust it
runc spec

# Run it. This is the entire stack below Docker.
sudo runc run demo

# From another shell, while it is running:
#   sudo runc list
#   sudo runc state demo
#   sudo runc ps demo

# Clean up
cd / && sudo rm -rf "$WORK"

runc spec writes a default config.json you can read end to end in a few minutes — it is the smallest complete description of a container that exists. Comparing it against the one Docker generates shows exactly what Docker adds: the log path, the network namespace join, the seccomp profile, the hostname, the mounts.

While runc run demo is in the foreground, ps in another shell shows your sh and no runc parent above it beyond the invoking shell — because runc run is create plus start and the exec has already happened.

Common runc-level failures

Each of these arrives at start time and each has a distinct cause.

IMAGE=myorg/myapp:1.0.0

# 1. Architecture match
docker image inspect "$IMAGE" --format 'image={{.Os}}/{{.Architecture}}'
docker info --format 'host={{.OSType}}/{{.Architecture}}'

# 2. cgroup version the host is running
stat -fc %T /sys/fs/cgroup
# 'cgroup2fs' is v2 (unified); 'tmpfs' means v1 or hybrid

# 3. Which runtime and version is in play
docker info --format 'default-runtime={{.DefaultRuntime}}'
runc --version

# 4. Kernel-level denials in the window the start happened
sudo journalctl -k --since '5 min ago' | grep -iE 'apparmor|audit|denied'

The cgroup version check catches a real and confusing class of failure. A host in cgroup v1 or hybrid mode with a containerd and runc expecting unified v2 produces unable to apply cgroup configuration for every container, which reads as a Docker installation problem and is a host boot-parameter problem. systemd.unified_cgroup_hierarchy=1 on the kernel command line is the switch.

Verification

#!/usr/bin/env bash
set -euo pipefail
CONTAINER=web
CID=$(docker inspect --format '{{.Id}}' "$CONTAINER")
PID=$(docker inspect --format '{{.State.Pid}}' "$CONTAINER")

# No runc process should exist for a running container
! pgrep -f "runc.*$CID" >/dev/null \
|| { echo 'FAIL: a runc process is still resident' >&2; exit 1; }

# The process is in its own PID and mount namespaces, not the host's
for ns in pid mnt net; do
c=$(sudo readlink /proc/"$PID"/ns/"$ns")
h=$(sudo readlink /proc/1/ns/"$ns")
[ "$c" != "$h" ] || { echo "FAIL: shares host $ns namespace" >&2; exit 1; }
done

# Capabilities were actually dropped
echo 'container capabilities:'
sudo grep CapEff /proc/"$PID"/status
echo 'host init capabilities:'
sudo grep CapEff /proc/1/status

# Seccomp is in enforcing mode (2 = filter)
sudo grep Seccomp: /proc/"$PID"/status

echo OK

Seccomp: 2 in /proc/<pid>/status means a filter is loaded and enforcing. Seccomp: 0 on a container you expected to be confined means the profile was disabled — check HostConfig.SecurityOpt for seccomp=unconfined, and check whether --privileged was used, which disables it along with much else.

Knowledge check