dockerd internals — what the daemon actually does · Docker & Containers

dockerd is a single binary, but it orchestrates many subsystems. Understanding the subsystems helps you diagnose daemon-level failures.

The most useful single fact about dockerd is negative: it is not in the data path of a running container. Once a container is running, its process has no relationship to dockerd at all — not as a parent, not as a supervisor, not as a proxy for its I/O beyond the log stream. That is why systemctl restart docker sometimes fixes everything, sometimes fixes nothing, and sometimes causes an outage, and knowing which is which is the practical payoff of this lesson.

The subsystems

flowchart TB
  Client[CLI]
  subgraph dockerd
    REST["REST API<br/>unix socket, optionally TCP"]
    Dispatcher[Dispatcher]
    ImageMgr[Image manager]
    VolumeMgr[Volume manager]
    NetworkMgr["Network manager<br/>libnetwork"]
    ExecMgr[Container exec manager]
    ContainerdClient[containerd client]
  end
  Client -->|REST| REST
  REST --> Dispatcher
  Dispatcher --> ImageMgr
  Dispatcher --> VolumeMgr
  Dispatcher --> NetworkMgr
  Dispatcher --> ExecMgr
  Dispatcher --> ContainerdClient
  ContainerdClient --> containerd

The dispatcher routes API calls to the right subsystem. Each subsystem owns its own data (image DB, volume DB, network DB).

The process tree, which is the real architecture

Read-only / Safesee the tree— Four processes, three of them not children of each other in the way people assume.

# The whole hierarchy
pstree -pas $(pgrep -x dockerd | head -1) | head -20

# Every layer at once
ps -eo pid,ppid,comm,args --forest \
| grep -E 'dockerd|containerd|shim|nginx' | grep -v grep

# The crucial relationship: who is the parent of a container's PID 1?
CONTAINER=web
PID=$(docker inspect --format '{{.State.Pid}}' "$CONTAINER")
ps -o pid,ppid,comm -p "$PID"
ps -o pid,comm -p "$(ps -o ppid= -p "$PID" | tr -d ' ')"

Illustrative output:

    PID    PPID COMMAND
  14822    9134 nginx
   9134       1 containerd-shim

The container’s PID 1 (14822 on the host, 1 inside its namespace) has containerd-shim-runc-v2 as its parent — and that shim’s own parent is 1, host init. Not dockerd. Not containerd. The shim was deliberately re-parented to init so that neither daemon is in the container’s ancestry.

That single fact determines everything about daemon restarts.

What the daemon owns on disk

Read-only / Safethe data root— Where each kind of state lives. Read-only inspection; do not edit these by hand.

DATA_ROOT=$(docker info --format '{{.DockerRootDir}}')
sudo ls -1 "$DATA_ROOT"

# Per-container runtime state and the json-file log
sudo ls -1 "$DATA_ROOT"/containers | head

# Image and layer metadata (classic graphdriver image store)
sudo ls -1 "$DATA_ROOT"/image/overlay2/

# Named volumes
sudo ls -1 "$DATA_ROOT"/volumes | head

# Ephemeral execution state, re-derived on daemon start
sudo ls -1 /var/run/docker

Path	Holds	Survives reboot
`<data-root>/containers/<id>/`	Container config, hostconfig, the `-json.log` file	yes
`<data-root>/image/<driver>/repositories.json`	Tag to image-ID mapping, as plain JSON	yes
`<data-root>/image/<driver>/imagedb/`	Image configs by digest	yes
`<data-root>/image/<driver>/layerdb/`	Layer chain metadata	yes
`<data-root>/overlay2/`	The actual layer contents	yes
`<data-root>/volumes/`	Named volume data	yes
`<data-root>/network/files/local-kv.db`	libnetwork’s BoltDB store	yes
`/var/run/docker/` (exec root)	Runtime state, netns handles, sockets	no

The split matters during recovery. The exec root is scratch: it is re-derived on daemon start and losing it costs nothing. The data root is your images, your volumes and your container definitions, and there is no mechanism that rebuilds it.

Deleting /var/lib/docker to 'fix' the daemon

It appears in a lot of forum answers and it does resolve a stubborn daemon, because it removes everything the daemon could be confused about. It also removes every named volume on the host.

Named volumes are where databases live. A docker compose stack with volumes: [pgdata:/var/lib/postgresql/data] stores that data at <data-root>/volumes/pgdata/_data, and rm -rf /var/lib/docker deletes it with no confirmation, no dangling reference, and no way back short of a backup.

If you genuinely need to reset daemon state, the recoverable sequence is: stop the daemon, move the directory aside rather than deleting it (mv /var/lib/docker /var/lib/docker.broken), start the daemon, confirm the problem is gone, and only then decide what to do with the old directory. The move costs one inode and buys you the ability to change your mind.

And check first whether the corruption is in the exec root rather than the data root: /var/run/docker is scratch and clearing it is genuinely safe.

Daemon configuration and the reload boundary

The daemon reads /etc/docker/daemon.json at startup — or ~/.config/docker/daemon.json in rootless mode. Some keys can be applied without a restart; most cannot.

Configuration changevalidate, then reload— Always validate before applying. An invalid daemon.json prevents the daemon from starting at all.

# Parse and validate without touching the running daemon
sudo dockerd --validate --config-file=/etc/docker/daemon.json

# SIGHUP reload for the keys that support it
sudo systemctl reload docker

# Confirm the daemon actually took the change
docker info --format 'debug={{.Debug}} live-restore={{.LiveRestoreEnabled}} driver={{.Driver}}'

The reloadable set is documented and short: debug, labels, live-restore, max-concurrent-downloads, max-concurrent-uploads, max-download-attempts, default-runtime, runtimes, authorization-plugin, insecure-registries, registry-mirrors, shutdown-timeout, and features.

Everything else — data-root, storage-driver, bip, default-address- pools, iptables, userland-proxy, log-driver — needs a full restart, which means it needs a maintenance window on a host running production containers.

--validate is the step people skip, and skipping it is how a host ends up with a daemon that will not start at 03:00 because of a trailing comma. It parses the file and exits without touching anything.

An unparseable daemon.json is a stopped daemon, not a warning

dockerd treats its configuration file as mandatory-if-present. A syntax error means the daemon refuses to start; it does not fall back to defaults and it does not start with the valid subset.

The failure surfaces as systemctl start docker returning immediately with a unit failure, and the reason is only in the journal:

sudo journalctl -u docker.service -n 30 --no-pager
# unable to configure the Docker daemon with file /etc/docker/daemon.json:
# invalid character '}' looking for beginning of object key string

On a host where docker is how you manage everything, this is a self-inflicted outage with a thirty-second fix and a long discovery time. Validate before you reload, and keep a known-good copy alongside.

Debug logging

Configuration changeturn on debug without a restart— debug is reloadable, so this does not interrupt containers.

# Option A: SIGHUP-reloadable via daemon.json
sudo tee /etc/docker/daemon.json >/dev/null <<'JSON'
{ "debug": true }
JSON
sudo dockerd --validate --config-file=/etc/docker/daemon.json
sudo systemctl reload docker
docker info --format 'debug={{.Debug}}'

# Watch the daemon's own log while you reproduce
sudo journalctl -u docker.service -f

# Force a full stack trace of every daemon goroutine into the log.
# Use this when dockerd is hung rather than erroring.
sudo kill -s USR1 "$(pgrep -x dockerd | head -1)"

# Turn it back off when you are done; debug output is voluminous
sudo tee /etc/docker/daemon.json >/dev/null <<'JSON'
{ "debug": false }
JSON
sudo systemctl reload docker

The SIGUSR1 stack dump is the tool for the specific case where docker ps hangs and nothing is being logged. Docker’s troubleshooting documentation describes it as forcing “a full stack trace of all threads to be added to the daemon log”. The trace names the function every goroutine is blocked in, which usually identifies the wedged subsystem — a containerd call that never returned, a lock held by a stuck image pull — without any guesswork.

Turn debug off afterwards. It is genuinely verbose, and on a busy host it becomes a disk-space problem of its own.

Why the daemon is one binary, and where that boundary really is

The Docker daemon ships as a single binary because it is packaged and supervised as a single systemd unit. Internally it is subsystems — image, volume, network, exec — each with its own store.

But the architecturally important split is outside the binary. dockerd does not create containers. It builds an OCI bundle description, hands it to containerd over gRPC, and containerd hands it to a shim, and the shim runs runc. Four processes, and the chain is deliberately arranged so that the two long-lived daemons are not ancestors of the container.

The consequence is a clean rule for triage:

Symptom	Owner
`docker ps` hangs or errors	dockerd
`docker` works but a container is misbehaving	the container
Containers keep running while `docker` is unusable	dockerd only; containerd and the shims are fine
Every container died at the same instant	containerd or the shims, or a daemon restart without live-restore
`docker exec` fails but the container serves traffic	the shim’s exec path, or the container’s own state

The fourth row is the one that matters most, because it is the only one where restarting dockerd is both necessary and dangerous.

live-restore: the feature that exists because of the shim

Because the shims are not children of dockerd, dockerd can exit without taking containers with it. By default it does not — it stops them on the way out. live-restore changes that.

Configuration changeenable live-restore— live-restore is itself reloadable, so enabling it does not require the outage it prevents.

sudo tee /etc/docker/daemon.json >/dev/null <<'JSON'
{ "live-restore": true }
JSON

sudo dockerd --validate --config-file=/etc/docker/daemon.json
sudo systemctl reload docker

docker info --format 'live-restore={{.LiveRestoreEnabled}}'

Docker’s documentation states the default plainly: “By default, when the Docker daemon terminates, it shuts down running containers.” With live-restore, containers “remain running if the daemon becomes unavailable”.

The limits are equally explicit and worth knowing before you rely on it:

It “is only supported when installing patch releases (YY.MM.x), not for major (YY.MM) daemon upgrades”. A major-version upgrade will stop your containers regardless.
It “only works to restore containers if the daemon options, such as bridge IP addresses and graph driver, didn’t change”. Change bip and restart, and you may need to stop the containers manually.
While the daemon is down, nothing is draining the containers’ log FIFOs. Docker warns that “running containers may fill up the FIFO log the daemon normally reads”, and “a full log blocks containers from logging more data” — with a 64K default buffer. A chatty container will block on its own stdout during an extended daemon outage, which looks like the application hanging.

That third limitation is the one that surprises people. live-restore keeps containers running; it does not keep them unaffected.

What if dockerd is dead but the containers are still up?

This is a good state, not a bad one, and treating it as an emergency is how it becomes one.

The containers are serving traffic. Their shims are alive and parented to init. What you have lost is the management plane: no docker ps, no docker logs, no new containers, no docker exec.

You can still see and control them through containerd directly:

sudo ctr -n moby containers list
sudo ctr -n moby tasks list
sudo ctr -n moby tasks kill --signal SIGTERM <id>

The moby namespace is where Docker’s containers live in containerd. The task list shows PIDs and statuses even with dockerd completely absent.

Order of operations that does not make it worse:

Confirm the containers really are serving — curl the published ports. Do not infer it.
Capture journalctl -u docker.service before restarting anything; whatever killed dockerd is in there and a restart may roll the log.
Check whether live-restore was on: grep live-restore /etc/docker/daemon.json. If it was not, starting the daemon may cause it to reconcile state by stopping containers it does not recognise.
Start the daemon and immediately verify container count is unchanged.

The mistake is step zero: rebooting the host because “Docker is broken”. That converts a management-plane outage with zero user impact into a full service outage.

Diagnosing a daemon that will not start

Read-only / Safedaemon start failure— Four causes, in the order they occur.

# 1. What does systemd say, and what does the daemon say?
systemctl status docker.service --no-pager
sudo journalctl -u docker.service -n 50 --no-pager

# 2. Is the config valid? This is the most common cause.
sudo dockerd --validate --config-file=/etc/docker/daemon.json

# 3. Is the data root's filesystem present and writable?
DATA_ROOT=$(grep -o '"data-root"[^,]*' /etc/docker/daemon.json 2>/dev/null || echo default)
echo "configured data-root: $DATA_ROOT"
df -h /var/lib/docker
df -i /var/lib/docker

# 4. Is containerd healthy? dockerd depends on it.
systemctl status containerd.service --no-pager
sudo ctr version

Step 4 catches a failure mode that reads as a Docker problem and is not: if containerd.service is down or its socket is missing, dockerd will start and then fail its connection, logging something about the containerd gRPC address. Restarting docker will not help; restarting containerd will — and on a host with running containers, restarting containerd is its own decision because the shims survive it but reconnection can be imperfect.

Verification

Read-only / Safeverify the daemon is genuinely healthy— Exercises the API path rather than just checking the unit is active.

#!/usr/bin/env bash
set -euo pipefail

# The unit being active is necessary, not sufficient
systemctl is-active --quiet docker.service \
|| { echo 'FAIL: unit not active' >&2; exit 1; }

# The API answers
docker info >/dev/null 2>&1 || { echo 'FAIL: API not responding' >&2; exit 1; }

# It can actually create and run something end to end
docker run --rm alpine:3.20 true \
|| { echo 'FAIL: cannot run a container' >&2; exit 1; }

# containerd is reachable from dockerd's point of view
docker info --format '{{.ContainerdCommit.ID}}' | grep -q . \
|| { echo 'FAIL: no containerd commit reported' >&2; exit 1; }

# Warnings the daemon is reporting about itself
docker info --format '{{range .Warnings}}WARN {{.}}
{{end}}'

echo OK

systemctl is-active alone is the check that lets a half-broken daemon pass. A dockerd that has lost its containerd connection is active and useless; the docker run line is what catches it.