systemd integration — managing dockerd as a unit

The Docker daemon, containerd, and every per-container shim are processes that systemd knows about. When dockerd “crashes”, systemd decides whether to restart it, how many times, how long to wait, and — crucially — whether to take your containers down with it.

Three directives in the packaged units do almost all of that work, and none of them is obvious from the name.

The docker.service unit

This is the upstream unit, verbatim from the Moby repository, minus a couple of comments:

[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target nss-lookup.target docker.socket firewalld.service containerd.service time-set.target
Wants=network-online.target containerd.service
Requires=docker.socket
StartLimitBurst=3
StartLimitIntervalSec=60

[Service]
Type=notify
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutStartSec=0
RestartSec=2
Restart=always

LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity

# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes

# kill only the docker process, not all processes in the cgroup
KillMode=process
OOMScoreAdjust=-500

[Install]
WantedBy=multi-user.target

The obvious fields first:

• Requires=docker.socket — the daemon will not start without the activation socket, and stopping the socket stops the service too.
• Wants=containerd.service — a weaker dependency: dockerd will start without containerd and then fail every container operation.
• ExecStart=... -H fd:// — listen on the file descriptor systemd hands over, rather than opening a socket itself. This is why a "hosts" key in daemon.json collides with the unit and stops the daemon from starting.
• ExecReload=/bin/kill -s HUP $MAINPID — systemctl reload docker is literally a SIGHUP, and the daemon applies only its documented subset of reloadable keys.
• Type=notify with TimeoutStartSec=0 — the daemon tells systemd when it is ready, and systemd will wait indefinitely. A host with thousands of container records genuinely takes minutes to become ready, and a timeout would turn that into a restart loop.

The three that actually matter are the ones with comments on them.

$ sudo systemctl start docker

Job for docker.service failed.
See "systemctl status docker.service" and "journalctl -xeu docker.service" for details.
docker.service: Start request repeated too quickly.
docker.service: Failed with result 'exit-code'.

“Start request repeated too quickly” is not the root cause and contains no information about it — it is systemd’s rate limiter refusing to try again. Two things follow:

# The real failure is in the FIRST attempt this boot, not the last.
journalctl -u docker -b --no-pager | head -40

# Clear the rate limiter so the next start is actually attempted.
sudo systemctl reset-failed docker.service

Resetting without fixing the cause just spends another three attempts.

The cgroup driver has to match

docker info reports a cgroup driver: systemd or cgroupfs. These are the only two values --exec-opt native.cgroupdriver= accepts. On a systemd host, the answer should be systemd.

{
"exec-opts": ["native.cgroupdriver=systemd"]
}

The reasoning is the same as the Delegate=yes argument, one level down. systemd allocates a cgroup per unit and acts as the cgroup manager for the host. If Docker writes the cgroup filesystem directly with cgroupfs while systemd manages the same tree, the host has two cgroup managers with two views of available and in-use resources. The upstream guidance on this is blunt: nodes configured that way “become unstable under resource pressure.”

$ docker info --format '{{.CgroupDriver}} cgroup v{{.CgroupVersion}}'

systemd cgroup v2

docker.socket — the activation mechanism

The socket unit is short:

[Unit]
Description=Docker Socket for the API

[Socket]
ListenStream=/run/docker.sock
SocketMode=0660
SocketUser=root
SocketGroup=docker

[Install]
WantedBy=sockets.target

systemd holds /run/docker.sock open even when dockerd is not running. When a client connects, systemd starts docker.service and hands it the file descriptor — which is what -H fd:// in ExecStart consumes. This is why docker info can succeed on a host where nobody started the daemon.

Note SocketGroup=docker and SocketMode=0660. That is the entire access control on the Docker API: membership of the docker group. The Docker post-install documentation states it plainly — “The docker group grants root-level privileges to the user.” Adding somebody to docker is administratively identical to giving them passwordless root, because the API they gain can mount the host root filesystem into a container.

sudo systemctl stop docker.socket
sudo systemctl stop docker.service

# Only if you also need the runtime down. This kills every container.
# sudo systemctl stop containerd.service

systemctl is-active docker.socket docker.service

For a stop that survives a reboot, disable both:

sudo systemctl disable --now docker.socket docker.service

# Stronger: refuses to start even if something pulls it in as a dependency.
# sudo systemctl mask docker.socket docker.service

Editing the unit correctly

Never edit /lib/systemd/system/docker.service in place. The next apt upgrade docker-ce overwrites it, silently reverting your change at the worst possible moment. Use a drop-in:

sudo systemctl edit docker.service
# writes /etc/systemd/system/docker.service.d/override.conf

[Service]
# ExecStart is a list-valued directive. Without the empty assignment first,
# systemd appends and then refuses to start with "more than one ExecStart".
ExecStart=
ExecStart=/usr/bin/dockerd --containerd=/run/containerd/containerd.sock
Environment="HTTPS_PROXY=http://proxy.example.com:3128"
Environment="NO_PROXY=localhost,127.0.0.1,.example.com"

That specific override — dropping -H fd:// — is the supported way to move the socket list into daemon.json without the flag conflict.

sudo systemctl daemon-reload
systemctl cat docker.service        # base unit plus every drop-in, in order
systemctl show docker.service -p ExecStart --no-pager

systemctl cat is the one to reach for during an incident on a host you did not build. It shows the base unit and every drop-in, which is the only way to see an override somebody added two years ago.

Diagnosing systemd-related failures

A safe daemon-restart procedure

1. Confirm live restore is already on: docker info --format "{{.LiveRestoreEnabled}}". If it is not, enable it with systemctl reload docker first — that costs nothing.
2. Classify the change. Networking, storage-driver or data-root changes are container-restarting; a daemon binary patch or a log-level change is not.
3. Record the ground truth you will compare against: docker inspect --format "{{.State.Pid}}" "$CONTAINER" for a representative container.
4. Validate config before you touch the service: sudo dockerd --validate --config-file=/etc/docker/daemon.json.
5. Restart: sudo systemctl restart docker.
6. Wait for readiness by polling docker info, not by sleeping a fixed interval — a busy host takes longer than you think.
7. Verify the PID you recorded is unchanged, and that the application itself answers. docker ps showing Up is not evidence.
8. Read journalctl -u docker -b --since "-5 min" for warnings the restart emitted.

Knowledge check