high riskservice affecting~30 min

Runbook: dockerd will not start

1 · Prerequisites

Confirm every item is in place before any state change.

Console or out-of-band access to the host - do not rely on an SSH path that depends on Docker
Root or sudo on the host
Knowledge of what changed recently: a package upgrade, a daemon.json edit, a disk change, a reboot
A copy of the last known-good /etc/docker/daemon.json, from configuration management or backup
Agreement that no one else is restarting the daemon while you diagnose

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· systemctl status docker.service --no-pager - read the Active line and the last few log lines it prints
· sudo journalctl -xu docker.service --since "1 hour ago" --no-pager - the actual failure message
· sudo dockerd --validate - exits non-zero and names the offending directive if /etc/docker/daemon.json is bad
· sudo python3 -m json.tool /etc/docker/daemon.json - separates a JSON syntax error from an unknown-key error
· df -h /var/lib/docker /var - a full filesystem stops the daemon starting and the message rarely says so plainly
· df -i /var/lib/docker - inode exhaustion looks identical to a disk-full condition but df -h looks fine
· sudo ss -ltnp | grep -E "2375|2376" - something else already listening on a TCP port the daemon wants
· systemctl status docker.socket --no-pager - the socket unit can be active while the service is failing
· ls -l /var/run/docker.sock /var/lib/docker - confirm the socket and the data root exist and are on the expected filesystem
· sudo dmesg -T | tail -50 - filesystem or device errors under the data root

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Capture the failure before changing anything: sudo journalctl -xu docker.service --since "1 hour ago" --no-pager > /var/tmp/dockerd-fail.log. Expect a file containing the last start attempt.
2Read the last start attempt in that file and find the line immediately before the unit entered a failed state. Expect one specific error, not a general "job failed" line.
3Test the configuration first, because it is the most common cause: run sudo dockerd --validate. Expect exit code 0. A non-zero exit names the directive that is wrong.
4If validate reports a JSON syntax error, restore the last known-good /etc/docker/daemon.json from configuration management, then run sudo dockerd --validate again and expect exit 0.
5If validate reports an unknown directive, remove that key rather than guessing a replacement, revalidate, and note the key for the upgrade ticket. Expect exit 0.
6If validate reports a directive specified both as a flag and in the file, inspect the unit ExecStart with systemctl cat docker.service. Expect to find the same option in both places; remove it from one.
7If the configuration is clean, check for a listener conflict: sudo ss -ltnp for the ports named in daemon.json hosts, and systemctl status docker.socket. Expect no other process bound to the daemon port and no stale socket unit in a failed state.
8If nothing conflicts, check capacity: df -h /var/lib/docker and df -i /var/lib/docker. Expect both usage figures below 100 percent. Free space before retrying, do not delete anything under /var/lib/docker.
9If capacity is fine, compare the configured storage driver with the one the data root was written with: read the Storage Driver line from the last successful docker info in your records, and list the directories under /var/lib/docker. Expect the driver directory named in daemon.json to be the one that already contains data.
10If the data root shows filesystem errors in dmesg, stop here and escalate. Expect not to fix a corrupt data root by restarting the daemon.
11When the discriminating check has identified one cause and you have corrected it, run the daemon in the foreground once: sudo dockerd --debug. Expect it to reach "API listen on" and stay up.
12Stop the foreground daemon with Ctrl-C, then start the unit properly: sudo systemctl start docker. Expect systemctl is-active docker to print active.
13Bring the workload back with docker compose -f "$COMPOSE_FILE" up -d for each stack on the host. Expect every service to reach a running state.

4 · Verification

Confirm the procedure actually fixed the problem.

✓systemctl is-active docker prints active, and stays active for five minutes without the unit restarting
✓sudo dockerd --validate exits 0
✓docker version --format "{{.Server.Version}}" prints a version, which proves the client reached the daemon over the socket
✓docker info runs and reports the same Storage Driver and Docker Root Dir as before the incident
✓systemctl show docker -p NRestarts prints a value that stops increasing
✓journalctl -u docker --since "10 minutes ago" contains no level=error lines
✓docker ps --filter health=unhealthy --format "{{.Names}}" prints nothing
✓The application answers its own health endpoint from outside the host, not just from localhost

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶If you edited /etc/docker/daemon.json, restore the copy you took before editing: sudo cp /etc/docker/daemon.json.bak /etc/docker/daemon.json, then sudo dockerd --validate.
↶If you added a systemd drop-in under /etc/systemd/system/docker.service.d/, remove the file and run sudo systemctl daemon-reload to return to the packaged unit.
↶If you changed the storage-driver key, put the previous value back before starting the daemon. Starting on the wrong driver hides every existing image and container.
↶If you changed data-root, put the previous path back. The old data root is untouched and is still the one holding your volumes.
↶Deleting /var/lib/docker is not a rollback. It destroys every image, container and named volume on the host and cannot be undone.
↶If the daemon started only under dockerd --debug in the foreground, do not leave it running that way - it is not supervised and will not survive your session.

6 · Escalation

When the runbook isn't enough, contact:

· dockerd --validate exits 0, no port conflict, disk and inodes are fine, and the daemon still fails: escalate to the platform team with /var/tmp/dockerd-fail.log attached.
· dmesg shows I/O errors or filesystem corruption under /var/lib/docker: escalate to the storage team immediately and do not restart the daemon in a loop.
· The daemon starts but every image and container is missing: escalate before pulling or rebuilding. A storage-driver or data-root change is recoverable; a prune is not.
· The failure began immediately after a package upgrade: escalate to whoever owns the upgrade change and follow the Engine upgrade runbook rollback rather than improvising here.
· The host is a Swarm manager and the daemon cannot be recovered within the change window: escalate to the Swarm owner so quorum can be protected on the remaining managers.

A daemon that will not start produces one of about six distinct failures, and they are not equally likely. Roughly half of real cases are a malformed daemon.json. This runbook is ordered by probability, and every branch has a check that discriminates - one that comes back differently for this cause than for the others.

Step 0: Capture the failure before you change anything

Read-only / Safecapture— Reads the journal and writes a copy for the ticket. Changes no Docker state.

systemctl status docker.service --no-pager

sudo journalctl -xu docker.service --since "1 hour ago" --no-pager \
> /var/tmp/dockerd-fail.log

# The last start attempt only
sudo journalctl -xu docker.service --no-pager | tail -60

The line you want is the last one before the unit reports that it failed. systemd’s own “Failed to start Docker Application Container Engine” is a summary, not a cause.

Step 1: The malformed daemon.json (start here)

This is the single most common cause and it is easy to misread, because the daemon fails closed: an invalid configuration file means no daemon at all, not a daemon running with defaults.

Read-only / Safedockerd --validate— Validates the configuration file without starting the daemon. Exits non-zero if it is invalid.

sudo dockerd --validate
echo "exit: $?"

# A non-default location
sudo dockerd --validate --config-file /etc/docker/daemon.json

# Separate a JSON syntax error from an unknown-key error
sudo python3 -m json.tool /etc/docker/daemon.json

dockerd --validate is the discriminating check for this whole branch: Docker documents it as validating a configuration file without starting the daemon, returning a non-zero exit code for an invalid file. Exit 0 rules the branch out entirely.

There are three different “bad daemon.json” failures, and they need three different fixes:

Journal message	Cause	Fix
`invalid character ... looking for beginning of object key string`	JSON syntax: a trailing comma, a missing brace, a smart quote pasted from a wiki	Restore the known-good file; do not hand-edit under pressure
`unable to configure the Docker daemon with file /etc/docker/daemon.json: the following directives don't match any configuration option`	A key that does not exist in this Engine version, often after an upgrade or a downgrade	Remove the key, revalidate, raise it on the upgrade ticket
`the following directives are specified both as a flag and in the configuration file`	The same option is in `daemon.json` and in the unit’s `ExecStart`	Remove it from one side, usually the unit

The 'both as a flag and in the file' case is the one that catches people

The packaged unit passes -H fd:// on ExecStart. Adding a "hosts" key to daemon.json therefore conflicts with the unit, and the daemon refuses to start with a message about a directive specified twice - even though both halves look correct on their own.

Docker documents the fix as a systemd override that blanks ExecStart and re-declares it:

/etc/systemd/system/docker.service.d/docker.conf

[Service]
ExecStart=
ExecStart=/usr/bin/dockerd

followed by sudo systemctl daemon-reload. The documentation also warns that overriding this without a hosts entry in daemon.json or a -H flag causes Docker to fail to start - so the override and the config key must land together.

Read-only / Safeunit vs file— Shows what the unit actually passes to dockerd. Changes nothing.

systemctl cat docker.service
ls -l /etc/systemd/system/docker.service.d/ 2>/dev/null
sudo cat /etc/docker/daemon.json

Step 2: Port and socket conflicts

Read-only / Safelisteners— Reads listening sockets and unit state. Changes nothing.

# Something else already on the daemon's TCP port
sudo ss -ltnp | grep -E '2375|2376'

# The socket unit and the service unit are separate things
systemctl status docker.socket --no-pager
systemctl status docker.service --no-pager

# The unix socket itself
ls -l /var/run/docker.sock

# Is another dockerd already running?
pgrep -a dockerd

The discriminating message is bind: address already in use or listen unix /var/run/docker.sock: bind: address already in use. Two shapes:

A second daemon. Often a foreground dockerd --debug left running from an earlier diagnosis. pgrep -a dockerd finds it.
A stale socket unit. docker.socket can be active while docker.service is failing; restarting only the service then reuses the same broken socket. sudo systemctl stop docker.socket docker.service and start the service again.

Step 3: Storage driver mismatch after a change

Read-only / Safestorage driver— Reads the configured driver and what is actually on disk.

# What the config asks for
sudo grep -E 'storage-driver|data-root' /etc/docker/daemon.json

# What the data root actually contains
sudo ls -1 /var/lib/docker
sudo du -sh /var/lib/docker/* 2>/dev/null | sort -h | tail

If daemon.json names overlay2 and /var/lib/docker has a populated overlay2/ directory, they agree. If it names something else while overlay2/ holds all the data, the daemon either refuses to start or - worse - starts cleanly into an empty store.

Step 4: Out of disk, or out of inodes

Read-only / Safecapacity— Reads filesystem usage. Changes nothing.

df -h /var/lib/docker /var /
df -i /var/lib/docker /var /

# Where the space went, without touching anything
sudo du -sh /var/lib/docker/overlay2 /var/lib/docker/volumes \
         /var/lib/docker/containers /var/lib/docker/image 2>/dev/null

df -i is the discriminating check, and it is the one people skip. A data root that is 60% full by bytes but 100% full by inodes fails in exactly the same way, and df -h looks reassuring. Millions of tiny layer files make this ordinary rather than exotic.

Free space by removing something that is not under /var/lib/docker first - rotated logs, old kernels, a stray tarball in /var/tmp. Reclaiming Docker’s own space is a separate runbook, and it is not safe to do with the daemon down.

Step 5: Corrupt state under the data root

Read-only / Safedevice and filesystem errors— Reads kernel messages and filesystem state.

sudo dmesg -T | tail -50
sudo journalctl -k --since "6 hours ago" --no-pager \
| grep -iE 'i/o error|ext4-fs error|xfs .* corruption|remounting.*read-only'

findmnt -no SOURCE,FSTYPE,OPTIONS /var/lib/docker

OPTIONS containing ro after an unplanned reboot is conclusive: the filesystem remounted read-only, and no amount of restarting the daemon will help. This is a storage incident, not a Docker one. Escalate rather than iterating.

If the filesystem is healthy but the daemon still dies on a specific object, the image store may be damaged - that has its own recovery runbook, and it starts by protecting the volumes.

Step 6: Prove it with a foreground run

Service impact possibledockerd --debug— Runs the daemon in the foreground with debug logging. Not supervised - stop it before starting the unit.

# Stop the unit first so the two do not fight over the socket
sudo systemctl stop docker.service docker.socket

sudo dockerd --debug

# Expect a line containing: API listen on /var/run/docker.sock
# Ctrl-C to stop, then start the unit properly:
sudo systemctl start docker
systemctl is-active docker

A foreground run prints the failure directly instead of through the journal, which turns a five-minute log hunt into a one-line answer. It is a diagnostic, not a destination: leave the daemon under systemd.

The tree, in one table

Discriminating check	Result that indicts this cause	Cause
`sudo dockerd --validate`	non-zero exit, names a directive	Malformed or incompatible `daemon.json`
`python3 -m json.tool /etc/docker/daemon.json`	JSON parse error	Syntax error, not an unknown key
`systemctl cat docker.service`	option present in both unit and file	Flag/file duplication
`sudo ss -ltnp` / `pgrep -a dockerd`	another listener or another dockerd	Port or socket conflict
`df -h` and `df -i /var/lib/docker`	either at 100%	Out of space or out of inodes
`grep storage-driver` vs `ls /var/lib/docker`	config names a driver with no data	Storage driver mismatch
`findmnt -no OPTIONS /var/lib/docker`	contains `ro`	Filesystem remounted read-only
`dmesg -T`	I/O or filesystem errors	Corrupt or failing storage