Runbook: Disk full — what to do when the host runs out of space

1 · Prerequisites

Confirm every item is in place before any state change.

• You have shell access on the Docker host and can use sudo
• You know which named volumes on this host hold production data and which are disposable
• A recent backup exists for every named volume, and its restore has been tested
• You have change approval to delete unused images, stopped containers and build cache
• An incident channel is open and the disk alert is acknowledged
• Set the variables reused below: CNAME=web, then CID=$(docker inspect -f "{{.Id}}" "$CNAME")

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

• · df -h /var/lib/docker prints the filling filesystem. Record Use% and Avail before you change anything
• · df -i /var/lib/docker prints inode usage. IUse% at 100% with bytes still free is a different fault and pruning will not fix it
• · docker system df prints RECLAIMABLE per category: Images, Containers, Local Volumes, Build Cache
• · docker system df -v prints the per-image, per-container and per-volume breakdown that names the actual consumer
• · docker image ls --filter dangling=true lists untagged images, the safest thing to remove
• · docker volume ls --filter dangling=true lists volumes no container references. Read this list against your data inventory before pruning anything: a stopped-but-wanted stack appears here
• · docker ps -a --filter status=exited --format "{{.Names}} {{.Status}}" lists exactly what container prune would destroy
• · docker buildx du prints build cache usage
• · sudo du -sh /var/lib/docker/overlay2 /var/lib/docker/volumes /var/lib/docker/containers ranks the top-level consumers
• · sudo find /var/lib/docker/containers -name "*-json.log" -size +100M -printf "%s\t%p\n" lists oversized container logs
• · docker inspect -f "{{.HostConfig.LogConfig.Type}} {{.HostConfig.LogConfig.Config}}" "$CID" prints the log driver and its options. An empty map means no rotation is configured, which is the usual root cause

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1. 1Announce the incident and halt any deploy in flight. A pull into a full filesystem leaves partial layers behind and makes the next step harder
2. 2Free the safest space first: docker image prune -f. It removes only untagged dangling images and prints Total reclaimed space. Tagged images you still reference are untouched
3. 3Re-check df -h /var/lib/docker. If Use% is now below 80%, skip to verification rather than pruning further
4. 4Remove stopped containers: docker container prune -f. It prints the deleted IDs and the reclaimed space. Every container the pre-check listed as exited is destroyed along with its writable layer
5. 5Remove aged build cache: docker builder prune -f --filter until=24h. It prints reclaimed space and keeps cache newer than 24h so the next build is not a cold rebuild
6. 6Truncate an oversized container log without stopping the service: sudo truncate -s 0 /var/lib/docker/containers/"$CID"/"$CID"-json.log. df shows the space returned immediately and the container keeps running
7. 7Only if space is still short, remove unreferenced volumes: docker volume prune -f. This destroys anonymous volumes permanently and there is no undo. Confirm the dangling list from the pre-checks against your data inventory first
8. 8Fix the cause, not the symptom. Add to /etc/docker/daemon.json: "log-driver": "json-file" and "log-opts": {"max-size": "10m", "max-file": "3"}. Both values must be quoted strings or the daemon refuses the file
9. 9Apply it with sudo systemctl restart docker, not reload. log-driver and log-opts are not in the reloadable set, so a reload silently changes nothing. Confirm live-restore is true first if the running containers must survive the restart
10. 10Recreate the stack so existing containers pick up the new options: docker compose up -d --force-recreate. Rotation applies only to containers created after the change
11. 11If the filesystem is still full after all of the above, the host is undersized. Grow the underlying volume and filesystem, or move data-root to a larger disk with the daemon stopped. data-root is not reloadable and requires a daemon restart

4 · Verification

Confirm the procedure actually fixed the problem.

• ✓df -h /var/lib/docker reports Use% at or below 80% and Avail above the agreed floor for this host
• ✓df -i /var/lib/docker reports IUse% below 80%
• ✓docker system df reports a total RECLAIMABLE below the threshold you set for this host, rather than merely lower than before
• ✓The operation that originally failed now succeeds: re-run the exact docker pull or docker compose up -d that returned "no space left on device" and it exits 0
• ✓sudo find /var/lib/docker/containers -name "*-json.log" -size +100M returns no output
• ✓docker inspect -f "{{.HostConfig.LogConfig.Config}}" "$CID" prints a map containing max-size and max-file. An empty map[] means rotation is still not in effect on this container
• ✓docker ps --format "{{.Names}} {{.HealthStatus}}" lists every expected service and no entry reads unhealthy
• ✓docker ps --filter health=unhealthy -q returns no output
• ✓The monitoring disk alert clears on its own within one scrape interval, without being manually silenced

5 · Rollback

If verification fails, undo the procedure in reverse order.

• ↶Pruning is not reversible. docker image prune, docker container prune, docker volume prune and docker builder prune delete data permanently. There is no undo and no recycle bin, which is why every pre-check lists what will be destroyed before any prune runs
• ↶A pruned image is recoverable only by pulling or rebuilding it: docker pull myorg/app@sha256:REPLACE_ME restores the exact image if the registry still holds that digest
• ↶A pruned anonymous volume is recoverable only from backup. If no backup exists the data is gone. Say so in the incident channel immediately rather than continuing to prune
• ↶A truncated container log is gone. If it was needed as evidence, record the gap in the incident timeline
• ↶To undo the rotation change, remove log-driver and log-opts from /etc/docker/daemon.json, run sudo systemctl restart docker, and recreate the containers
• ↶To undo a data-root relocation, stop the daemon, restore the original data-root value in daemon.json, and start the daemon. The original directory must still exist

6 · Escalation

When the runbook isn't enough, contact:

• · The disk refills within an hour of pruning: the cause is a live writer, not accumulated junk. Escalate to the owning application team with docker system df -v output and the ten largest log files
• · df reports free bytes but writes still fail with "no space left on device": you are out of inodes. Escalate to the platform team, because pruning will not help
• · df and du disagree by a wide margin: a deleted file is still held open. Escalate to the platform team with sudo lsof +L1 output
• · The filesystem holding /var/lib/docker is on shared or network storage: escalate to the storage team before deleting anything
• · The only reclaimable space left is in named volumes holding production data: do not prune. Escalate to the data owner and the incident commander for an explicit, recorded decision
• · The daemon will not start after a restart or a data-root move: escalate to the platform team. Do not delete anything under /var/lib/docker to make it start
• · Hand over: df -h and df -i before and after, the full docker system df -v, an itemised list of everything pruned, and the verified backup age for each volume

References

1. docker system df
2. docker system prune
3. docker volume prune
4. docker builder prune
5. json-file logging driver: max-size and max-file
6. dockerd: daemon.json options and which are reloadable
7. Keep containers alive during daemon downtime (live-restore)