critical riskdata loss risk~30 min

Runbook: Reclaim Docker disk space safely

1 · Prerequisites

Confirm every item is in place before any state change.

A verified, restorable backup exists for every named volume on this host - not a backup job that is green, a restore you have actually performed
You know which volumes on this host hold production data and who owns them
You know which images on this host cannot be re-pulled or rebuilt, because their registry tag was overwritten or the build is not reproducible
A change record exists, because every step after the accounting phase deletes something permanently
The host is not mid-deploy and no build is running - a prune during a build removes the cache the build is using
Root or sudo, or membership of the docker group
Someone else knows you are doing this, so a second person can stop you before the --volumes step

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· df -h /var/lib/docker - how much space is actually needed, so you can stop deleting once you have it
· df -i /var/lib/docker - inode exhaustion looks like a full disk but is not fixed by deleting large files
· docker system df - the summary: TYPE, TOTAL, ACTIVE, SIZE and RECLAIMABLE for images, containers, volumes and build cache
· docker system df -v - the itemised accounting: every image with its shared and unique size, every container, every local volume with its LINKS count
· docker volume ls --format "{{.Name}} {{.Driver}}" > /var/tmp/docker-volumes-before.txt - the list you must be able to compare against afterwards
· docker volume ls --filter dangling=true --format "{{.Name}}" - exactly which volumes a prune would consider unreferenced
· docker image ls --filter dangling=true - which images a default prune would remove
· docker buildx du --verbose - the build cache, itemised, which docker system df reports only as a total
· docker ps -a --format "{{.Names}} {{.Status}}" - stopped containers are what is holding many images and volumes referenced
· sudo du -sh /var/lib/docker/overlay2 /var/lib/docker/volumes /var/lib/docker/containers /var/lib/docker/image - is the space even inside Docker?

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1ACCOUNT FIRST, DELETE NOTHING. Run docker system df and docker system df -v and save the output to /var/tmp/docker-df-before.txt. Expect a file naming every image, container and volume with its size.
2Write down the number of bytes you actually need to free, from df -h. Expect a specific figure. Without one you will keep deleting past the point of safety.
3Confirm the space is inside Docker at all. Compare du -sh on /var/lib/docker against df -h for its filesystem. Expect them to agree; if the filesystem is full but Docker is small, this is the wrong runbook.
4Snapshot the volume list: docker volume ls --format "{{.Name}}" > /var/tmp/docker-volumes-before.txt. Expect a file you can diff after every deletion.
5Start with container logs, which are usually the largest single win and are the least dangerous. Find them with sudo find /var/lib/docker/containers -name "*-json.log" -size +100M. Expect a short list of specific files.
6Fix log growth at the source rather than truncating repeatedly: set log-opts max-size and max-file in /etc/docker/daemon.json, validate with sudo dockerd --validate, and apply with sudo systemctl reload docker. Expect exit 0 from validate.
7Reclaim build cache next: docker builder prune. It touches no image, no container and no volume. Expect the reclaimed figure to match what docker buildx du reported as reclaimable.
8If more is needed, remove stopped containers: docker container prune. Read the list it prints BEFORE confirming. Expect only containers you recognise as finished.
9If more is needed, remove dangling images: docker image prune. This removes only untagged images with no tag pointing at them. Expect no tagged image to disappear.
10STOP AND RE-MEASURE. Run docker system df and df -h. If you have the bytes you needed, stop here. Everything beyond this point removes things you may want back.
11Only if still short, consider docker image prune -a. This removes every image with no container associated with it, including tagged images you would have to pull again. Confirm the registry still serves every tag you are about to delete before you run it.
12Do NOT reach for --volumes as an escalation of the same idea. Volumes are the only thing in this list that is not reproducible. Treat volume deletion as a separate, individually authorised change.
13If volumes genuinely must be reclaimed, identify candidates individually: docker volume ls --filter dangling=true, then docker volume inspect each one and confirm with its owner that the data is not needed. Expect a named, approved list.
14Remove approved volumes one at a time by name with docker volume rm "$VOLUME". Expect the command to fail if the volume is still in use, which is a safety feature, not an obstacle.
15Re-measure and close: docker system df, df -h, and diff /var/tmp/docker-volumes-before.txt against the current volume list. Expect only the approved names to be missing.

4 · Verification

Confirm the procedure actually fixed the problem.

✓df -h /var/lib/docker shows the free space you recorded as required, not merely more than before
✓df -i /var/lib/docker shows inode usage below 80 percent
✓diff of docker volume ls against /var/tmp/docker-volumes-before.txt shows only volumes you individually approved for deletion
✓docker ps --format "{{.Names}}" still lists every container that was running before the reclaim
✓docker ps --filter health=unhealthy --format "{{.Names}}" prints nothing
✓The operation that originally failed for lack of space now succeeds - retry the exact pull, build or container start
✓docker system df reports RECLAIMABLE close to zero for the categories you pruned
✓docker info still reports the same Docker Root Dir and Storage Driver as the pre-check
✓daemon.json log rotation is in effect: docker inspect --format "{{.HostConfig.LogConfig}}" on a newly started container shows the max-size and max-file you set

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶There is no rollback. Every prune and every docker volume rm is immediate and permanent; Docker keeps no recycle bin and no undo.
↶Images removed by docker image prune -a can be recovered only by pulling the same tag again, and only if the registry still holds that exact tag and digest. A tag that was overwritten upstream is gone.
↶Locally built images that were never pushed cannot be pulled back. They can only be rebuilt, and only if the build is reproducible - the base image, the packages and the build cache must all still exist.
↶Build cache removed by docker builder prune is not recoverable. The consequence is slower builds, not lost data.
↶Containers removed by docker container prune are gone with their writable layers. Anything written inside a container and not into a volume is lost.
↶A volume removed by docker volume prune --volumes or docker volume rm is deleted from disk. The ONLY recovery is a restore from backup, performed with the data owner, following the restore runbook.
↶If a volume was deleted in error, stop writing to the host immediately, do not run any further prune, and escalate to the data owner before attempting anything else.

6 · Escalation

When the runbook isn't enough, contact:

· You are about to run anything with --volumes on a production host: stop and get explicit, named authorisation from the data owner first. This is not a judgement call to make at 03:00 alone.
· A named volume is missing after a prune: escalate to the data owner immediately, quiesce writes on the host, and begin the restore runbook. Do not run further prunes.
· Space is still short after every safe step: escalate to the host owner to extend the filesystem or move the data root, rather than escalating to --volumes.
· df shows the filesystem full but du on /var/lib/docker accounts for only a fraction of it: escalate to the host owner - deleted files held open by a process, or a different consumer, is not a Docker problem.
· Inodes are exhausted while bytes are not: escalate to the storage team; this usually needs a filesystem rebuild rather than deletion.
· The host fills again within a day of the reclaim: escalate to the application owner as a logging or write-volume defect rather than repeating this runbook on a schedule.

Step 1: Account for the space before deleting anything

This step is read-only and it is the step people skip.

Read-only / Safedocker system df— Reads disk accounting. Deletes nothing.

# The summary: what exists, what is active, what is reclaimable
docker system df

# The itemised version: every image, container and volume by name
docker system df -v | tee /var/tmp/docker-df-before.txt

# Build cache, itemised - system df only gives a total
docker buildx du --verbose

# Is the space even inside Docker?
df -h /var/lib/docker
df -i /var/lib/docker
sudo du -sh /var/lib/docker/overlay2 /var/lib/docker/volumes \
          /var/lib/docker/containers /var/lib/docker/image

Read three things out of docker system df -v:

RECLAIMABLE in the summary is the honest ceiling. If it is 4 GB and you need 40 GB, no amount of pruning will save this host and you should be extending the filesystem.
SHARED SIZE vs UNIQUE SIZE per image. Deleting an image whose bytes are almost all shared frees almost nothing.
LINKS in the Local Volumes section. A volume with LINKS 0 is unreferenced, which is exactly what a prune targets - and “unreferenced right now” is not the same as “not needed”.

Then write down the number of bytes you need. Deleting until the alert clears is how people end up three commands past the point where they should have stopped.

Step 2: Exactly what each prune deletes

This table is the reason the runbook exists. The differences are small in the command line and enormous in consequence.

Command	Deletes	Does NOT delete
`docker builder prune`	Dangling build cache	Images, containers, volumes, networks
`docker builder prune -a`	All unused build cache	Images, containers, volumes, networks
`docker container prune`	All stopped containers, with their writable layers	Images, volumes, networks
`docker image prune`	Dangling (untagged) images only	Tagged images, containers, volumes
`docker image prune -a`	All images with no container associated with them, including tagged ones	Volumes, running containers
`docker network prune`	Networks not used by at least one container	Everything else
`docker volume prune`	Anonymous volumes not used by at least one container	Named volumes, images, containers
`docker volume prune -a`	All unused volumes, anonymous and named	Volumes currently attached to a container
`docker system prune`	Stopped containers, unused networks, dangling images, unused build cache	Tagged images, all volumes
`docker system prune -a`	The above, plus all unused images without at least one container associated with them	All volumes
`docker system prune --volumes`	The above, plus anonymous volumes	Named volumes
`docker system prune -a --volumes`	Everything unused: images, containers, networks, build cache and anonymous volumes	Named volumes, and anything attached to a running container

Two details in that table are the ones that catch people:

docker system prune already deletes build cache by default. The confirmation prompt says so - “unused build cache” is in the default warning text - but people read the prompt as boilerplate. If your builds suddenly take twenty minutes after a routine prune, that is why.

--volumes on system prune and -a on volume prune are not the same scope. docker system prune --volumes prunes anonymous volumes. docker volume prune -a prunes all unused volumes, named ones included. The second is broader, and its warning text is one line.

Step 3: Delete in order of increasing regret

Read-only / Safefind the log files first— Reads file sizes under the data root. Deletes nothing.

sudo find /var/lib/docker/containers -name '*-json.log' -size +100M \
-printf '%s\t%p\n' | sort -rn | head

Container logs are usually the largest single consumer and the safest to address, because the fix is configuration rather than deletion.

Configuration changebound the logs at source— Sets log rotation defaults. Applies to newly created containers.

# /etc/docker/daemon.json
# {
#   "log-driver": "json-file",
#   "log-opts": { "max-size": "50m", "max-file": "3" }
# }

sudo dockerd --validate
sudo systemctl reload docker

Note that these defaults apply to containers created after the reload. Existing containers keep the log configuration they were created with, so recreate them when the window allows.

Destructivebuild cache— Removes build cache. Slower builds afterwards; no data is lost.

docker buildx du
docker builder prune          # dangling cache only
docker builder prune -a       # all unused cache

Destructivestopped containers and dangling images— Removes stopped containers and untagged images. Read the list before confirming.

# Look first
docker ps -a --format '{{.Names}}\t{{.Status}}' --filter status=exited
docker image ls --filter dangling=true

# Then delete. Both print what they will remove and wait for y.
docker container prune
docker image prune

Data-loss riskdocker image prune -a— Removes every image with no container associated with it, including tagged images.

# Prove the registry can give them back before you delete them
docker image ls --format '{{.Repository}}:{{.Tag}}' --filter dangling=false

# Safer variant: only images older than a week
docker image prune -a --filter 'until=168h'

# The blunt version - re-pulling everything is the cost of getting this wrong
docker image prune -a

An image you built locally and never pushed cannot be pulled back. Neither can a tag that has since been overwritten upstream. -a is reversible only in the sense that a rebuild is possible, and a rebuild at 03:00 during an incident is not a plan.

Step 4: Volumes, if you must

Read-only / Safeidentify volume candidates— Reads volume metadata and sizes. Deletes nothing.

docker volume ls --format '{{.Name}}' > /var/tmp/docker-volumes-before.txt

# Which ones a prune would consider fair game
docker volume ls --filter dangling=true --format '{{.Name}}'

# What each one actually is, and how big
for V in $(docker volume ls --filter dangling=true --format '{{.Name}}'); do
MP=$(docker volume inspect --format '{{ .Mountpoint }}' "$V")
printf '%s\t%s\t%s\n' "$V" "$(sudo du -sh "$MP" 2>/dev/null | cut -f1)" "$MP"
done

# Which containers - including stopped ones - still reference a volume
docker inspect --format '{{.Name}} {{range .Mounts}}{{.Name}} {{end}}' \
$(docker ps -aq)

Data-loss riskremove one approved volume— Permanently deletes the volume's data. There is no undo.

VOLUME=build-scratch-2024

# Last look at what you are about to destroy
docker volume inspect "$VOLUME"
sudo du -sh "$(docker volume inspect --format '{{ .Mountpoint }}' "$VOLUME")"

# Fails if the volume is in use - that is a feature
docker volume rm "$VOLUME"

# Confirm only the approved name disappeared
docker volume ls --format '{{.Name}}' | diff /var/tmp/docker-volumes-before.txt -

What happens if you run `docker system prune -a --volumes` on a busy production host?

Everything not attached to a currently running container is removed in one pass:

Every stopped container, with its writable layer.
Every image with no container associated with it - including the previous release you were about to roll back to, and every locally built image you never pushed.
Every unused network.
All unused build cache.
Every anonymous volume, including the data directories of any database container that happened to be stopped.

The command completes in seconds and prints a reclaimed-bytes total. It does not tell you what the volumes contained. There is no undo, no recycle bin, and no way to reconstruct the data except a restore from backup.

The only defence is not running it. On a production host, prefer the narrow commands and delete volumes by name.

Step 5: Re-measure and stop

Read-only / Safeverify— Confirms the space was freed and nothing unintended was removed.

df -h /var/lib/docker
df -i /var/lib/docker
docker system df

# Nothing missing that should be there
docker ps --format '{{.Names}}'
docker ps --filter health=unhealthy --format '{{.Names}}'
docker volume ls --format '{{.Name}}' | diff /var/tmp/docker-volumes-before.txt -

# The thing that failed originally now works
docker pull "$IMAGE"

Stop as soon as you have the bytes you wrote down in step 1. “A bit more headroom while I am here” is how the volume step gets reached on a host that never needed it.

Common patterns

Symptom	Likely cause	Resolution
`df` full, `docker system df` small	The space is not Docker’s	Look outside `/var/lib/docker`; check deleted-but-open files
Prune frees almost nothing	Most images are shared layers, or everything is in use	Read `SHARED SIZE` vs `UNIQUE SIZE`; extend the filesystem
Builds became very slow after a routine prune	`docker system prune` removes unused build cache by default	Expected; nothing to recover
Disk full again within a day	Unbounded container logs	Set `log-opts` `max-size` and `max-file`, then recreate containers
Rollback image missing after a cleanup	`docker image prune -a` removed the previous tag	Pull it again if the registry still has it; otherwise rebuild
Database empty after a cleanup	An anonymous volume was pruned while its container was stopped	Restore from backup; escalate to the data owner
`docker volume rm` says the volume is in use	A container, possibly stopped, still references it	Correct behaviour - identify the container before forcing anything
`df -h` fine but `df -i` at 100%	Inode exhaustion from many small layer files	Deleting large files will not help; escalate to storage