critical riskdata loss risk~90 min

Runbook: Recover a corrupt image store or overlay2 layer

1 · Prerequisites

Confirm every item is in place before any state change.

A verified, restorable backup exists for every named volume on this host, or you accept that the last-resort path ends the data
Console or out-of-band access to the host - the SSH path must not depend on Docker
A change record, and the data owner reachable, because the last resort in this runbook destroys volumes
The registry that holds every image on this host is reachable, and you know which images are NOT in a registry
Free space on another filesystem for a volume copy - at least the size reported by docker system df for local volumes
Root or sudo on the host
You have read the Engine upgrade and daemon-will-not-start runbooks and ruled out a storage-driver change, which presents identically and is not corruption

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· systemctl is-active docker - the daemon may be running fine while individual images are broken
· sudo journalctl -u docker --since "6 hours ago" --no-pager | grep -iE "failed to mount|invalid argument|parent|overlay" - the exact error text
· docker info - read Storage Driver and Docker Root Dir, and compare with what you recorded before the incident
· sudo dmesg -T | grep -iE "i/o error|ext4-fs error|xfs . corrupt|remounting.read-only" - is the underlying disk healthy?
· findmnt -no SOURCE,FSTYPE,OPTIONS /var/lib/docker - a filesystem remounted ro produces exactly these Docker errors
· df -h /var/lib/docker and df -i /var/lib/docker - a full disk or exhausted inodes produces mount failures that are not corruption
· docker image ls --digests - which images the daemon still believes it has
· docker ps -a --format "{{.Names}} {{.Image}} {{.Status}}" - which containers are affected and which are still running
· docker volume ls --format "{{.Name}}" > /var/tmp/docker-volumes-before.txt - the list that must survive whatever happens next
· docker system df -v - sizes, so you can tell a copy that finished from one that was truncated

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Do not restart the daemon and do not delete anything yet. A restart discards the in-memory state that tells you which layer failed.
2Capture the evidence: sudo journalctl -u docker --since "6 hours ago" --no-pager > /var/tmp/docker-overlay-fail.log, plus docker info and docker image ls --digests appended to the same file. Expect a file naming the failing image or layer.
3Rule out the impostors first. Confirm findmnt does not show ro, df -h and df -i are below 100 percent, and dmesg shows no I/O errors. Expect all three clean; if any is not, this is a storage incident and the escalation path applies.
4Rule out a storage-driver or data-root change. Compare the Storage Driver and Docker Root Dir from docker info against your records. Expect them unchanged; if they changed, restore the previous value instead - the images are not corrupt, the daemon is looking elsewhere.
5PROTECT THE VOLUMES BEFORE ANY REPAIR. Volumes live under the same data root as the broken layers and every last-resort step destroys them. Copy each named volume off the host now.
6For each volume, run a copy container that mounts the volume read-only and writes a tar to a path outside the data root: docker run --rm -v "$VOLUME":/from:ro -v /srv/backup:/to alpine tar -C /from -cf "/to/$VOLUME.tar" . Expect a tar whose size is consistent with docker system df -v.
7Verify each tar before you rely on it: tar -tf "/srv/backup/$VOLUME.tar" | head and tar -tf ... | wc -l. Expect a file listing, not an error.
8Identify the affected images precisely. Run docker image inspect on each image named in the error, and try docker run --rm --entrypoint true "$IMAGE" for each. Expect the healthy images to exit 0 and the broken ones to reproduce the mount error.
9For each broken image that exists in a registry, remove only that image and pull it again: docker image rm "$IMAGE" then docker pull "$IMAGE". Expect the pull to succeed and docker run --rm --entrypoint true "$IMAGE" to exit 0.
10If docker image rm fails because a container references the image, remove the stopped container first with docker rm "$CONTAINER". Do not use docker rm -f on a running container that holds unsaved state.
11For each broken image that is NOT in a registry, rebuild it from source with docker build --no-cache -t "$IMAGE" "$CONTEXT". Expect a fresh build that does not reuse the damaged layers.
12If pulls themselves fail with a parent-layer error, clear the build cache and retry: docker builder prune -a, then docker pull "$IMAGE". Expect the pull to proceed past the layer that previously failed.
13Recreate the affected containers from their compose files rather than starting the old ones: docker compose -f "$COMPOSE_FILE" up -d --force-recreate. Expect every service to reach a running and healthy state on the freshly pulled images.
14Re-measure. Run docker system df, docker image ls and docker ps and compare against the pre-check. Expect the same containers, the same volumes, and no image still producing a mount error.
15LAST RESORT ONLY, and only after the volume copies are verified off-host: stop the daemon with sudo systemctl stop docker docker.socket, rename the data root with sudo mv /var/lib/docker /var/lib/docker.broken, start the daemon, and rebuild the host state from compose files and the restored volumes.
16After a data-root rebuild, restore each volume: docker volume create "$VOLUME" then docker run --rm -v "$VOLUME":/to -v /srv/backup:/from alpine tar -C /to -xf "/from/$VOLUME.tar". Expect the restored volume size to match the recorded figure.

4 · Verification

Confirm the procedure actually fixed the problem.

✓docker run --rm --entrypoint true "$IMAGE" exits 0 for every image that previously failed to mount
✓sudo journalctl -u docker --since "15 minutes ago" contains no "failed to mount", "invalid argument" or missing-parent messages
✓docker ps --format "{{.Names}}" lists every container recorded in the pre-check, with none missing
✓docker ps --filter health=unhealthy --format "{{.Names}}" prints nothing
✓docker volume ls --format "{{.Name}}" matches /var/tmp/docker-volumes-before.txt exactly
✓For each restored volume, the application reads its own data - a database answers a query against a real table, not merely starts
✓docker system df reports image and volume sizes consistent with the pre-check figures, not a fraction of them
✓docker info reports the same Storage Driver and Docker Root Dir as before the incident
✓A fresh docker pull of an unrelated image succeeds, proving the store accepts new layers

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶Re-pulling an image is not reversible in itself, but it is not destructive: the previous copy was already unusable.
↶If a rebuild produced a different image than the one that was running, redeploy the previous tag from the registry and confirm the digest with docker image ls --digests.
↶If you renamed the data root and the outcome is worse, stop the daemon, remove the new /var/lib/docker, and rename /var/lib/docker.broken back. This works ONLY if you renamed rather than deleted - which is why this runbook never says rm.
↶Deleting /var/lib/docker is irreversible. It destroys every image, every container and every named volume on the host at once. There is no undo and no partial recovery.
↶A volume restored from a tar is only as good as the tar. If the restore is wrong, do not overwrite the tar - keep it and escalate to the data owner.
↶If the host cannot be recovered, rebuild it from configuration management and restore volumes from the backup system, with the data owner confirming each restore point.

6 · Escalation

When the runbook isn't enough, contact:

· dmesg shows I/O errors or the filesystem under /var/lib/docker remounted read-only: escalate to the storage team immediately. This is failing hardware or a failing volume, and Docker-level repair will not hold.
· You are about to rename or remove /var/lib/docker: escalate for explicit authorisation from the data owner first, and only after the volume tars are verified on another host.
· An image that only exists locally is corrupt and its source is not in version control: escalate to the application owner - this is a rebuild that may not be reproducible.
· Volume data is missing or unreadable after a restore: stop, keep the tar, and escalate to the data owner and the backup team before any further attempt.
· Corruption reappears within days of a clean rebuild: escalate to the platform and storage teams. Repeat overlay2 damage on healthy hardware usually indicates a kernel, filesystem or storage-driver mismatch rather than bad luck.
· The host is a Swarm manager: escalate to the Swarm owner before stopping the daemon, so quorum is protected on the remaining managers.

Corrupt image store is what you call it after you have ruled out four things that look identical and are not corruption at all. The errors are unhelpfully generic:

failed to mount ... invalid argument
error creating overlay mount to /var/lib/docker/overlay2/.../merged
layer does not exist or a missing parent layer during a pull
open /var/lib/docker/overlay2/.../link: no such file or directory

Step 1: Capture evidence, change nothing

Read-only / Safecapture— Reads logs and daemon state, writes a copy for the ticket.

sudo journalctl -u docker --since "6 hours ago" --no-pager \
> /var/tmp/docker-overlay-fail.log

sudo journalctl -u docker --since "6 hours ago" --no-pager \
| grep -iE 'failed to mount|invalid argument|parent|overlay|layer'

docker info      >> /var/tmp/docker-overlay-fail.log
docker image ls --digests >> /var/tmp/docker-overlay-fail.log
docker ps -a --format '{{.Names}}\t{{.Image}}\t{{.Status}}' \
>> /var/tmp/docker-overlay-fail.log

Restarting the daemon is the first instinct and it is the wrong one here: it clears the in-memory state that names the failing layer, and it can start a partial cleanup you did not authorise. Capture first.

Step 2: Rule out the four impostors

Every one of these produces overlay2 mount errors and none of them is a corrupt image store.

Read-only / Safeimpostor checks— Reads filesystem and daemon state. Changes nothing.

# 1. Filesystem remounted read-only
findmnt -no SOURCE,FSTYPE,OPTIONS /var/lib/docker

# 2. Out of space, or out of inodes
df -h /var/lib/docker
df -i /var/lib/docker

# 3. Failing hardware underneath
sudo dmesg -T | grep -iE 'i/o error|ext4-fs error|xfs .* corrupt|remounting'

# 4. Storage driver or data root changed under the daemon
docker info | grep -E 'Storage Driver|Docker Root Dir'
sudo grep -E 'storage-driver|data-root' /etc/docker/daemon.json
sudo ls -1 /var/lib/docker

Check	Result that means it is NOT corruption	What it actually is
`findmnt`	`OPTIONS` contains `ro`	Filesystem remounted read-only; a storage incident
`df -h`	100% used	Out of space; use the disk-reclaim runbook
`df -i`	100% used	Inode exhaustion; escalate to storage
`dmesg`	I/O or filesystem errors	Failing device; escalate, do not repair Docker
`docker info`	Storage Driver differs from your records	Driver switch; images are intact but invisible

The driver case deserves emphasis. Docker documents that changing the storage driver makes existing containers and images inaccessible on the local system. Nothing is damaged - the daemon is reading a store that has never been written to. Restoring the previous storage-driver value fixes it, and any deletion at that moment destroys the store you still have.

Step 3: Protect the volumes before any repair

Read-only / Safeinventory— Reads the volume list and sizes. Changes nothing.

docker volume ls --format '{{.Name}}' > /var/tmp/docker-volumes-before.txt
docker system df -v

for V in $(docker volume ls --format '{{.Name}}'); do
MP=$(docker volume inspect --format '{{ .Mountpoint }}' "$V")
printf '%s\t%s\n' "$V" "$(sudo du -sh "$MP" 2>/dev/null | cut -f1)"
done

Configuration changecopy each volume off the host— Reads the volume through a helper container and writes a tar outside the data root.

BACKUP_DIR=/srv/backup
sudo mkdir -p "$BACKUP_DIR"

for V in $(docker volume ls --format '{{.Name}}'); do
docker run --rm \
  -v "$V":/from:ro \
  -v "$BACKUP_DIR":/to \
  alpine tar -C /from -cf "/to/$V.tar" .
done

# Verify every tar. An unreadable tar is not a backup.
for T in "$BACKUP_DIR"/*.tar; do
printf '%s: %s entries\n' "$T" "$(tar -tf "$T" | wc -l)"
done

The :ro on the source mount matters: the helper container must not be able to write to the volume it is rescuing. Copy the tars to a different host before you continue. A backup on the filesystem you are about to rename is not a backup.

Step 4: Identify exactly which images are affected

Corruption is usually narrow. Do not treat one broken layer as proof that the whole store is gone.

Read-only / Safeprobe each image— Starts and immediately exits a container per image to test whether its layers mount.

# Exits 0 if the image's layers mount; reproduces the error if they do not
for I in $(docker image ls --format '{{.Repository}}:{{.Tag}}' | grep -v '<none>'); do
if docker run --rm --entrypoint true "$I" 2>/dev/null; then
  printf 'OK    %s\n' "$I"
else
  printf 'BROKEN %s\n' "$I"
fi
done

docker image inspect "$IMAGE" | head -40

That loop is the whole diagnosis: it turns “Docker is broken” into a list of image names, and usually that list is short.

Step 5: Re-pull, or rebuild

Destructivere-pull one broken image— Removes the local copy of one image and pulls it again. Registry must hold the tag.

IMAGE=registry.example.com/team/api:1.4.2

# Confirm the registry can give it back BEFORE removing it
docker manifest inspect "$IMAGE" > /dev/null && echo "registry has it"

# A stopped container may hold a reference; remove that first
docker ps -a --filter ancestor="$IMAGE" --format '{{.Names}}\t{{.Status}}'

docker image rm "$IMAGE"
docker pull "$IMAGE"
docker run --rm --entrypoint true "$IMAGE" && echo "mounts cleanly"

Destructiverebuild an image with no registry copy— Rebuilds from source without reusing cached layers.

CONTEXT=/srv/src/api
IMAGE=team/api:1.4.2

# Do not let the build reuse the damaged cache
docker builder prune -a
docker build --no-cache -t "$IMAGE" "$CONTEXT"
docker run --rm --entrypoint true "$IMAGE" && echo "mounts cleanly"

If a docker pull itself fails with a missing-parent error, the build cache or a partially written layer is interfering. docker builder prune -a clears the cache and costs only build time. It removes no image, no container and no volume.

Step 6: The last resort

Data-loss riskrebuild the data root— Abandons all local Docker state. Requires verified volume tars on another host.

# HARD PRECONDITION: every volume tar is verified AND on another host.
ls -lh /srv/backup/*.tar

sudo systemctl stop docker docker.socket
sudo mv /var/lib/docker /var/lib/docker.broken
sudo systemctl start docker

docker info | grep -E 'Storage Driver|Docker Root Dir'
docker image ls          # expect empty - this is the point of the step

Configuration changerestore the volumes— Recreates each volume and unpacks its verified tar.

BACKUP_DIR=/srv/backup

for T in "$BACKUP_DIR"/*.tar; do
V=$(basename "$T" .tar)
docker volume create "$V"
docker run --rm \
  -v "$V":/to \
  -v "$BACKUP_DIR":/from \
  alpine tar -C /to -xf "/from/$V.tar"
done

docker volume ls --format '{{.Name}}' | diff /var/tmp/docker-volumes-before.txt -
docker compose -f "$COMPOSE_FILE" up -d
docker ps --filter health=unhealthy --format '{{.Names}}'

The final check is not docker ps. It is the application reading its own data: a query returning real rows, a file the users recognise. A container that starts on an empty volume looks exactly like a container that started on a restored one.

Common patterns

Symptom	Likely cause	Resolution
`failed to mount ... invalid argument` on one image only	That image’s layers are damaged	Remove and re-pull that image alone
Same error on every image	Not the images - check `findmnt`, `df -i`, `dmesg`	Fix the filesystem or the disk first
`docker pull` fails with a missing parent layer	Interference from cached or partially written layers	`docker builder prune -a`, then pull again
All images vanished, disk still full	Storage driver or data root changed	Restore the previous value; do not prune or pull
Errors returned days after a clean rebuild	Kernel, filesystem or driver mismatch, or failing hardware	Escalate to platform and storage; stop repairing locally
`docker image rm` refuses: image is in use	A stopped container still references it	Remove the stopped container first
Volumes empty after a data-root rebuild	The tars were never verified, or were on the renamed filesystem	Restore from the backup system; escalate to the data owner