Lab 16: Recover from log-disk exhaustion

Objective

A common production incident: a container wrote 10 GB of logs and filled the disk. Find the cause, free space safely, and verify log rotation is now in place.

Tasks

Task 1: Reproduce the problem

Create a small root filesystem (loopback file) so we can fill it deliberately:

truncate -s 200M /tmp/disk.img
mkfs.ext4 -F /tmp/disk.img
sudo mkdir -p /mnt/smalldisk
sudo mount -o loop /tmp/disk.img /mnt/smalldisk
sudo mkdir -p /mnt/smalldisk/var-lib-docker

For this lab we will use the host’s actual /var/lib/docker. The first task is enough to simulate the symptom:

docker run -d --name log-bomb \
  --log-opt max-size=10G --log-opt max-file=1 \
  busybox sh -c 'while true; do echo "$(date) $(head -c 1000 /dev/urandom | base64)"; done'

Task 2: Diagnose

df -h /var/lib/docker
du -sh /var/lib/docker/containers/*/*-json.log | sort -h | tail -5
docker system df

You should see one log file dominating.

Task 3: Free space by stopping the writer

docker stop log-bomb

The container stops; the log file is now stable.

Task 4: Truncate the log (safe because container is stopped)

LOG=$(docker inspect log-bomb --format '{{.LogPath}}')
ls -la $LOG
sudo truncate -s 0 $LOG
df -h /var/lib/docker

Task 5: Reconfigure for prevention

Either set daemon-wide log rotation:

sudo tee /etc/docker/daemon.json > /dev/null <<'EOF'
{
  "log-driver": "json-file",
  "log-opts": { "max-size": "10m", "max-file": "5" }
}
EOF
sudo systemctl restart docker

Or set per-container log options:

docker run --log-opt max-size=10m --log-opt max-file=5 ...

Task 6: Verify

Start a new container with no log limits:

docker run -d --name log-test2 --log-opt max-size=1m --log-opt max-file=2 \
  busybox sh -c 'while true; do echo "padding $(date)"; sleep 0.1; done'
sleep 90
ls -la $(docker inspect log-test2 --format '{{.LogPath}}' | xargs dirname)

You should see at most 2 rotated files plus the active one, each ~1 MB.

Cleanup

docker rm -f log-bomb log-test2
sudo umount /mnt/smalldisk 2>/dev/null
sudo rm -rf /mnt/smalldisk /tmp/disk.img

Objectives

Prerequisites