Skip to main content
RunBook Academy

← All break/fix scenarios in Docker & Containers

beginnerStorage~15 min

Break/Fix 5: Container filesystem fills up at runtime

Reported symptoms

  • ●Container cannot write to /var/log/myapp.log (or similar)
  • ●Application logs errors about disk space
  • ●docker inspect shows increasing WriteableLayer size

Evidence

  • · docker exec container df -h shows 100% on overlay
  • · ls -la /var/lib/docker/overlay2/<id>/diff/ shows large files
Diagnosis and resolutionclick to reveal

Root cause

The container's writable layer is growing without bound. Common causes: application logs not rotated; container writing to a path that lives in the writable layer; volume not mounted correctly.

Remediation

Identify the largest files in the writable layer. Truncate or remove them. Configure log rotation. Mount a volume for any path that should persist. Recreate the container with proper resource limits.

Verification

Writable layer size is bounded. df -h inside the container shows free space. Application resumes normal operation.

Prevention

Configure log rotation (max-size, max-file in daemon.json). Mount volumes for any persistent data. Use tmpfs for ephemeral caches. Set logrotate inside the container for application logs.

Reported symptoms

  • Application logs errors about disk space.
  • Container runs out of storage unexpectedly fast.

Evidence provided

$ docker exec CONTAINER df -h
Filesystem      Size  Used Avail Use% Mounted on
overlay          10G  10G     0  100% /

$ du -sh /var/lib/docker/overlay2/<id>/diff/
9.5G  /var/lib/docker/overlay2/<id>/diff/

Resolution path

  1. Find the largest files in the writable layer.
  2. find /var/lib/docker/overlay2/<id>/diff -size +100M -type f
  3. Common culprits: log files, database files, downloaded artefacts.
  4. Truncate or remove the largest files.
  5. truncate -s 0 /var/lib/docker/overlay2/<id>/diff/var/log/myapp.log
  6. rm /var/lib/docker/overlay2/<id>/diff/tmp/largefile
  7. Configure log rotation.
  8. Update daemon.json:
  9. ```json
  10. {
  11. "log-driver": "json-file",
  12. "log-opts": { "max-size": "10m", "max-file": "3" }
  13. }
  14. ```
  15. Restart Docker: systemctl restart docker.
  16. Recreate the container.
  17. docker compose up -d
  18. Use a volume for persistent data.
  19. Mount a named volume for paths that need to persist beyond container lifecycle.
  20. Set tmpfs for ephemeral caches.
  21. --tmpfs /tmp:rw,size=64m

Verification

  1. Writable layer size is bounded.
  2. docker exec CONTAINER df -h shows free space.
  3. Application resumes normal operation.
  4. No more "no space left on device" errors.

Prevention

  • Configure log rotation at the daemon level.
  • Mount volumes for any persistent data the application writes.
  • Use tmpfs for caches and temporary files.
  • Set per-container storage quotas (when supported by your storage driver).
  • Monitor disk usage; alert at 80%.