Skip to main content
RunBook Academy

← All break/fix scenarios in Docker & Containers

intermediateVolume~20 min

Break/Fix 17: A new bind mount empties a directory the image shipped

Reported symptoms

  • ●The application starts, then fails at runtime with "template not found" or "no migrations found" for files the image definitely contains.
  • ●The identical image tag runs correctly on a developer laptop and fails on the production host.
  • ●Rebuilding and re-pushing the image changes nothing. Neither does rolling back to the previous tag.
  • ●No error appears anywhere in the daemon log — the deploy is reported as successful.

Evidence

  • · `docker run --rm --entrypoint ls IMAGE -la /app/config` lists the files, proving they are in the image.
  • · `docker exec app ls -la /app/config` on the running container shows an empty directory owned by root:root.
  • · `docker inspect app --format '{{json .Mounts}}' | jq` shows a bind of /srv/app/config onto /app/config.
  • · `ls -la /srv/app` on the host shows `config` created at the time of the first deploy, owned by root:root, mode 0755, empty.
Diagnosis and resolutionclick to reveal

Root cause

A mount overlays its mountpoint. The files the image shipped at /app/config still exist in the image layer, but the bind's contents replace them wholesale in the container's view. A named volume would have been seeded from the image on first use; a bind mount never is. Worse, when the host source path does not exist, dockerd creates it as an empty root-owned directory instead of refusing to start the container — so the mistake that hides the entire directory produces no error at all, and the evidence points at the image build.

Remediation

Stop overlaying a populated directory. Bind the single file instead of its parent (`-v /srv/app/config/app.yaml:/app/config/app.yaml:ro`), or seed the host source from the image before mounting it (`docker run --rm -v /srv/app/config:/out IMAGE cp -a /app/config/. /out/`), or move the operator override to a path the image deliberately leaves empty, such as /etc/app/conf.d. Use a named volume only when seed-from-image is the behaviour you actually want.

Verification

`docker exec app ls -la /app/config` lists both the image's files and the operator override. The application starts and serves. The check is repeated after a full `docker compose down && docker compose up -d` on a host where the source path does not already exist, because that is the state the failure needs to reproduce.

Prevention

Never bind a host directory over a directory the image populates. Design images with a dedicated, empty override directory for operator configuration. Add a CI job that runs the image with the production mount set and asserts the expected file list, so the shadowing is caught by a test rather than by an outage. Treat a host source path that dockerd had to create as a deploy-blocking condition.

Reported symptoms

A release adds one operator-supplied file: a per-environment app.yaml. The Compose change is a single line.

services:
  app:
    image: registry.example.com/app:2.7.0
    volumes:
      - /srv/app/config:/app/config:ro

The container starts. Sixty seconds later the application exits:

FATAL: configuration directory /app/config contains no defaults.yaml

defaults.yaml is in the image. It has been in the image for two years. The image is rebuilt, re-pushed, and rolled back, and the error does not move.

Diagnosis

Ask the image and the container the same question and compare the answers.

IMAGE=registry.example.com/app:2.7.0

docker run --rm --entrypoint ls "$IMAGE" -la /app/config
docker exec app ls -la /app/config
# from the image
-rw-r--r-- 1 root root 4096 defaults.yaml
-rw-r--r-- 1 root root  812 logging.yaml
drwxr-xr-x 2 root root 4096 templates

# from the running container
total 8
drwxr-xr-x 2 root root 4096 Aug 11 09:14 .
drwxr-xr-x 1 app  app  4096 Aug 11 09:14 ..

The image has the files. The container does not. Nothing in between touched the image, so something is covering the directory.

docker inspect app --format '{{json .Mounts}}' | jq
ls -ld /srv/app/config

The host path exists, is empty, is owned by root:root, and its mtime is the moment of the first deploy. Nobody created it — the daemon did, because the bind source was missing and dockerd creates a missing source rather than failing.

Resolution path

  1. Decide which of the three intents you actually had.
  2. Add one operator file alongside the image defaults → bind the file, not the directory.
  3. Replace the whole directory with operator content → seed the host path from the image first, then keep it under configuration management.
  4. Persist data the container writes → this should be a named volume, not a bind of a populated path.
  5. Preferred fix: mount the file.
  6. - /srv/app/config/app.yaml:/app/config/app.yaml:ro leaves the rest of the directory visible.
  7. Alternative: seed the host directory from the image.
  8. docker run --rm -v /srv/app/config:/out registry.example.com/app:2.7.0 cp -a /app/config/. /out/
  9. The host path now holds the image defaults plus your override — and is now your responsibility to update on every image upgrade.
  10. Best fix: change the image so this cannot recur.
  11. Ship defaults in /app/config and read operator overrides from an /etc/app/conf.d that the image creates empty. Nothing is ever shadowed because nothing is ever mounted over a populated path.
  12. Recreate the container and re-verify.
  13. docker compose up -d --force-recreate app

Verification

  1. The container sees both sets of files.
  2. docker exec app ls -la /app/config lists the image defaults and the override.
  3. The application starts and serves a request.
  4. Reproduce the original failure condition. Remove the host source path on a staging host, deploy, and confirm the deploy now fails loudly rather than starting with an empty directory.
  5. Confirm on a clean host. docker compose down && docker compose up -d where /srv/app/config does not pre-exist.

Prevention

  • Treat “dockerd had to create the bind source” as a deploy failure. It is silent by design and it is never what you meant.
  • Give every image an explicitly empty override directory. A mount over an empty directory cannot shadow anything.
  • Assert the mounted file list in CI: run the image with the real production mount set and ls the directories the application reads. This test costs two seconds and catches the entire class.
  • Review bind mounts in code review the way you review a DROP TABLE. A one-line volume entry can remove an entire directory from the application’s view without producing a single log line.