Where state lives — stateful services on a production Docker host

Host design, the edge, segmentation and capacity are all decisions you can revise. Where the data lives is the one that decides how hard every other decision is to revise, because it determines whether a host is a thing you can rebuild or a thing you must recover.

This lesson is not about volume mechanics — the storage part of this course covers those. It is about the architectural question they serve: what, on this host, cannot be recreated?

Four kinds of data, and the only distinction that matters

Everything on a Docker host falls into one of four categories, and they differ in exactly one respect: what it takes to get the data back after the disk is gone.

Only row three is state in the sense that matters. The architectural work is to know exactly what is in it, and to have decided that each item belongs there.

The inventory

Most people discover what is in row three during an incident. Do it now instead — the enumeration takes two minutes.

$ docker ps --format '{{.Names}}' | xargs -r docker inspect --format '{{.Name}} {{range .Mounts}}{{.Type}}:{{.Source}}->{{.Destination}} {{end}}'

/postgres bind:/srv/data/postgres->/var/lib/postgresql/data
/api volume:/var/lib/docker/volumes/api-uploads/_data->/app/uploads
/redis volume:/var/lib/docker/volumes/redis-data/_data->/data
/proxy bind:/srv/config/nginx->/etc/nginx/conf.d

Then the volumes nothing is currently using, which is where the unpleasant surprises are:

$ docker volume ls --filter dangling=true

DRIVER    VOLUME NAME
local     8f3c1a92b7e0d45f6a1c8e2b4d7f0931a5c6e8b2d4f70193a5c7e9b1d3f5a709
local     old-postgres-data

Now classify each entry with one question: if this host’s disk died right now, how would I get this back?

• “Pull the image again” → not state.
• “It is in git” → not state; the nginx config bind mount above is configuration, and should be reproducible from a repository.
• “Restore last night’s backup” → state, with a known RPO.
• “I do not know” → state, with no RPO, and this is the finding.

The redis-data volume above is a good example of the ambiguity. Redis used as a cache is not state — losing it costs a cold start. Redis used as a queue or a session store is state, and losing it loses work. The volume looks identical in both cases; only you know which it is, and writing that down is the deliverable.

Should the database run on this host at all?

The container-versus-not framing is a distraction. Running PostgreSQL in a container is fine; PostgreSQL does not know or care. The real question is whether the production data should sit on one host’s local disk.

Reasons it is defensible:

• One deployment mechanism, one monitoring path, one set of resource controls for everything you run.
• Dev, staging and production share a definition, so “works on my machine” gaps close.
• For a small system, an external managed database can cost more per month than the entire host.

Reasons to be careful:

• The failure modes become storage failure modes — disk, filesystem, power loss — and containers do nothing about those.
• Backup requires consistency, which requires quiescing or a database-aware dump, not a tar of the volume.
• A major-version upgrade is coupled to an image tag, and databases are the software least tolerant of an unplanned rollback.
• Restart semantics are wrong for it. A restart policy that helpfully restarts a database after an unclean shutdown can restart it into recovery repeatedly.

Named volume or bind mount for production state?

Both end up as a directory on the host. The difference is operational, and it matters more than the documentation suggests.

For genuinely irreplaceable data on a production host, a bind mount to a path on a dedicated filesystem is usually the better choice — not because volumes are worse, but because it puts the data somewhere your existing backup, monitoring and capacity tooling already looks, and on a filesystem you can size and snapshot independently of /var/lib/docker.

services:
  postgres:
    image: postgres:18
    volumes:
      # A dedicated filesystem, in a path the backup job already covers.
      - /srv/data/postgres:/var/lib/postgresql/data

The named-volume equivalent can also point off-host entirely, which is worth knowing about:

$ docker volume create --driver local --opt type=nfs --opt o=addr=192.0.2.10,rw,nfsvers=4 --opt device=:/export/appdata app-data

app-data

That moves the failure domain to the NFS server, which is a real improvement if the NFS server is better protected than the Docker host, and a real regression if it is not. Moving state does not reduce risk by itself; it relocates it somewhere you have hopefully already solved.

The rebuild test

The property worth designing for is stateless-by-default: every container on this host can be destroyed and recreated from its image and its Compose file, with no data loss.

That is a claim, and claims about recovery are worth exactly as much as the last time they were tested.

1. Write down which paths on the host are state. This is the inventory above.
2. Confirm each one is covered by a backup with a known frequency, and that a restore has been performed from it — the backup part of this course covers verifying restores.
3. On a test host, provision from configuration alone: install Docker, clone the repository, restore the state paths from backup, and bring the stack up.
4. Time it. That number is your host-loss recovery time, and it is usually two to five times what people estimate.
5. Record everything the drill needed that was not in the repository or the backup. That list is your actual gap.

Knowledge check

Where next

The backup and disaster-recovery parts of this course take the state you have just inventoried and cover how to protect and restore it.