Production Compose patterns — what works at scale · Docker & Containers

Compose is a perfectly good production tool for a single host. The line is clear and worth stating plainly: one host, one Compose project. Past that boundary you need an orchestrator, and no amount of scripting around Compose will build you one.

Inside the boundary, what distinguishes a stack that survives from one that does not is almost never the YAML. It is the deploy procedure and the handful of commands that must never appear in it.

What works, and what does not

Compose does well:

A complete app stack on one VM — web, API, database, cache, log shipper, reverse proxy.
Sidecar patterns — a metrics exporter or log forwarder alongside a service.
Ephemeral environments — a whole stack per pull request, torn down on merge.
Self-hosted internal tools — dashboards, status pages, runners.

Compose does not do:

Multi-host scheduling. There is no scheduler.
Zero-downtime rolling deploys. up replaces containers; it does not shift traffic.
Autoscaling. No metric-driven scale controller exists.
Self-healing across host failure. Restart policies need a live daemon on a live host to act.

The replacement window nobody measures

This is the honest limitation and it is worth quantifying rather than hand-waving.

What a single-service redeploy actually does

docker compose up -d api on a service whose config hash changed performs a strict sequence, and every step is serial:

Pull the new image if it is not local.
SIGTERM to the old container’s PID 1, then wait up to stop_grace_period (default 10 seconds) before SIGKILL.
Remove the old container. The published port is released here.
Create a new container from the new spec, allocating a new IP on the bridge and a new veth pair.
Start it. The port is bound again once the process listens.
If the service has a healthcheck, it begins in starting and stays there for at least one interval.

Between step 2 and step 5 nothing is serving. If the application takes eight seconds to shut down cleanly and six to warm up, the window is fourteen seconds of connection refused, per deploy, on every published port.

Nothing about that is a bug. Compose is doing exactly what a create-time-immutable container model requires. But it is the number you should know before you promise “we deploy without downtime”, and the two things that shrink it are within your control: make the application handle SIGTERM promptly so step 2 does not run to the full grace period, and put a reverse proxy in front so step 5 is retried by the proxy rather than seen by the client.

Read-only / Safemeasure it— How long a redeploy of one service actually leaves the port dead.

$ time docker compose up -d --wait api

[+] Running 1/1
✔ Container shop-api-1  Healthy                          13.8s

real    0m13.842s
user    0m0.089s
sys     0m0.041s

Illustrative output

Run that once against your real stack. Whatever it prints is your deploy outage, and it is usually longer than people guess.

The deploy sequence

The order matters, and each step exists because of a specific failure.

Service impact possibledeploy— A deploy that fails before it touches the running stack.

set -euo pipefail
cd /srv/shop

# 1. Validate the file before anything else. Catches a bad merge or a
#    missing variable without contacting the daemon.
docker compose config --quiet

# 2. Pull first, separately. A registry outage fails here, while the old
#    stack is still serving, instead of halfway through a replacement.
docker compose pull

# 3. See what is about to be replaced. Log it; this is your change record.
docker compose --dry-run up -d

# 4. Apply, and block on readiness. Non-zero exit means a failed deploy.
docker compose up -d --wait --wait-timeout 180

# 5. Verify from outside the stack, not from inside it.
curl -fsS https://shop.example.com/healthz

Step 2 is the one people skip and the one that pays. docker compose up -d on its own pulls lazily, service by service, mid-replacement — so a registry that goes away at the wrong moment leaves you with half the stack on the new version and half stopped.

Step 5 matters because every check inside the stack shares the stack’s assumptions. A container that reports healthy to its own healthcheck can still be unreachable through the proxy, because the proxy resolved the old container’s IP and has not re-resolved it.

`down -v` is not the opposite of `up`

The mental model that causes this is “up created it, down removes it, -v makes it thorough”. It is thorough. docker compose down -v removes the named volumes declared in the volumes: section and the anonymous volumes attached to the containers. That is the database, the uploads directory, the certificate store the ACME client spent a rate-limited quota on.

There is no confirmation prompt, no dry-run of the destructive part, and no undo. The volume is gone from /var/lib/docker/volumes before the command returns.

It gets into production scripts by a predictable route. Somebody is iterating on a schema change locally, where down -v is the fastest way to get a clean database, and the command goes into a redeploy.sh that later gets copied to a server. Or a runbook says “if the stack is in a weird state, docker compose down -v && docker compose up -d”, which is excellent advice on a laptop and a resignation letter on a database host.

The rules that actually hold:

-v never appears in a script that can run outside a developer machine.
A clean redeploy is docker compose down then docker compose up -d. Plain down leaves the volumes: section untouched.
If a stack really must be rebuilt from empty on a production host, restore from backup afterwards — which means testing that the restore works before you need it.

Command	Containers	Networks	Named volumes	Anonymous volumes
`docker compose stop`	Stopped	Kept	Kept	Kept
`docker compose down`	Removed	Removed	Kept	Kept
`docker compose down -v`	Removed	Removed	Deleted	Deleted
`docker compose down --rmi all`	Removed	Removed	Kept	Kept

The patterns

Reverse proxy in front

services:
  caddy:
    image: caddy:2.9
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - caddy_data:/data
      - caddy_config:/config
    networks: [edge, app]

  api:
    image: myorg/api:1.4.0
    restart: unless-stopped
    networks: [app]
    healthcheck:
      test: ["CMD", "/app/server", "healthcheck"]
      interval: 10s
      timeout: 3s
      retries: 3
      start_period: 30s

caddy_data is not optional decoration. It holds the ACME account key and the issued certificates; losing it means re-issuing every certificate, and Let’s Encrypt rate limits are per-week. This is the volume that down -v takes that nobody thinks about until the reissue fails.

Note also that the proxy does not use depends_on: service_healthy here. A proxy that refuses to start because one backend is unhealthy takes down every other site on the host. Let it start, let it return 502 for the backend that is down, and let it recover on its own when the backend comes back.

Restart policy on everything

restart: unless-stopped

no is the default, and a container with the default does not come back after a host reboot. The two candidates in production are always and unless-stopped; the difference appears exactly once, on the reboot after you deliberately stopped a container. always restarts it anyway; unless-stopped respects your decision. Prefer unless-stopped — the manual stop was almost certainly intentional.

A restart policy needs a running daemon to act on it. If dockerd itself dies, nothing restarts anything, which is what live-restore in daemon.json addresses for the containers already running.

Resource limits on everything

services:
  api:
    deploy:
      resources:
        limits:
          cpus: "2.0"
          memory: 1g
          pids: 500
        reservations:
          memory: 256m
  db:
    deploy:
      resources:
        limits:
          cpus: "4.0"
          memory: 4g
          pids: 200

Compose applies these; deploy.resources is not Swarm-only. The shorter non-deploy spellings mem_limit, cpus and pids_limit are equivalent — pick one style per file rather than mixing them.

pids is the limit people leave out and it is the cheapest one. A fork bomb, or more realistically a runaway worker pool, exhausts the host PID space and you cannot even SSH in to fix it. A few hundred is generous for most services.

Log rotation, before it is a problem

services:
  api:
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "3"

The json-file driver’s default is unbounded. A chatty service writes to /var/lib/docker/containers/<id>/<id>-json.log until the filesystem is full, at which point every container on the host starts failing writes at once and the cause is one service nobody was watching.

Set it per service, or better, set it once in daemon.json so a service that forgets still inherits a bound. Note that changing the default only affects containers created afterwards.

What if the host runs out of memory?

Two very different things happen depending on whether the container has a limit, and confusing them costs a lot of debugging time.

With memory set, the container has its own cgroup ceiling at memory.max. When its own usage hits that, the kernel invokes the OOM killer scoped to that cgroup and kills a process inside the container — usually PID 1, which exits 137. Nothing else on the host is affected. The evidence is docker inspect showing "OOMKilled": true and a counter incrementing in memory.events for that cgroup.

Without a limit, the container can allocate until the host is out. The global OOM killer then chooses a victim by oom_score, which is weighted by memory footprint — so it kills the biggest process, which is very often the database rather than the leaking service that caused it. The symptom is a database that died for no apparent reason while the real culprit is still running.

That asymmetry is the argument for limits on every service, not just the ones you distrust. A limit turns a host-wide incident into a single container restart, and it makes the evidence point at the right container.

The sizing rule is arithmetic: the sum of every container’s memory limit, plus roughly 1 GB for the kernel and daemon, must fit inside physical RAM with headroom. If the sum exceeds RAM you have not set limits, you have written down intentions.

Read-only / Safeaudit— Find services with no memory limit before the host does.

docker ps -q | xargs -r docker inspect \
--format '{{.Name}} {{.HostConfig.Memory}}' | awk '$2 == 0 { print $1, "NO LIMIT" }'

Set a top-level name: so the project cannot collide with another checkout.
Pin every image by digest, and record the digests in the change ticket.
Give every long-running service a healthcheck that fails when it should.
Use depends_on with condition: service_healthy for stateful dependencies.
Set memory, cpus and pids limits on every service; sum them against host RAM.
Set restart: unless-stopped on every service.
Bound the log driver, per service or in daemon.json.
Put a reverse proxy in front for TLS, and do not gate it on backend health.
Mount secrets from files synced by a secret manager, never from environment:.
Deploy with config --quiet, then pull, then up -d --wait; verify from outside.
Keep -v out of every script that can run on a host you care about.