What production-readiness means here · Docker & Containers

The single most useful question a sysadmin can ask of any Docker content is:

Would this survive at 03:00 with no one awake to debug it?

Most Docker tutorials are written for developers who want their app to run on their laptop. Most production Docker environments are written for operators who need the platform to run unattended for months. The patterns overlap less than you think.

Production vs developer defaults

Concern	Developer default	Production default	Verify with
Image base	`latest`	Pinned digest	`docker inspect --format '{{.Image}}'`
Container user	root	Non-root via `USER` directive	`docker inspect --format '{{.Config.User}}'`
Restart policy	none	`unless-stopped`	`docker inspect --format '{{.HostConfig.RestartPolicy.Name}}'`
Health check	none	Defined `HEALTHCHECK` in image	`docker inspect --format '{{.State.Health.Status}}'`
Resource limits	none	CPU + memory + PIDs	`docker inspect --format '{{.HostConfig.Memory}}'`
Logging	json-file unbounded	rotated + shipped to central store	`docker inspect --format '{{.HostConfig.LogConfig}}'`
Storage	anonymous volumes	named volumes with backup plan	`docker inspect --format '{{json .Mounts}}'`
Networking	default bridge	user-defined bridge with segmentation	`docker network ls`
Secrets	env vars in compose	external secret manager	`docker inspect --format '{{json .Config.Env}}'`
Updates	`docker pull && restart`	staged rollouts with rollback	change record

The fourth column is the point. A production default that nobody checks reverts within a quarter, quietly, one deployment at a time.

Auditing a host against the list

Read-only / Safehost audit— Reports every running container that violates a production default. Nothing is changed. Run it on a host you have inherited before you change anything.

docker ps --format '{{.Names}}' | while read -r C; do
docker inspect "$C" --format '{{.Name}}
user={{if .Config.User}}{{.Config.User}}{{else}}ROOT(unset){{end}}
restart={{if .HostConfig.RestartPolicy.Name}}{{.HostConfig.RestartPolicy.Name}}{{else}}none{{end}}
memory={{if .HostConfig.Memory}}{{.HostConfig.Memory}}{{else}}UNLIMITED{{end}}
pids={{if .HostConfig.PidsLimit}}{{.HostConfig.PidsLimit}}{{else}}UNLIMITED{{end}}
health={{if .State.Health}}{{.State.Health.Status}}{{else}}NONE{{end}}
logdriver={{.HostConfig.LogConfig.Type}} opts={{if .HostConfig.LogConfig.Config}}{{.HostConfig.LogConfig.Config}}{{else}}DEFAULTS(unbounded){{end}}'
done

Read-only / Safeaudit output— A typical inherited host. Every capitalised word is a finding.

$ ./docker-audit.sh

/web
user=ROOT(unset)
restart=unless-stopped
memory=UNLIMITED
pids=UNLIMITED
health=healthy
logdriver=json-file opts=DEFAULTS(unbounded)
/postgres
user=999
restart=always
memory=2147483648
pids=512
health=NONE
logdriver=json-file opts=map[max-file:3 max-size:10m]

Illustrative output

Two containers, six findings. None of them is an outage today, and each is a plausible cause of one: an unbounded web can take the host’s memory, an unbounded log can take the host’s disk, and postgres has no health signal at all.

The five questions every production decision must answer

What happens when this fails? Every component will fail. The question is what the failure looks like and how contained it is.
What is the rollback? If a change makes things worse, how do you reverse it? Rollback is part of the change, not a separate concern.
Who gets paged? Every change that affects observability, capacity, or reliability changes the on-call experience.
What is the blast radius? A bug or attack affects a container, a host, a network segment, an entire fleet. The blast radius should be proportional to the value.
Is the change observable? A change that cannot be measured is a change that cannot be reasoned about.

The five things production Docker always does

Pins image versions by digest, not tag.
Runs containers as non-root with the smallest capability set the workload needs.
Defines a HEALTHCHECK for every long-running service.
Ships logs out of the host (to journald, syslog, Loki, or a sidecar).
Backs up volumes and validates restores — quarterly, on a clock, with a clock.

The healthcheck that nobody acts on

A team adds HEALTHCHECK to every image, sees healthy in docker ps, and considers self-healing solved. Months later a service wedges — accepting TCP connections, answering nothing — and stays wedged all night. docker ps said unhealthy the whole time.

A restart policy does not respond to health. The documented triggers are exits: on-failure acts on “a non-zero exit code”, always and unless-stopped act on the container stopping. A container that is running and unhealthy has not stopped, so nothing fires. Docker Engine records the health status and takes no action on it.

The first diagnosis is usually that the healthcheck is broken, because the obvious mental model says an unhealthy container gets restarted. The healthcheck worked perfectly; it was correctly reporting a failure to an audience of nobody.

A health status is only useful if something reads it:

Compose, at startup: depends_on with condition: service_healthy gates dependent containers — but only during startup, not for the life of the service.
A load balancer or reverse proxy removing the backend, which is what actually protects users.
An orchestrator with its own health-driven restarts.
Monitoring, which is the honest minimum on a single host: alert on .State.Health.Status, and let a human act.

Pick one deliberately and write down which. “We have healthchecks” is not an answer to “what restarts it”.

Read-only / Safewho is unhealthy right now— Lists unhealthy containers. Zero output is the passing case; this is the check to alert on.

docker ps --filter health=unhealthy --format '{{.Names}} {{.Status}}'

always and unless-stopped are not synonyms, and the difference shows up at reboot

Both restart a container that stops on its own. They differ over one case: a container you stopped on purpose.

always — “Always restart the container if it stops. If it’s manually stopped, it’s restarted only when Docker daemon restarts or the container itself is manually restarted.”
unless-stopped — “Similar to always, except that when the container is stopped (manually or otherwise), it isn’t restarted even after Docker daemon restarts.”

Read that first bullet again with an operator’s eye. You stop a container at 22:00 because it is misbehaving. At 03:00 an unattended package upgrade restarts dockerd. With always, the container you deliberately took out of service comes back — into the same failure, unattended, with the person who stopped it asleep and no record that anything restarted it.

That is why unless-stopped is the production default in this course: it makes “stopped” a durable statement of intent that survives a daemon restart and a host reboot.

on-failure:5 has its own place — a batch job that should retry a handful of times and then stay dead so that somebody notices. The one to avoid is an unbounded always on something that crash-loops in under a second: Docker backs the restarts off, but the log volume and the image-pull churn are still real.

What we will NOT do

This course will not:

Pretend docker run -v /:/host is fine because “it’s a lab”.
Show you how to skip healthchecks to “save time”.
Treat latest as a valid production tag.
Encourage disabling security features for convenience.
Promise that Kubernetes will fix your Compose problems.

It will, instead, give you the operational foundation to run Docker the way it is run in environments that take production seriously.

What if nobody configures log rotation?

The default logging driver is json-file, and with no options it has no size limit. A container writing a modest 5 MB of logs an hour fills 100 GB in under two years, and a container in a crash loop with a stack trace on every restart can do it in a weekend.

The failure is not “logs are large”. When the filesystem holding /var/lib/docker fills:

containers that stop cannot start, because their writable layer cannot be created;
image pulls fail, so you cannot deploy the fix;
the daemon itself starts erroring on writes;
and docker system prune — the reflex — deletes the evidence you need without necessarily freeing the log files, which belong to running containers and are not reclaimable while those containers exist.

Set it daemon-wide so that every future container inherits it, rather than per container where the next deployment forgets:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

This applies to containers created after the change; existing containers keep the options they were created with. Verify with the audit command above, not by reading daemon.json, because those two answer different questions.

The number to watch is not log size but free space: df -h /var/lib/docker, alerted at 80%, is worth more than any amount of retrospective analysis.

What if a container has no memory limit and the host runs out?

Without --memory, a container may use all of the host’s RAM. When the host exhausts memory the kernel OOM killer chooses a victim by score, and it does not know or care which process is your production database — it tends to pick the largest resident process, which frequently is your production database rather than the container that caused the problem.

The blast radius of one leaking container is therefore the whole host, and the symptom appears somewhere else entirely. docker inspect --format '{{.State.OOMKilled}}' on the container that died tells you it was killed; it does not tell you who caused it.

A memory limit converts a host-wide incident into a container-wide one: the offender is killed inside its own cgroup, exit code 137, its restart policy handles it, and everything else keeps serving. That containment is the entire argument for limits, and it is why they belong on every container rather than only on the ones you suspect.

Production vs developer defaults

Auditing a host against the list

The five questions every production decision must answer

The five things production Docker always does

What we will NOT do

Knowledge check