Docker host design — single host, multi host, edge cases · Docker & Containers

A Docker host is a Linux machine running dockerd, containerd, runc and some containers. Designing one well is mostly two decisions: where the growing data goes, and how you keep reaching the host when the workload misbehaves.

Hardware sizing, as a starting point

Workload	CPU	RAM	Disk	Network
Small (1–10 services)	4 cores	8 GB	100 GB SSD	1 GbE
Medium (10–50 services)	16 cores	64 GB	500 GB NVMe	10 GbE
Large (50–200 services)	32+ cores	128+ GB	1+ TB NVMe	10 GbE+
Very large (200+)	Multiple hosts	Multiple hosts	Network storage	10 GbE+

Treat that as an opening bid, not an answer — the next lesson does the arithmetic properly. Any x86-64 CPU from 2017 or later has the kernel features Docker needs (cgroup v2, user namespaces, seccomp, and either AppArmor or SELinux). Kernel version matters more than CPU generation: overlay2 needs 4.0 or newer, and cgroup v2 as the sole hierarchy wants 5.8 or newer to be pleasant.

Storage layout is the decision that matters

Everything Docker accumulates lives under one directory: /var/lib/docker. Images, container writable layers, volumes, build cache and — critically — container logs, all in one place, all growing.

What is actually under /var/lib/docker

/var/lib/docker/
├── overlay2/            image layers and container writable layers
│   └── <layer-id>/
│       ├── diff/        this layer's own files
│       ├── link         short name used in the mount option string
│       └── lower        the chain of layers beneath it
├── containers/
│   └── <container-id>/
│       └── <id>-json.log   the json-file log, unbounded by default
├── volumes/
│   └── <volume-name>/_data
├── image/overlay2/      the metadata database mapping digests to layers
└── buildkit/            build cache

An image is a stack of read-only layers. Starting a container adds one writable layer on top, and the union of the two is what the process sees. The mechanism is an overlay mount with lowerdir (the image layers, colon-separated, bottom last), upperdir (the container’s writable layer) and merged (what gets bind-mounted as the container root). docker inspect shows all four under GraphDriver.Data.

Two accounting consequences follow, and both surprise people:

Layers are shared, so the sum of image sizes is not disk usage. Ten images built FROM the same base share that base exactly once. docker images shows each one’s full logical size; docker system df shows what is actually on disk plus what is reclaimable.

A deleted file still costs space. Removing a file that exists in a lower layer does not free anything; overlayfs records a whiteout — a character device with major and minor 0 — in the upper layer, which hides the file while the original bytes stay in the layer below. This is why RUN apt-get install ... && rm -rf /var/lib/apt/lists/* only helps when both halves are in the same layer, and why an image with a secret in an early layer still contains it after a later RUN rm.

The host that went read-only because a container was chatty

The failure looks like a total host failure and its cause is one service.

A container starts logging a stack trace on every request — a dependency timing out, an exception loop, a debug flag left on. The json-file driver’s default is unbounded: no max-size, no max-file. It writes to /var/lib/docker/containers/<id>/<id>-json.log until the filesystem is full.

If /var/lib/docker is on the root filesystem, and it usually is, the consequences are not confined to Docker. journald cannot write. sshd cannot create a session file for a new login, so you may not be able to get in. The package manager cannot run. Every other container starts failing writes at the same instant, so the monitoring you would use to diagnose it is also broken, and the dashboard shows twenty services failing simultaneously — which reads like a host or network event, not like one noisy container.

The first diagnosis is almost always wrong for that reason. It looks like infrastructure. df -h is the command that ends the confusion in two seconds, and it is rarely the first one anybody runs.

Two design decisions prevent it outright:

Give /var/lib/docker its own filesystem. Then a full Docker directory is a Docker problem: the host stays up, SSH still works, journald still records what happened, and you can log in and fix it.
Bound the log driver globally in daemon.json, so a service that forgets logging: still inherits a limit.

Configuration changedaemon.json— /etc/docker/daemon.json — the baseline every production host should have before it runs anything.

{
"log-driver": "json-file",
"log-opts": { "max-size": "10m", "max-file": "3" },
"live-restore": true,
"default-address-pools": [
  { "base": "172.20.0.0/14", "size": 24 }
]
}

Three deliberate choices there:

log-opts caps every container at 30 MB unless it overrides them. This applies only to containers created after the change; existing ones keep the settings they were created with.
live-restore keeps containers running when dockerd stops, which turns a daemon upgrade into a control-plane blip instead of an outage.
default-address-pools stops Docker allocating bridge subnets that collide with your corporate network. The default pools include 172.17.0.0/16 upward, and the day a new Compose network takes the same range as your VPN, every container loses access to an internal service in a way that is very hard to attribute.

live-restore is narrower than it sounds

It is genuinely useful and it has three documented limits worth knowing before you rely on it:

It works across patch upgrades of the daemon, not across major version jumps, and not if you skip releases.
It only restores if the daemon-level options — storage driver, bridge addresses — did not change. Change any of those and you may have to stop the containers by hand.
While the daemon is down, container stdout goes into a FIFO whose default buffer is 64 KB. A chatty container fills it, blocks on write, and effectively hangs. If the daemon is down for a long time you have to restart it to drain the buffers.

So it covers “restart the daemon” well and “the daemon has been dead for an hour” poorly. It is not a substitute for a restart policy, which is what brings containers back after the host reboots.

Verify the backing filesystem, do not assume it

overlay2 on XFS requires d_type support, which means the filesystem must have been formatted with ftype=1. Older RHEL and CentOS installers defaulted to ftype=0, and the failure mode is not a clean refusal — it is subtle corruption of overlay behaviour that surfaces weeks later.

Read-only / Safestorage check— The two lines that must be right before this host runs anything.

$ docker info --format 'Driver={{.Driver}} Backing={{index .DriverStatus 0 1}} d_type={{index .DriverStatus 1 1}}' && findmnt -no SOURCE,FSTYPE,SIZE,USE% /var/lib/docker

Driver=overlay2 Backing=xfs d_type=true
/dev/mapper/vg0-docker xfs   500G  38%

Illustrative output

d_type=false means reformat, because there is no fix short of it. And findmnt returning nothing for /var/lib/docker means it is on the root filesystem — which is the condition the callout above describes.

Changing the storage driver later makes every existing image and container inaccessible. Get this right at build time; there is no in-place migration.

Filesystem choices

NVMe over SATA SSD for anything doing builds. Layer extraction is metadata-heavy and small-file-heavy, which is where the gap is widest.
noatime on the Docker filesystem removes a metadata write per file read. Free.
ext4 or XFS. Both are fine. btrfs and zfs have their own Docker storage drivers with different tradeoffs; if you are not deliberately choosing one, you want overlay2.
Network storage for volumes, not for /var/lib/docker itself. NFS under the whole directory produces overlayfs behaviour nobody wants. Mount network storage at specific volume paths instead.

Separating the management plane

The point of separating planes is that when the data plane is saturated — by traffic, by a runaway container, by a disk that just filled — you can still reach the host to fix it.

A dedicated management interface. SSH, monitoring exporters and any out-of-band access bind the management address only, so a flooded public interface does not lock you out.
Published ports bind an explicit address. -p 203.0.113.10:443:443, never the bare -p 443:443 that binds every interface including the management one.
The daemon is never on TCP. DOCKER_HOST=ssh://... for remote access; an open tcp:// daemon socket is unauthenticated root.
Reserve resources for the host. If the sum of container limits equals physical RAM, there is nothing left for sshd to fork with. Leave headroom deliberately rather than discovering it is absent during an incident.

flowchart LR
  Internet((Internet)) --> FW[Host firewall<br/>DOCKER-USER rules]
  FW --> EXT[eth0 203.0.113.10]
  EXT --> Proxy[Reverse proxy<br/>TLS terminates here]
  Proxy --> AppNet[app-net bridge]
  AppNet --> A[Service A]
  AppNet --> B[Service B]
  AppNet --> DataNet[data-net<br/>internal: true]
  DataNet --> DB[(Database)]
  VPN((VPN / jump host)) --> MGMT[eth1 192.0.2.10]
  MGMT --> HostSvc[sshd, node_exporter,<br/>dockerd]

Failure planning

For each of these, the deliverable is not the analysis. It is a written recovery step somebody can follow at 03:00.

Failure	What happens	What you need in place
Host reboots	Only containers with a restart policy return	`restart: unless-stopped` everywhere, and a boot-time check
Daemon restarts	Containers die, unless `live-restore`	`live-restore: true`, and knowing its limits
A container leaks memory	Its own cgroup OOM kill, or a host-wide one if unlimited	Memory limits on every service
`/var/lib/docker` fills	Every container fails writes at once	Separate filesystem, log caps, alert at 80%
Registry unreachable	Redeploys fail; running containers unaffected	Pre-pulled images, a pull-through mirror
Public NIC saturated	Host unreachable if management shares it	Separate management interface
Disk fails	Everything	Volumes backed up somewhere else, restore tested

The last row is the one that gets written down and never tested. A backup you have not restored from is a hypothesis.

What if /var/lib/docker is 95% full right now?

Work in this order, because the first two steps are safe and the third is not.

First, find out what is using it. docker system df splits usage into images, containers, local volumes and build cache, with a RECLAIMABLE column. Add -v for the per-object breakdown. Nine times out of ten the answer is build cache or logs, and both are safe to reclaim.

Second, deal with logs, which system df does not count. Container logs live under containers/ and are invisible to docker system df, so a host can be full with system df reporting only a few gigabytes in use. Find them directly:

Read-only / Safelog sizes— The usage docker system df does not show you.

du -sh /var/lib/docker/containers/*/*-json.log 2>/dev/null \
| sort -rh | head -10

Truncating a live log file with truncate -s 0 frees the space immediately and is safe, because the daemon holds the file open and keeps appending at the new offset. It is a stopgap: without a max-size the file grows straight back, so fix the driver config and recreate the container.

Third, and only then, prune. docker builder prune reclaims build cache and is the safest large win. docker image prune -a removes every image not used by a container — including the images you would need to recreate a container during an incident, and including anything you cannot re-pull if the registry is also unreachable. Never run docker system prune --volumes; it reaches named volumes, which is the one Docker object whose contents exist nowhere else.

Give /var/lib/docker its own filesystem, sized from the next lesson.
Verify docker info reports overlay2 with d_type=true before deploying anything.
Mount that filesystem noatime.
Set log-opts with max-size and max-file in daemon.json.
Set live-restore: true, and know that it covers patch upgrades only.
Set default-address-pools so bridge subnets cannot collide with your network.
Bind SSH and exporters to the management address; publish container ports to an explicit address.
Alert on /var/lib/docker at 80%, not at 95%.
Write the recovery step for every row of the failure table, and test the restore.