Log rotation and disk consumption · Docker & Containers

Docker’s default logging driver is json-file. Its default max-size is -1, which means unlimited, and its default max-file is 1, which means there is nothing to rotate into. Put those two together and you get the sentence that this whole lesson exists to make unforgettable:

A stock Docker installation writes container logs to disk with no upper bound whatsoever.

Not “a generous bound”. Not “a bound you should tune”. No bound. A container that logs one kilobyte per second writes 86 MB a day, 2.6 GB a month, and 31 GB a year, into a file nobody is watching, until the filesystem holding /var/lib/docker is full and every workload on the host starts failing at once.

This is the most common self-inflicted Docker outage there is, and it is entirely a configuration default.

The defaults, stated exactly

Read-only / Safejson-file defaults— What you get if you never touch a log-opt.

$ docker info --format '{{.LoggingDriver}}'

json-file

Illustrative output

Option	`json-file` default	`local` default
`max-size`	`-1` (unlimited)	`20m`
`max-file`	`1`	`5`
`compress`	`false`	`true`

The right-hand column is the interesting one. The local driver — which is also built in, also stores logs on the host, and is also readable with docker logs — ships with rotation on by default: five files of 20 MB each, compressed, for a hard ceiling of 100 MB per container. Same engine, same maintainers, opposite default.

The reason json-file is unbounded is compatibility. Its on-disk format is a documented, greppable, one-JSON-object-per-line file that a whole ecosystem of log shippers reads directly, and capping it by default would have silently truncated those pipelines on upgrade. That is a defensible engineering decision and a terrible operational one, and you inherit the consequences.

Where the files actually are

Every json-file log is one file per container, under the container’s own directory in the daemon’s state tree:

Read-only / Safelocate the log— Resolve a container name to its log file and measure it.

CONTAINER=web

CID=$(docker inspect --format '{{.Id}}' "$CONTAINER")
LOGFILE="/var/lib/docker/containers/$CID/$CID-json.log"

sudo ls -lh "$LOGFILE"*
sudo du -ch /var/lib/docker/containers/"$CID"/ | tail -1

The trailing * matters: it picks up -json.log.1, -json.log.2 and, if compress is on, -json.log.1.gz. Measuring only the active file on a rotating container reports a fraction of the real consumption.

There is also a much faster way to find the offender across the whole host, and it is the command worth memorising:

Read-only / Saferank by log size— The five biggest container log directories on the host.

$ sudo du -sh /var/lib/docker/containers/* | sort -h | tail -5

12M	/var/lib/docker/containers/1f2a9c4e8b31...
48M	/var/lib/docker/containers/7d3e5a90c2ff...
104M	/var/lib/docker/containers/b81c40de77a2...
1.1G	/var/lib/docker/containers/44ab9f0e13cd...
37G	/var/lib/docker/containers/9c7e2b58af04...

Illustrative output

Note what this is not. It is not docker system df. Container log files live outside the container’s writable layer, so the Containers row of docker system df — which counts writable layers only — reports kilobytes for the container in the last line of that output. Docker’s own disk accounting cannot see the 37 GB. You have to look with du.

Configuring rotation

Two places, and you need to understand the difference before you touch either.

Daemon-wide default

/etc/docker/daemon.json sets the default for containers created after the daemon reloads:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3",
    "compress": "true"
  }
}

Every value in log-opts is a string, including the numeric ones. "max-file": 3 without quotes is a type error and the daemon will refuse to start with it. That is a five-minute outage if you edit this file on a production host and restart without validating first.

Service impact possibleapply daemon config— Validate the JSON, then reload. Reload re-reads log-opts without restarting containers; restart is only needed if you changed something reload does not cover.

sudo python3 -m json.tool /etc/docker/daemon.json > /dev/null && echo 'JSON ok'

sudo systemctl reload docker

docker info --format '{{.LoggingDriver}}'

Per container

A per-container setting overrides the daemon default and is fixed at creation time:

Configuration changeper-container rotation— Override the daemon default for one noisy workload.

docker run -d --name chatty \
--log-driver json-file \
--log-opt max-size=50m \
--log-opt max-file=2 \
nginx:1.27-alpine

In Compose, the same thing lives under logging::

services:
  chatty:
    image: nginx:1.27-alpine
    logging:
      driver: json-file
      options:
        max-size: "50m"
        max-file: "2"

The half-fix that everybody does

Here is the failure that turns a fixed problem back into an outage.

Editing daemon.json does not cap the containers you already have

Log driver configuration is baked into the container at creation time. The daemon documentation is explicit: changing the default logging driver or its options only affects containers created after the change. Existing containers retain what they were created with.

So the usual sequence — disk fills, engineer adds max-size to daemon.json, reloads the daemon, watches df for a few minutes, declares victory — leaves every currently running container writing to an uncapped file. The disk fills again a week later and the second incident is worse, because now everyone believes rotation is configured.

The fix requires a second step: recreate the containers. Not restart — docker restart reuses the existing container object and its existing log config. You need docker rm and docker run again, or docker compose up -d --force-recreate, which builds a new container object that picks up the new default.

Which is why the verification below matters more than the configuration above.

Verification that can fail

Do not check that you edited a file. Check what the container is actually using, which is a different question with a different answer:

Read-only / Safeper-container audit— The effective log configuration of every running container. Any row with an empty options map is uncapped.

$ docker ps -q | xargs -r docker inspect \
--format '{{.Name}} {{.HostConfig.LogConfig.Type}} {{.HostConfig.LogConfig.Config}}'

/api        json-file map[max-file:3 max-size:10m]
/worker     json-file map[max-file:3 max-size:10m]
/legacy-etl json-file map[]
/redis      json-file map[max-file:3 max-size:10m]

Illustrative output

/legacy-etl is the finding. Its Config map is empty, so it inherited the daemon default as it stood when the container was created — before you fixed it. Recreate that one.

This command distinguishes working from broken, which “run it and see” does not. Wire it into a nightly check and the second incident never happens.

How the daemon rotates, and why the fd matters

The json-file driver is a logger.Logger implementation inside the daemon process. For each container it holds an open file descriptor on <id>-json.log and appends one JSON object per log line — the message, the stream (stdout or stderr), and an RFC3339Nano timestamp.

Rotation is a size check performed by the daemon, not by logrotate, not by copytruncate, and not by anything you can trigger externally. After a write pushes the file past max-size, the driver renames -json.log to -json.log.1 (shifting existing numbered files up, dropping anything past max-file), creates a fresh -json.log, and reopens its descriptor onto the new inode. If compress is on, the rotated file is gzipped in the background.

Two consequences follow directly from “the daemon holds an open fd”:

rm on the active log frees nothing. Unlinking a file whose descriptor is still open removes the directory entry but not the inode. The blocks stay allocated until the last descriptor closes, which here means until the daemon closes it — on container stop, or on daemon restart. df will not move. Worse, the daemon keeps happily appending to a file with no name, so the space keeps growing and you can no longer even measure it with du on the path.

truncate -s 0 works and is safe. It keeps the inode, so the descriptor stays valid; it just releases the blocks. The daemon’s next write goes to the same inode. You lose the log contents, and because the driver appends at the current offset you may briefly see a sparse region at the head of the file, but the container never notices and the space comes back immediately.

Destructivereclaim space now— Discards the container's log history. Capture anything you need first — this is unrecoverable.

CONTAINER=chatty
CID=$(docker inspect --format '{{.Id}}' "$CONTAINER")
LOGFILE="/var/lib/docker/containers/$CID/$CID-json.log"

# Keep the tail as evidence before you destroy it.
sudo tail -c 2M "$LOGFILE" > "/var/tmp/$CONTAINER-last2M.json"

# Release the blocks without breaking the daemon's file descriptor.
sudo truncate -s 0 "$LOGFILE"

df -h /var/lib/docker

Planning log volume

For a fleet of N containers each capped at C bytes, the worst case on-disk cost is the part you can actually guarantee:

Ceiling = N * max-size * max-file

Fifty containers at max-size=10m, max-file=3 is a hard ceiling of 1.5 GB regardless of how loudly anything logs. That is the number to put on the capacity plan, because it is the only one an application team cannot blow through by shipping a debug log level.

Retention-based sizing — “N containers at L MB/day retained R days” — belongs to your central store, not to the host. The host’s job is a fixed, guaranteed ceiling; the central store’s job is retention. Confusing the two is how people end up sizing a host partition against a retention policy and then being surprised when a single misbehaving service consumes the whole budget.

What happens if you set max-size but the disk fills anyway?

Three explanations, in the order you should test them.

The containers predate the change. Run the per-container audit above. An empty Config map is the answer, and the fix is recreation.

It is not the logs. du -sh /var/lib/docker/containers/* gives you the number directly. If the biggest entry is 40 MB, your rotation is working and the disk is going somewhere else — images, build cache, volumes, or a bind mount that is not a Docker object at all. docker system df covers the first three; df -h on the host covers the fourth.

A single log line is larger than max-size. Rotation happens at line boundaries, so a service that emits a 200 MB stack trace as one line will write all 200 MB into a file capped at 10 MB before the driver gets a chance to roll it. Rare, but it is a genuine hole in the guarantee, and the fix is in the application.

Undoing it

Rotation settings have no destructive residue. To back out, remove log-opts from daemon.json, reload, and recreate. The already-rotated files are gone permanently — rotation deleted them, and nothing in Docker retains a copy — which is the one irreversible part of the change and the reason to have central collection in place before you turn aggressive rotation on.

Knowledge check

Why this matters

Disk-full on a Docker host is not a subtle failure. Containers get ENOSPC on their next write and some of them corrupt rather than crash. Image pulls fail. The daemon cannot write container state, so docker ps hangs or lies. The journal stops recording, so the evidence of what happened is missing from exactly the window you need. And your own shell — history, editor swap files, the redirect you were about to use to capture output — stops working too.

All of it traces back to a driver default that means “unlimited”, on a subsystem nobody looks at until it breaks, in a directory that Docker’s own df command cannot see.

Two lines in daemon.json and one recreation of your workloads removes an entire class of incident permanently. It is the highest-value ten minutes available anywhere in Docker operations, and the reason it keeps not getting done is that the failure takes months to arrive and the fix takes minutes — which is precisely the shape of problem that operational discipline exists to solve.