Log drivers at the daemon level · Docker & Containers

The default log driver is json-file. Every container writes to /var/lib/docker/containers/<id>/<id>-json.log, and the documented default for max-size is -1, meaning unlimited. That single default is responsible for more Docker outages than any other setting on the host.

It is worse than “a container fills the disk”, because the file lives under /var/lib/docker. When that filesystem fills, the daemon cannot write container metadata, image layers, or its own state — so you do not lose one container, you lose the host.

The drivers

Driver	Where logs go	`docker logs` reads it?
`json-file`	Local JSON file under `/var/lib/docker/containers/`	Yes
`local`	Local file in a compact custom format	Yes
`journald`	The systemd journal	Yes
`none`	Nowhere; logs are discarded	No — nothing to read
`syslog`	A syslog facility on the host	Via the dual-logging cache
`fluentd`	Fluentd forward input	Via the dual-logging cache
`gelf`	A GELF endpoint such as Graylog or Logstash	Via the dual-logging cache
`awslogs`	Amazon CloudWatch Logs	Via the dual-logging cache
`gcplogs`	Google Cloud Logging	Via the dual-logging cache
`splunk`	Splunk HTTP Event Collector	Via the dual-logging cache
`etwlogs`	Event Tracing for Windows	Windows only

For a Linux host with no central log pipeline, journald and local are both better defaults than json-file, for different reasons: journald gives you one place to correlate container output with kernel and unit messages, and local gives you a compact on-disk format with sane rotation defaults and no journald configuration to get wrong.

Switching the default driver

Configuration changedaemon.json— Daemon-wide default. Applies to containers created after the change only.

{
"log-driver": "journald",
"log-opts": {
  "tag": "{{.Name}}"
}
}

Two things about applying this are commonly got wrong.

log-driver is not a reloadable key. It is not on the documented list of options the daemon applies on SIGHUP. systemctl reload docker will exit 0 and change nothing. You need systemctl restart docker, and with live-restore enabled that costs no container downtime.

Existing containers keep their original driver. The driver is recorded in a container’s specification at create time, like everything else in the spec, and restarting a container replays that spec. To migrate a container you must recreate it.

Service impact possiblemigrate— Find containers still on the old driver. Recreating them is the only migration.

docker ps -a --format '{{.Names}}' | while read -r c; do
  drv=$(docker inspect --format '{{.HostConfig.LogConfig.Type}}' "$c")
  echo "$c $drv"
done | grep -v ' journald$'

docker restart will not do it. Under Compose, docker compose up -d will recreate containers whose effective configuration changed, which is the practical migration path for a stack.

Rotation that actually applies

For json-file, both options are needed, and one of them has a trap:

Configuration changerotation— max-file is only effective when max-size is also set.

{
"log-driver": "json-file",
"log-opts": {
  "max-size": "10m",
  "max-file": "3",
  "compress": "true"
}
}

Option	Default	Notes
`max-size`	`-1` (unlimited)	The whole problem. Always set it.
`max-file`	`1`	Only effective when `max-size` is also set.
`compress`	`false`	Compresses rotated files, not the active one.

Setting max-file: "5" alone is a config change that looks like it did something and did nothing at all, because with no max-size the file never rolls. That is a real audit finding on real fleets.

Budget the worst case honestly: the cap is max-size multiplied by max-file per container, and the active file counts. Two hundred containers at 10m/3 is roughly 6 GB of log before compression, on the same filesystem the daemon needs to function.

Per-container overrides use the same names:

Configuration changeper container— Override for one noisy service that genuinely needs more history.

docker run -d --name web \
--log-driver json-file \
--log-opt max-size=50m \
--log-opt max-file=2 \
nginx:1.27

The rotation is set, and the disk fills anyway

You set max-size and max-file in daemon.json, restarted the daemon, and three weeks later /var/lib/docker is full. Three distinct causes, all of which get misdiagnosed as “rotation is not working”:

Containers created before the change. The log config is create-time state. Every container that predates the restart still has max-size unset. Check the containers, not the daemon: docker inspect --format '{{.HostConfig.LogConfig}}' on the biggest offender will show an empty options map.

Stopped containers still hold their logs. Removing a container deletes its log directory; stopping it does not. A CI host with four hundred exited containers is holding four hundred log files, and docker ps shows none of them.

It is not the logs at all. docker system df will tell you in one command whether the space is images, volumes, build cache or containers. Reaching for du on /var/lib/docker first is the slow path, and on a large overlay2 tree du can take a very long time.

Read-only / Safefind the space— Ask the daemon before you ask the filesystem.

$ docker system df

TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          48        9         14.2GB    11.1GB (78%)
Containers      312       9         6.4GB     6.3GB (98%)
Local Volumes   21        7         3.1GB     1.9GB (61%)
Build Cache     190       0         8.7GB     8.7GB

Illustrative output

A Containers size that large with only nine active is almost always logs held by exited containers.

Read-only / Safebiggest logs— Rank container log files by size, without walking the whole data root.

sudo du -h /var/lib/docker/containers/*/*-json.log 2>/dev/null \
| sort -rh | head -10

Delivery mode — the failure nobody expects

Docker supports two delivery modes via the mode log option: blocking, which is the default, and non-blocking. In blocking mode the container’s write to stdout does not complete until the log driver has accepted the message.

The application is not hung. The log backend is.

This is the single most misdiagnosed logging incident there is, and it only happens with a remote driver.

Your Fluentd aggregator goes down, or the network path to it gets slow, or its disk fills. Docker’s fluentd driver is in blocking mode, so it stops accepting messages. Every container write to stdout now blocks. The application threads that were writing a log line stop, then the ones behind them stop, and within a minute the service is not serving — while being completely healthy by every measure you normally look at.

The symptoms are perfectly designed to mislead:

CPU is near zero. Memory is flat. No OOM, no restarts.
docker ps says Up 6 days. The container never exited.
Thread dumps show threads parked on a write to fd 1.
The application’s own logs show nothing after a certain timestamp — because writing them is what is blocked.

Everyone looks at the database first. The log pipeline is not on anybody’s dependency diagram, because logging is supposed to be observability, not a dependency in the request path. With blocking mode it is one.

The fix, before it happens, is one option:

Configuration changenon-blocking— Drop log lines under back-pressure instead of stalling the application.

{
"log-driver": "fluentd",
"log-opts": {
  "fluentd-address": "logs.example.com:24224",
  "mode": "non-blocking",
  "max-buffer-size": "4m"
}
}

max-buffer-size sets the intermediate ring buffer used in non-blocking mode; its default is 1m. When the buffer is full, messages are dropped. That is a deliberate trade: you lose log lines rather than losing the service. For almost every workload that is the right trade, and it should be a conscious decision made in advance rather than at 03:00.

Where the log line actually goes

The container’s PID 1 does not write to a file. It writes to fd 1 and fd 2, which the runtime wired to a pipe (or a PTY, if the container has a TTY) whose read end is held by containerd-shim-runc-v2. The shim forwards to the daemon, and the daemon’s logging plugin for that container writes it wherever the driver says.

Three consequences follow directly from that chain:

Everything in the chain is per-line-ish and lossy at the edges. A process that writes a partial line and then crashes can leave that fragment unflushed. Applications that buffer stdout — Python without -u, many JVM logging configurations — will lose their tail on a hard kill, and no log driver can recover it.
The blocking chain reaches all the way back to the application. The pipe has a finite kernel buffer, typically 64 KB. When the driver stops draining and the pipe fills, write(2) in the application blocks. That is the mechanism behind the incident above; it is not a Docker quirk, it is how pipes work.
The daemon is in the path. While dockerd is restarting, nothing drains the shim’s buffer. This is a real, if brief, reason to keep daemon restarts short on a chatty host.

Dual logging — how `docker logs` still works

When you configure a remote driver, Docker also writes to a local local-driver cache, so docker logs keeps working. That is dual logging, and it is on by default.

log-opt	Default	Meaning
`cache-disabled`	`"false"`	Set `"true"` to turn the local cache off entirely
`cache-max-size`	`"20m"`	Size of each cache file before rotation
`cache-max-file`	`"5"`	Number of cache files retained
`cache-compress`	`"true"`	Compress rotated cache files

Those defaults mean roughly 100 MB per container of local cache on top of whatever you ship remotely. On a host with hundreds of containers that is worth budgeting, and cache-disabled is the lever — with the cost being this:

Read-only / Safecache disabled— The exact error when the driver cannot serve reads and the cache is off.

$ docker logs web

Error response from daemon: configured logging driver does not support reading

Illustrative output

That message is unambiguous and worth memorising, because it turns a confusing “why can’t I see the logs” into a one-line answer: this container ships its logs somewhere else, and the local cache is disabled.

journald specifics

journald avoids the unbounded-file problem by handing rotation to the journal, which is configured in /etc/systemd/journald.conf:

[Journal]
Storage=persistent
SystemMaxUse=2G
SystemKeepFree=4G
MaxFileSec=1month

Docker sets journal fields you can filter on: CONTAINER_ID (truncated to 12 characters), CONTAINER_ID_FULL, CONTAINER_NAME (the name at start time), CONTAINER_TAG / SYSLOG_IDENTIFIER, and IMAGE_NAME.

Read-only / Safejournalctl— Query container output with the journal's own tooling, including time windows and JSON.

CONTAINER=web

# All output from one container, by the name it had when it started.
sudo journalctl CONTAINER_NAME="$CONTAINER"

# A time window, following, with full structured fields.
sudo journalctl CONTAINER_NAME="$CONTAINER" --since "-30 min" -f -o json-pretty

# Everything from one image across every container that ran it.
sudo journalctl IMAGE_NAME=nginx:1.27 --since today

That last query is the reason to pick journald. Correlating “every container that ran this image” or “container output interleaved with the kernel’s OOM message, in timestamp order” is a single command, and it is not possible at all with per-container JSON files.

Two journald defaults that lose your logs

Storage=auto may mean volatile. With the default, journald writes to /var/log/journal only if that directory already exists; otherwise it writes to /run/log/journal, which is tmpfs. On a host where nobody created the directory, every log is discarded at reboot — including the logs from the crash that caused the reboot. Set Storage=persistent explicitly, or run sudo mkdir -p /var/log/journal && sudo systemd-tmpfiles --create --prefix /var/log/journal.

Rate limiting drops messages silently. journald defaults to 10,000 messages per 30 seconds per service; beyond that, further messages in the interval are dropped and a note about how many were dropped is written. A container in a crash loop or an application logging every request will exceed that, and the gap in your logs is exactly the window you need. Tune RateLimitIntervalSec and RateLimitBurst, or accept that journald is not the right driver for a high-volume service.

Verification that can fail

Reading the config file back proves nothing. Ask the daemon and the container.

Read-only / Safedaemon default— What new containers will get.

$ docker info --format 'default driver: {{.LoggingDriver}}'

default driver: journald

Illustrative output

Read-only / Safeaudit— Fail if any running container is unrotated or on the wrong driver.

FAIL=0
for c in $(docker ps -q); do
  name=$(docker inspect --format '{{.Name}}' "$c")
  drv=$(docker inspect --format '{{.HostConfig.LogConfig.Type}}' "$c")
  size=$(docker inspect --format '{{index .HostConfig.LogConfig.Config "max-size"}}' "$c")
  if [ "$drv" = "json-file" ] && [ -z "$size" ]; then
      echo "UNROTATED: $name" >&2
      FAIL=1
  fi
done
[ "$FAIL" -eq 0 ] && echo "all running containers have bounded logs"
exit "$FAIL"

That is verification: it distinguishes a working configuration from a configuration that was written down, and it exits non-zero so a pipeline can gate on it.

What happens if you set the driver to `none`?

Container output is discarded at the daemon, before any file is written. docker logs returns nothing at all — not an error about an unsupported driver, just no output, which is a strictly worse diagnostic experience.

There is a narrow legitimate case: a container whose application already ships its own logs, over its own transport, to somewhere you trust, and whose stdout is pure noise you are paying to store. Setting none on that one container is reasonable.

Setting none as the daemon-wide default to fix a disk-space problem is not. It solves the symptom by destroying the evidence, and the next incident on that host has no container output at all — including the crash message that would have told you what happened. If the volume is genuinely the problem, the answers are rotation, non-blocking mode with a bounded buffer, or fixing the application that logs every request at INFO.

Why this matters

Logging looks like the least interesting thing on a Docker host until you notice that two of its defaults are actively hostile. json-file with an unlimited max-size will eventually fill a disk, and a full /var/lib/docker is a daemon outage, not a logging outage. Blocking delivery mode puts your log aggregator in the request path of every container on the host, so a Fluentd problem becomes an application problem while every dashboard says the application is fine.

Neither is hard to fix, and both take one line in daemon.json. What is hard is recognising them during the incident, because in both cases the evidence you would normally use — the logs — is the thing that failed. That is why the audit above matters more than any of the configuration in this lesson: it is the check that runs before the incident, on every host, and fails loudly.

The drivers

Switching the default driver

Rotation that actually applies

Delivery mode — the failure nobody expects

Dual logging — how docker logs still works

journald specifics

Verification that can fail

Knowledge check

Dual logging — how `docker logs` still works