Docker Logs

A production container has been logging for six months. The on-call engineer opens the host, runs df -h /var/lib/docker, and finds 30 GiB consumed by /var/lib/docker/containers/<id>/<id>-json.log. The container’s application has been writing a debug line per request for six months because someone enabled debug logging to investigate a ticket and never turned it off. The json-file driver kept every line because no rotation was configured. The host disk filled slowly; the container kept logging.

docker logs works because the Docker daemon captures every container’s stdout and stderr through a log driver, writes the result somewhere, and serves it back. This lesson is about the driver, the destination, and the right shape for a production host.

What it is

docker logs <container> reads the captured stdout and stderr of a container from wherever the log driver has been writing. The captured stream is a faithful record of the file descriptors the container’s main process wrote to. The driver is pluggable; the default on a fresh install is json-file. The available drivers cover local disk, the systemd journal, syslog, fluentd, gelf, splunk, awslogs, and etwlogs.

    container main process
        |
        |  fd 1 (stdout) and fd 2 (stderr)
        v
    Docker daemon log driver
        |
        +---- json-file  -> /var/lib/docker/containers/<id>/*.log
        +---- journald   -> systemd journal
        +---- syslog     -> syslogd over the network
        +---- fluentd    -> fluentd over the network
        +---- loki       -> Grafana Loki (via the Loki plugin)
        +---- splunk     -> Splunk HEC
        +---- awslogs    -> CloudWatch Logs
        ...

The driver is a daemon-level setting in /etc/docker/daemon.json or a per-container override in docker run --log-driver=<name>. Every driver is configured by the same shape: a list of key-value options passed through --log-opt key=value.

Why a sysadmin cares

Container logs are the most operationally important signal at 03:00. Metrics tell you that something is slow; logs tell you what the application is actually doing. The shape of the log pipeline decides whether you can find the relevant line, whether the host disk fills, and whether a single container can drown the rest of the platform in noise.

The four production failure shapes that come from the log driver:

• Disk fills because rotation is off. The default json-file configuration has no rotation; a chatty container fills the host disk in days.
• Logs reach the journal but no Loki. A team that moves to journald and forgets to point a shipper at the journal has logs that survive only on the host.
• A single high-cardinality label blows up Loki. A team that puts a request UUID into a Loki label pays for it several times over; Loki is label-indexed and a high-cardinality label multiplies the storage.
• Structured logs are missed because the application emits unstructured text. Loki’s pipeline assumes structured records; a free-text log line is searchable but cannot be filtered by field.

How it works

The Docker daemon forks a log driver process for every container. The driver reads the container’s captured stdout and stderr through a pipe; the containerised process writes to its own stdout and stderr as if it were attached to a terminal. The Docker daemon’s pty layer captures the writes and pipes them to the driver.

The driver’s job is to serialise the stream and write it to its configured destination. For json-file the destination is a file on the host; for journald it is the systemd journal; for fluentd it is a fluentd server. Each line carries a small header that identifies the stream (stdout or stderr) and the container.

    {"log":"2026-01-15 12:34:56 INFO request handled\n",
     "stream":"stdout","time":"2026-01-15T12:34:56.123Z"}

    {"log":"2026-01-15 12:34:57 ERROR connection refused\n",
     "stream":"stderr","time":"2026-01-15T12:34:57.456Z"}

The time field is set by the daemon at the moment it captures the line, not by the application. If the application emits a timestamp in its log line, that is a separate field inside log. Loki’s pipeline parses both.

How to configure it

Three configuration shapes cover most production needs.

Daemon-level configuration for json-file with rotation

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "5",
    "labels": "service,env,version",
    "tag": "{{.Name}}/{{.ID}}"
  }
}

# CONFIGURATION: reload the daemon. New containers pick up
# the new driver; running containers keep their old driver.
sudo systemctl reload docker

# READ-ONLY: confirm the running daemon has the new options.
docker info --format '{{.LoggingDriver}}'
# json-file

Severity: SERVICE-IMPACT. A reload of the daemon is required; running containers continue with their old driver until they restart.

Walk through the important options:

• max-size: 10m rotates the file when it reaches 10 MiB. Ten MiB is small enough that a chatty container does not fill the disk before rotation kicks in; large enough that rotation overhead is negligible.
• max-file: 5 keeps five rotated files plus the active one for a total of 60 MiB per container. The cap is per container; a host with 200 containers caps at 12 GiB.
• labels: service,env,version attaches three specific labels as fields on every log line. Loki uses these as label dimensions.
• tag: "{{.Name}}/{{.ID}} sets a per-line tag. The default is the container ID, which is opaque. A human-readable tag helps when reading the raw file.

Daemon-level configuration for journald

{
  "log-driver": "journald",
  "log-opts": {
    "tag": "{{.Name}}",
    "labels": "service,env,version"
  }
}

# CONFIGURATION: reload the daemon.
sudo systemctl reload docker

# READ-ONLY: confirm a container is logging to the journal.
docker run -d --name test --log-driver=journald alpine \
  sh -c 'echo hello; sleep 3600'
sudo journalctl CONTAINER_NAME=test --no-pager -n 5
# Jan 15 12:34:56 host docker-test[1234]: hello

Severity: CONFIGURATION. A reload of the daemon is required.

The journald driver integrates with the host’s existing log pipeline. The journal is rotated by systemd-journald per its own configuration (SystemMaxUse, SystemKeepFree); Docker does not manage rotation.

Per-container override

# CONTAINER-SPECIFIC: a chatty batch container with tighter caps.
docker run -d \
  --name migration \
  --log-driver=json-file \
  --log-opt max-size=2m \
  --log-opt max-file=2 \
  --log-opt labels=service,env \
  registry.example.com/migrations:v1.4.0

Severity: CONFIGURATION. A docker run invocation.

The per-container override is the right pattern for individual chatty workloads. The defaults from /etc/docker/daemon.json apply otherwise.

Grafana Alloy shipper for journald

# /etc/alloy/config.river
loki.source.journal "containers" {
  forward_to = [loki.write.local.receiver]
  matches    = ["CONTAINER_NAME=~".+""]
  labels = {
    component = "docker",
  }
}

loki.write "local" {
  endpoint {
    url = "http://loki.internal:3100/loki/api/v1/push"
  }
}

Severity: CONFIGURATION. Reload Alloy to apply.

The Alloy journal source reads every journal entry tagged as a container and forwards it to Loki. The labels block adds dimensions for downstream filtering. Per-container Docker labels (service, env, version) become Loki labels automatically because the journal driver attaches them as fields.

Structured log emission from the application

import logging, json

logger = logging.getLogger("checkout")
handler = logging.StreamHandler()
handler.setFormatter(logging.Formatter("%(message)s"))
logger.addHandler(handler)
logger.setLevel(logging.INFO)

def handle_request(req):
    logger.info("request handled", extra={
        "service": "checkout",
        "trace_id": req.headers.get("traceparent", ""),
        "duration_ms": req.elapsed_ms,
        "status": req.status,
    })

Severity: CONFIGURATION. Application deployment.

The application emits a single-line JSON object per log entry. The Loki pipeline parses the JSON into fields; a query like {service="checkout"} | json | duration_ms > 1000 filters by the parsed field. Unstructured text is searchable but cannot be filtered by field.

How to validate it

Five checks, cheapest first.

# READ-ONLY: the daemon driver.
docker info --format '{{.LoggingDriver}}'
# json-file

# READ-ONLY: the running container driver.
docker inspect --format '{{.HostConfig.LogConfig.Type}}' migration
# json-file

# READ-ONLY: the raw log lines.
docker logs --tail 10 migration
# 2026-01-15 12:34:56 INFO request handled
# 2026-01-15 12:34:57 ERROR connection refused

# READ-ONLY: the rotated files.
ls -la /var/lib/docker/containers/$(docker inspect --format '{{.Id}}' migration)/*.log
# -rw-r----- 1 root root 10485760 Jan 15 12:30 ...-json.log
# -rw-r----- 1 root root 10485760 Jan 15 12:00 ...-json.log.1

# READ-ONLY: Loki has the lines.
curl -fsS -G http://loki.internal:3100/loki/api/v1/query \
  --data-urlencode 'query={service="checkout"}' \
  --data-urlencode 'limit=3'

A clean validation: the daemon and the running container report the configured driver, docker logs returns the expected lines, the rotated files exist with the configured size cap, and Loki returns the structured query.

How it can fail

1. The host disk fills because no rotation is configured. Cause: /etc/docker/daemon.json has no max-size and max-file. A chatty container fills the disk in days. Detection: df -h /var/lib/docker shows 100 percent; du -sh /var/lib/docker/containers/*/*.log shows the offender.
2. A high-cardinality label blows up Loki. Cause: the application emits a request UUID as a Loki label rather than a parsed field. Loki’s index grows by the number of unique values. Detection: loki_tsdb_index_writes_total rises sharply; Loki’s storage cost spikes.
3. Logs arrive at the journal but never at Loki. Cause: the team moved to journald and never pointed a shipper at the journal. The host has the logs; the platform does not. Detection: docker logs works; Loki returns nothing.
4. Structured logs are emitted as pretty-printed JSON. Cause: the logging framework emits multi-line objects. The first line parses; subsequent lines fail and become unstructured text. Detection: Loki shows partial fields on every entry.
5. docker logs --follow blocks because the driver is blocking. Cause: the driver is fluentd or splunk and the destination is unreachable. The container’s log writes back-pressure. Detection: the container’s CPU is elevated; the log file is growing slowly.
6. The daemon reload drops log stream continuity. Cause: systemctl reload docker re-creates the driver processes for new containers; running containers keep their old driver, and a brief gap appears for restarted containers. Detection: Loki shows a small gap for a single container at the reload time.

How to troubleshoot it

1. What driver is the daemon configured for? docker info --format '\{\{.LoggingDriver\}\}'.
2. What driver is the container running? docker inspect --format '\{\{.HostConfig.LogConfig.Type\}\}' <container>.
3. Where are the bytes going? docker logs --tail 10 <container> reads from the driver’s destination. For json-file this is the on-disk file; for journald this is the journal.
4. Is rotation working? ls -la /var/lib/docker/containers/<id>/. If only one file exists, rotation is off.
5. Is the shipper running? systemctl status alloy (or whichever shipper). journalctl -u alloy -n 20 shows the last twenty log lines from the shipper itself.
6. Is Loki receiving? curl -fsS http://loki:3100/ready. A 200 confirms Loki is accepting pushes.

Security implications

• Log files contain everything the application prints. Passwords in URLs, session tokens in error messages, and customer data in debug logs are all captured by the log driver and shipped to Loki. Audit the application’s log lines.
• Log files survive container deletion. A json-file log file is not deleted when the container is removed unless rotation is configured; a stale log file persists on disk until the rotation cap kicks in.
• The journal is a privileged target. journald exposes every container’s logs to anyone who can read the journal. Restrict the journal reader group.
• Structured logs expose schema. A JSON log line reveals the application’s internal field names. Treat the schema as sensitive.

Performance implications

• Driver cost. json-file is local disk and is the cheapest driver. journald adds a journald syscall per line. fluentd and splunk add a network round trip per batch.
• Back-pressure. A driver that cannot keep up (because the destination is unreachable) back-pressures the container’s stdout. The container slows.
• Log volume. A chatty container can produce hundreds of MiB per hour. The cost is paid in disk, network, and Loki ingestion.

Production guidance

• Always set max-size and max-file on the json-file driver. The defaults are unbounded and the failure mode is silent disk fill.
• Use journald if the host already runs systemd-journald with a sensible SystemMaxUse. The integration is free.
• Set Loki labels deliberately. The cardinality budget per service should be in the dozens, not the millions.
• Emit structured JSON logs from the application. One JSON object per line, no pretty-printing, no embedded newlines.
• Cross-reference Loki with cAdvisor’s container labels. Every log line should carry the container’s service and env so that Loki queries can pivot to the container metrics for the same workload.
• Rotate the journal on the same schedule as the application logs. A journal that fills the disk is a host outage.

Verification

You should now be able to answer:

• What does the json-file driver write to disk, and where?
• Why is no-rotation a host-disk failure mode?
• When is journald the right driver and when is json-file?
• Why is a request UUID a bad Loki label?
• What does a single-line JSON log line buy you in Loki?

Quiz