Stdout and stderr — the container logging contract

Docker’s logging model is one sentence: applications write to stdout and stderr; the logging driver ships the bytes onward. That is the whole contract. It is the twelve-factor position, and every production logging pipeline on Docker — docker logs, journald, Loki, CloudWatch, Splunk — assumes it holds.

The interesting part of this lesson is not the contract. It is the three distinct ways applications break it, only one of which is obvious.

What the contract buys you

• Capture without configuration. The driver tags every line with the container, the stream it came from, and an RFC3339Nano timestamp. You configure that once, on the host, not once per application.
• Rotation as a platform concern. One daemon.json stanza bounds every container on the host. No logrotate config inside any image.
• Survival past the container. Logs leave the container’s filesystem, so they outlive docker rm. This is the property you care about at 03:00, when the container that failed is already gone and its replacement is healthy.
• One shipping path. Change the destination for the whole fleet by changing the driver, without rebuilding a single image.

How a log line actually gets out

Break #1: logging to a file inside the container

The obvious violation. An application writes /var/log/app.log inside its own filesystem.

That file lives on the container’s OverlayFS upper layer, which means:

• It is deleted with the container. docker rm destroys the evidence.
• It grows against the same disk as everything else on the host, with no driver-level rotation and no docker system df row that counts it.
• No logging driver can see it. Shipping it centrally requires a sidecar that shares the filesystem, or an application that ships its own logs.
• docker logs returns nothing, so the operator’s first instinct — “the container is producing no output, it must be hung” — points at the wrong failure.

The fix is to configure the application to write to stdout. Nearly every mainstream server can be told to:

When the application genuinely cannot be configured — a vendor binary with a hardcoded path — the symlink trick is the fallback:

RUN mkdir -p /var/log/app \
&& ln -sf /dev/stdout /var/log/app/access.log \
&& ln -sf /dev/stderr /var/log/app/error.log

Break #2: the process daemonises

An application that forks into the background and detaches from its controlling terminal closes fds 0, 1 and 2 as part of daemonising. Under Docker this produces two symptoms at once: no logs, and the container exits immediately, because PID 1 returned.

The stock fixes are foreground flags, and every mainstream daemon has one:

nginx      -g 'daemon off;'
httpd      -DFOREGROUND
rsyslogd   -n
sshd       -D
php-fpm    -F

The container’s entrypoint should be the application process itself, in exec form, so that the application is PID 1 and its stdout is the container’s stdout:

ENTRYPOINT ["nginx", "-g", "daemon off;"]

Shell form — ENTRYPOINT nginx -g 'daemon off;' — wraps the command in /bin/sh -c, which stays PID 1 and makes the application a child. Output still flows, because the child inherits fd 1, but signal handling breaks: the shell does not forward SIGTERM, so docker stop waits out its full timeout and then kills the container, and the application never gets to flush its final log lines or close its connections cleanly. If you must use a wrapper script, end it with exec so the application replaces the shell:

#!/bin/sh
set -eu
# ... setup that must happen before the app starts ...
exec /usr/local/bin/myapp --config /etc/myapp.toml

Break #3: buffering — the container that looks silent

This is the one that is worth the price of the lesson.

The C standard library chooses a buffering mode for stdout based on what stdout is connected to. Connected to a terminal, it is line-buffered: every newline triggers a flush. Connected to anything else — a file, a socket, or a pipe — it switches to fully buffered, typically in 4 KB or 8 KB blocks.

Under Docker, without -t, stdout is a pipe. So a program that prints perfectly on your laptop prints nothing in docker logs until it has accumulated a full block, or until it exits and the runtime flushes on teardown.

$ docker run --rm python:3.12-slim python -c "import time
print('starting')
time.sleep(30)"

(no output for 30 seconds, then:)
starting

The container is healthy. The application ran. The line was printed on time. It sat in a userspace buffer inside the container for thirty seconds, and an operator watching docker logs -f during an incident concluded the process was hung.

The fixes, in order of preference:

Go and Rust are not affected: neither uses libc stdio for their standard logging paths. Java’s System.out is line-flushed by default via a PrintStream with autoflush, but a BufferedWriter wrapped around it by the application is not. Node’s process.stdout is synchronous on pipes on Linux, so it is safe but can block the event loop under heavy logging — a different problem with the same root cause.

Verifying the contract holds

Three checks, each of which can fail:

CONTAINER=web

# 1. Is anything arriving at all, and how recently?
docker logs --timestamps --tail 5 "$CONTAINER"

# 2. Which stream is it on? docker logs writes container stdout to its own
#    stdout and container stderr to its own stderr, so you can separate them.
echo '--- stdout only ---'
docker logs --tail 20 "$CONTAINER" 2>/dev/null
echo '--- stderr only ---'
docker logs --tail 20 "$CONTAINER" 2>&1 1>/dev/null

# 3. Is the application still writing to a file inside the container?
docker exec "$CONTAINER" sh -c 'find / -xdev -name "*.log" -size +1M 2>/dev/null' \
|| echo 'no shell in image; check with docker diff instead'

# 4. Whatever the image does, docker diff shows files it has created.
docker diff "$CONTAINER" | grep -E '^A .*\.log$' || echo 'no new .log files: contract holds'

Check 4 is the one that distinguishes working from broken without any assumption about the image. docker diff reports the container’s OverlayFS upper layer — every file added (A), changed (C) or deleted (D) since the container started. A growing .log file in that list is a contract violation, stated by the platform rather than inferred.

Knowledge check