journald and syslog — container logs where the host already looks

Every Linux host already runs a log pipeline. It has retention, it has rotation, it is already forwarded somewhere, and your on-call engineers already know how to search it. The default json-file driver ignores all of that and writes container output to a file under /var/lib/docker that nothing else on the host knows exists.

This lesson is about the two drivers that connect the two worlds, and about the three ways each of them drops messages without telling you.

journald

Set it per container:

$ docker run -d --name api --log-driver journald --log-opt tag=api example.com/api:2.4.0

c3f81a04b7d29e6a5f0c8b3d1e7a92f4068c5b1d3a7e9f2b4c6d8e0a2f4b6c8d

Or, more usefully, as the host default in /etc/docker/daemon.json:

{
  "log-driver": "journald",
  "log-opts": {
    "tag": "{{.Name}}"
  }
}

A daemon restart is required, and it applies to containers created afterwards — existing containers keep the driver they were created with. That surprises people during a migration: half the host is on one driver for weeks.

Querying

Docker writes structured fields alongside the message, and those fields are what make the journal better than a text file:

$ journalctl CONTAINER_NAME=api -n 5 --no-pager

Aug 11 09:14:02 docker-host-01 api[184213]: {"level":"info","msg":"listening on :8080"}
Aug 11 09:14:19 docker-host-01 api[184213]: {"level":"warn","msg":"slow query","ms":1841}
Aug 11 09:14:31 docker-host-01 api[184213]: {"level":"error","msg":"upstream timeout"}

$ journalctl CONTAINER_NAME=api -n 1 -o json-pretty --no-pager

{
  "MESSAGE" : "upstream timeout",
  "CONTAINER_ID" : "c3f81a04b7d2",
  "CONTAINER_ID_FULL" : "c3f81a04b7d29e6a5f0c8b3d1e7a92f4068c5b1d3a7e9f2b4c6d8e0a2f4b6c8d",
  "CONTAINER_NAME" : "api",
  "CONTAINER_TAG" : "api",
  "IMAGE_NAME" : "example.com/api:2.4.0",
  "PRIORITY" : "3",
  "_HOSTNAME" : "docker-host-01"
}

The real prize is that this is now in the same stream as everything else, so a correlation that was previously impossible becomes one command:

$ journalctl --since '09:14:25' --until '09:14:40' --no-pager

Aug 11 09:14:31 docker-host-01 api[184213]: {"level":"error","msg":"upstream timeout"}
Aug 11 09:14:33 docker-host-01 kernel: Out of memory: Killed process 184390 (node)
Aug 11 09:14:33 docker-host-01 dockerd[1204]: container died c3f81a04b7d2 exitCode=137

An OOM kill, the container’s last words, and the daemon’s account of it, interleaved by timestamp in a plain time-range query. With json-file those three lines live in three unrelated places.

To narrow that to specific sources, note that journalctl combines matches on different fields with AND and matches on the same field with OR — so -u docker.service CONTAINER_NAME=api returns nothing, because no message satisfies both. Use + to separate alternatives:

$ journalctl --since '09:14' _SYSTEMD_UNIT=docker.service + CONTAINER_NAME=api --no-pager

Aug 11 09:14:31 docker-host-01 api[184213]: {"level":"error","msg":"upstream timeout"}
Aug 11 09:14:33 docker-host-01 dockerd[1204]: container died c3f81a04b7d2 exitCode=137

The docker logs cliff

The three ways journald loses messages

Rate limiting

journald drops messages from a service that exceeds a burst threshold. The defaults are visible in journald.conf:

$ grep -E 'RateLimit' /etc/systemd/journald.conf

#RateLimitIntervalSec=30s
#RateLimitBurst=10000

Ten thousand messages per 30 seconds, per service. A chatty container under load crosses that easily, and when it does the journal records a single line and silently discards the rest:

Suppressed 41822 messages from /system.slice/docker.service

If you have never grepped for Suppressed on a busy host, do it now. The messages you lose are exactly the ones produced during the burst, which is to say during the incident.

Raise the limits, or disable them for the journal and impose the bound elsewhere:

# /etc/systemd/journald.conf
RateLimitIntervalSec=30s
RateLimitBurst=100000

Volatile storage

Storage=auto, the default, means: persist to /var/log/journal if that directory exists, otherwise keep the journal in memory under /run/log/journal and lose everything at reboot.

$ ls -d /var/log/journal 2>/dev/null || echo 'VOLATILE - journal is lost on reboot'

/var/log/journal

On a host built from a minimal image, that directory frequently does not exist. You then have a logging setup that works perfectly until the reboot you most want to investigate.

Disk bounds

The journal caps its own size, and the default derives from the filesystem size rather than from your retention requirement. Set it explicitly so you know what you have:

# /etc/systemd/journald.conf
SystemMaxUse=4G
SystemMaxFileSize=256M

$ journalctl --disk-usage

Archived and active journals take up 3.4G in the file system.

syslog

Use it when you have an existing syslog estate that container logs must join.

$ docker run -d --name api --log-driver syslog --log-opt syslog-address=tcp://192.0.2.10:514 --log-opt syslog-format=rfc5424 --log-opt tag=api example.com/api:2.4.0

5b2e9c71a04f8d3e6b1c0a95f2d7e438b6c1a9d0e3f5b7c9a1d3e5f7b9c1d3e5

syslog-address takes tcp://, udp:// or unix://. The choice is not cosmetic:

• UDP never blocks the container and silently drops datagrams under load or when the far end is gone. You lose messages and nothing anywhere records that you did.
• TCP is reliable and therefore applies backpressure, which leads directly to the next section.
• unix:///dev/log hands to the local syslog daemon, which then owns the forwarding problem. On a host running rsyslog this is usually the sanest option, because rsyslog has queueing and disk spooling that the Docker driver does not.

syslog-format=rfc5424 is worth setting: RFC 3164 is the older format with a resolution of one second and no structured data, and rfc5424micro gives microsecond timestamps, which matters when you are ordering events across services.

For TLS to the collector, the driver takes syslog-tls-ca-cert, syslog-tls-cert and syslog-tls-key. Sending application logs in plaintext across anything but a trusted segment is a data-exposure decision, not a convenience one.

Blocking, and the outage it causes

The remedy is a driver-agnostic pair of options:

{
  "log-driver": "syslog",
  "log-opts": {
    "syslog-address": "tcp://192.0.2.10:514",
    "mode": "non-blocking",
    "max-buffer-size": "4m"
  }
}

mode=non-blocking puts a ring buffer between the application and the driver. When the buffer fills, log messages are dropped and the application keeps running.

Choosing

• journald — a systemd host whose journal is already collected and forwarded. Keeps docker logs working, correlates container output with kernel and daemon events, and needs its rate limit and storage settings checked.
• **syslog to unix:///dev/log** — a host running rsyslog with an established central estate. Let rsyslog own queueing and retry rather than the Docker driver.
• **syslog to tcp://** — when there is no local syslog daemon. Set mode=non-blocking and accept that docker logs no longer works.
• Fluent Bit or Vector — when you need parsing, enrichment, or routing to more than one destination. Covered in the central collection lesson in this part.
• json-file with rotation — a single host with no central logging. Set max-size and max-file, as the rotation lesson covers, or the disk fills.

Knowledge check

Where next

That closes the logging part. The monitoring and observability parts take up the other two signals — metrics and traces — and the question of how all three are correlated.