Linux logging architecture — kernel, journald, syslog

A Linux system produces logs from at least four sources. Each has its own destination, format, and lifetime. Understanding them is the foundation of every operational diagnostic.

The four layers

flowchart LR
  K["Kernel<br/>printk / /dev/kmsg"]
  S["System services<br/>sd_notify / syslog"]
  A["Applications<br/>stdout/stderr / files"]
  J["journald<br/>binary journal"]
  Y["syslog<br/>rsyslog / syslog-ng"]
  F["Log files<br/>/var/log/"]

  K --> J
  S --> J
  S --> Y
  A --> Y
  A --> F
  J --> SI["central shipper"]
  Y --> SI
  F --> SI

$ journalctl --list-boots; ls -la /var/log/journal/; ls -la /var/log/

-1 1234567890abc... Mon 2026-08-04 11:23:45 UTC—Mon 2026-08-04 11:24:00 UTC
0 abcdef123456... Mon 2026-08-04 11:24:00 UTC—Mon 2026-08-09 12:00:00 UTC

6 -rw-r-x---+ 1 root root 12345678 Aug  9 12:00 /var/log/journal/user-1000.journal
drwxr-xr-x  2 root root      4096 Aug  9 12:00 /var/log/journal/remote/

-rw-r-----  1 root adm    12345 Aug  9 12:00 /var/log/auth.log
-rw-r-----  1 root syslog  67890 Aug  9 12:00 /var/log/syslog
-rw-r-----  1 root root   234567 Aug  9 12:00 /var/log/kern.log
drwxr-xr-x  2 root root    4096 Aug  9 12:00 /var/log/nginx/

Kernel logs: dmesg and /dev/kmsg

The kernel logs through printk to a ring buffer. On modern systems, journald reads the ring buffer (via /dev/kmsg) into the journal. dmesg is a convenience command for reading the ring buffer; on systemd-managed systems, prefer journalctl -k.

$ dmesg | tail -5; journalctl -k --no-pager | tail -5

[    5.234] systemd[1]: Starting Journal Service...
[    5.456] sshd[1234]: Server listening on 0.0.0.0 port 22.
[    5.789] chronyd[567]: Selected source 169.254.169.254 (PHC0)
...

journald — systemd’s logging service

journald is the system service that captures log messages from every systemd-managed unit. It writes to a binary journal in /var/log/journal/. The journal is queryable via journalctl with rich filtering.

journalctl -u sshd                       # by unit
journalctl -p err                        # by priority
journalctl -b                            # since boot
journalctl --since "1 hour ago"          # by time
journalctl -f                            # follow (tail -f equivalent)
journalctl -k                            # kernel messages
journalctl -u sshd -p err                # combine filters

$ journalctl -u sshd --since '1 hour ago' --no-pager | tail

Aug  9 11:23:45 host sshd[12345]: Accepted publickey for alice from 10.0.0.5 port 51234 ssh2: RSA SHA256:abc...
Aug  9 11:24:12 host sshd[12345]: pam_unix(sshd:session): session opened for user alice by (uid=0)
Aug  9 11:30:01 host sshd[12345]: Failed password for invalid user admin from 203.0.113.45 port 51234 ssh2

syslog — the traditional text logging

Traditional syslog writes text records to /var/log/ files organised by facility and priority:

$ logger -p auth.warning -t myapp "failed login attempt for admin"; tail -1 /var/log/auth.log

Aug  9 12:00:01 host myapp: failed login attempt for admin

Application files — the unstructured layer

Some applications do not write to syslog or journald directly. They write to files under /var/log/<application>/:

• nginx: /var/log/nginx/access.log, error.log
• mysql: /var/log/mysql/error.log
• PostgreSQL: /var/log/postgresql/
• Apache: /var/log/apache2/, /var/log/httpd/

These files are managed by logrotate; rotation and retention are configured in /etc/logrotate.d/<application> or /etc/logrotate.conf.

$ ls /etc/logrotate.d/

apt
dpkg
nginx
rsyslog
sshd
...

The default flow

flowchart TB
  APP["Application / service"]
  SD["systemd service"] --> J["journald"]
  K["Kernel"] --> J
  J --> JS["/var/log/journal/"]
  J --> SH["Forward to rsyslog"]
  APP --> SY["Traditional syslog call"]
  SY --> RS["rsyslog / syslog-ng"]
  RS --> RL["/var/log/auth.log, syslog, kern.log"]
  APP --> AF["/var/log/APPNAME/*.log"]

Knowledge check