Local logging — rsyslog, journald, logrotate, retention

Local logging is the router’s first line of forensic evidence. When something fails, the operator reviews the local logs first to identify the symptom and the root cause. The defensive pattern: every router has bounded local logging (the logs do not fill the disk), every log is exported to a central server (the logs survive router failure), and every retention period is documented.

This lesson covers the local logging configuration on VyOS 1.5 LTS, the three subsystems (rsyslog, journald, logrotate), the disk-space failure mode, and the production discipline of bounded retention.

The local logging stack

flowchart LR
  APP["Applications<br/>FRR, Kea, charon, wg"] -->|syslog| R["rsyslog"]
  APP -->|stdout/stderr| J["journald"]
  R -->|/var/log/messages| FS["Filesystem"]
  J -->|/var/log/journal| FS
  J -->|binary journal| FS
  FS --> L["logrotate"]
  L -->|rotates| FS
  FS -.->|export to central| REM["Remote syslog<br/>server"]

The three subsystems:

• rsyslog — receives syslog messages from applications and writes them to text files in /var/log/. The traditional Unix syslog daemon.
• journald — receives structured log messages from systemd-managed services and the kernel. Writes to a binary journal in /var/log/journal/.
• logrotate — rotates log files on a schedule (typically daily), compresses old logs, and deletes logs older than the retention period.

The operator can query either rsyslog (text files) or journald (binary journal) for forensic analysis.

Configuring rsyslog

configure
set system syslog log facility all level 'info'
set system syslog log facility local7 level 'debug'
set system syslog global facility all level 'info'
set system syslog global archive file '/var/log/messages'
set system syslog global archive size '500000'
set system syslog global archive file '/var/log/debug'
set system syslog global archive file '/var/log/debug' size '500000'
commit
save

The configuration:

• log facility all level info — log messages of info and above for all facilities.
• log facility local7 level debug — log debug and above for local7 (the standard facility for DHCP).
• global facility all level info — same as the per-facility configuration, applied globally.
• archive file — write the log to the specified file.
• archive size — limit the file size to 500 MB (logrotate kicks in when the file exceeds this size).

After commit, the rsyslog daemon is reloaded; the new configuration takes effect immediately.

Configuring journald

configure
set system logs max-size '500'
set system logs max-files '30'
commit
save

The configuration limits the journal to 500 MB total size, with at most 30 files retained. The journal daemon automatically rotates when the size limit is reached.

Configuring logrotate

The logrotate configuration is at /etc/logrotate.conf. The VyOS defaults are:

/var/log/messages {
    rotate 30
    daily
    compress
    missingok
    notifempty
    postrotate
        /usr/sbin/logrotate /etc/logrotate.d/vyos-rsyslog
    endscript
}

/var/log/debug {
    rotate 7
    daily
    compress
    missingok
    notifempty
}

The configuration:

• /var/log/messages rotates daily, keeps 30 days of compressed logs.
• /var/log/debug rotates daily, keeps 7 days (debug logs are noisy).
• compress — old logs are gzipped to save space.
• missingok — if the log file is missing, don’t error.
• notifempty — if the log file is empty, don’t rotate.

The operator can override these defaults by editing /etc/logrotate.conf directly, or by adding a custom file to /etc/logrotate.d/.

How the result is validated

show log
show log auth.log
journalctl -u ssh
journalctl -p err --since today
ls -la /var/log/
logrotate -d /etc/logrotate.conf

The first two show the rsyslog text files. The third and fourth show the journald binary journal. The fifth lists the log files. The sixth tests logrotate in dry-run mode (no actual rotation, but the operator can see what would happen).

A working local logging setup:

• Logs are written to /var/log/messages and other files.
• Logs are rotated daily; old logs are compressed.
• The journal is bounded to 500 MB.
• Disk usage is monitored; alerts fire at 80% utilisation.

vyos@R1:~$ show log
Aug 15 12:00:01 R1 kea-dhcp4: INFO [kea-dhcp4.leases] DHCP4_LEASE_ALLOC ...
Aug 15 12:00:05 R1 charon: 09[NET] received packet: from 203.0.113.50[500] ...
Aug 15 12:00:10 R1 frr zebra: interface eth0 state changed to up
Aug 15 12:00:15 R1 kernel: [12345.678] drop: TCP 192.0.2.50:54321 -> 192.0.2.1:22 ...

Bounded retention — the production discipline

The defensive pattern: every log file has a maximum size and a maximum age. The operator documents the retention period and the disk-space budget.

flowchart TD
  A["Log written to file"] --> B{"File exceeds<br/>max-size?"}
  B -- "no" --> C["Continue writing"]
  B -- "yes" --> D["Rotate: rename to .1<br/>create new file"]
  D --> E{"Files exceed<br/>max-files?"}
  E -- "no" --> F["Continue"]
  E -- "yes" --> G["Delete oldest<br/>compressed archive"]

The rotation cycle:

1. Log file exceeds max-size. Logrotate renames the current file to <name>.1 and creates a new file.
2. Old files accumulate. Each rotation adds a new archive.
3. max-files limit reached. The oldest archive is deleted.

The bound: with max-size 500M and max-files 30, the total disk usage for /var/log/messages is at most 15 GB (30 × 500 MB compressed).

The disk-space failure mode

When the disk fills:

1. rsyslog cannot write to /var/log/messages. rsyslog drops the message or queues it in memory.
2. journald cannot write to /var/log/journal/. journald drops the message or queues it in memory.
3. The router cannot write to /tmp/. Configuration backups fail.
4. The router cannot update the boot image. Upgrades fail.
5. SSH login fails. The PAM module cannot write the auth log.

The operator sees: SSH login attempts fail with “disk full”; the router logs nothing; the operator cannot troubleshoot the issue.

The fix:

• Free up disk space: delete old logs, old kernel images, old configuration backups.
• Configure logrotate to bound the log size.
• Configure monitoring for disk utilisation (alert at 80%).
• Configure remote syslog export so logs survive even if the local disk fills.

How it fails

The production failure modes:

• Logging fills the disk. Verbose debug logs without rotation. The fix: configure bounded retention.
• Logrotate not configured. The default logrotate configuration is minimal; the operator must configure per-file retention. The fix: edit /etc/logrotate.conf and /etc/logrotate.d/.
• journald unbounded. The journal grows without limit until the disk fills. The fix: configure max-size and max-files.
• Time zone not set. Logs are in UTC but the operator expects local time. The fix: configure the time zone with set system time-zone.
• Logs not exported. Local logs fill the disk and are lost. The fix: configure remote syslog export.
• Sensitive data in logs. Logs include API keys, passwords, or PII. The fix: filter sensitive data before logging.

Rollback

The recovery from a logging failure:

• Disk full: delete old logs, configure logrotate, restart rsyslog and journald.
• Logs not writing: check the rsyslog and journald configuration; restart the daemons.
• Logs filling the disk: configure bounded retention; delete old logs.

The VyOS configuration rollback (rollback N) restores the previous configuration if the logging change breaks the system.

Production discipline

Cross-course references

• XLVIII-VyOS-Logging (vyos-xlviii-02-routing-daemon-logs, vyos-xlviii-03-firewall-logs, vyos-xlviii-04-vpn-logs, vyos-xlviii-05-remote-syslog, vyos-xlviii-06-log-validation) cover the rest of the logging subsystem.
• VIII-Linux-Logging (Linux course) covers the underlying Linux logging stack.

Quiz