Persistent journals and disk pressure

The journal is bounded, but the size limit is configurable. A host with verbose services and a small journal limit rotates aggressively; a host with a large journal and no SIEM forwarding keeps everything until the disk fills. The discipline is to size the journal based on the workload and forward to long-term storage before the journal cycles.

First: is the journal persistent at all?

Sizing an ephemeral journal is wasted effort. journald decides where to write from the Storage= directive in journald.conf:

The trap is auto. It does not create /var/log/journal/. On a minimal image where nobody ever created it, journald quietly runs in memory, and every log line is destroyed at reboot.

$ journalctl --header | grep -i storage; ls -d /var/log/journal 2>&1; journalctl --list-boots | tail -3

Storage: persistent
/var/log/journal
-2 8f3c... Sat 2026-08-08 09:14:02 UTC—Sat 2026-08-08 21:02:55 UTC
-1 21ab... Sat 2026-08-08 21:03:31 UTC—Sun 2026-08-09 06:41:10 UTC
0 5d90... Sun 2026-08-09 06:41:44 UTC—Sun 2026-08-09 11:22:07 UTC

Make it persistent with a drop-in, not by editing the shipped journald.conf (a package update can replace that file):

# /etc/systemd/journald.conf.d/10-persistent.conf
[Journal]
Storage=persistent

sudo mkdir -p /etc/systemd/journald.conf.d
sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald
journalctl --flush                       # move /run entries into /var
journalctl --header | grep -i storage    # must report: persistent

journalctl --flush asks journald to migrate whatever it already has in /run/log/journal/ into the new persistent directory, so the boot in progress is not lost at the moment you make the switch.

The journal limits

None of these defaults is a fixed number. They are computed from the size of the filesystem the journal lives on, then capped:

man 5 journald.conf: the MaxUse pair “defaults to 10% and the [KeepFree pair] to 15% of the size of the respective file system, but each of the calculated default values is capped to 4G”, and MaxFileSize “defaults to one eighth of the values configured with SystemMaxUse= and RuntimeMaxUse= capped to 128M, so that usually seven rotated journal files are kept as history”.

The percentage matters more than the cap on the hosts where this bites. “4 GB” is only the default on a filesystem of 40 GB or more; on a 10 GB /var the journal caps itself at 1 GB, and a runbook that assumes 4 GB will size retention wrong by a factor of four. Read the effective values rather than the table:

# What journald computed, in its own words. This is the authoritative line.
journalctl -u systemd-journald --no-pager | grep -i 'System Journal' | tail -1
#   System Journal (/var/log/journal/<id>) is 215.3M, max 4G, 3.7G free.

# What the journal occupies right now
journalctl --disk-usage

# The merged configuration, across every drop-in
systemd-analyze cat-config systemd/journald.conf | grep -vE '^\s*(#|$)'

SystemMaxUse is the hard cap on what the journal occupies. SystemKeepFree is the floor under free space on the filesystem. Whichever binds first triggers rotation — which is why a journal well under SystemMaxUse still rotates on a nearly full disk.

$ journalctl -u systemd-journald --no-pager | grep -i 'System Journal' | tail -1; journalctl --disk-usage

Aug 11 06:01:20 host systemd-journald[1670868]: System Journal (/var/log/journal/c850e0f0) is 1.2G, max 4G, 15.9G free.
Archived and active journals take up 1.2G in the file system.

Sizing the journal

For production, the right answer depends on the workload:

The journal cycle rate (when entries are deleted) is roughly SystemMaxUse divided by daily log volume. On a host that produces 1 GB of logs per day with SystemMaxUse=4 GB, the journal cycles every four days.

Tuning

Journal sizing is configured in journald.conf, so the drop-in goes in /etc/systemd/journald.conf.d/:

# /etc/systemd/journald.conf.d/90-sizing.conf
[Journal]
SystemMaxUse=8G
SystemKeepFree=20%
SystemMaxFileSize=256M

$ sudo mkdir -p /etc/systemd/journald.conf.d && sudoedit /etc/systemd/journald.conf.d/90-sizing.conf && sudo systemctl restart systemd-journald && journalctl --header | head

Storage: persistent
Current journal size: 7.9G (max 8.0G, leaving 20% free)
Maximum journal size: 8.0G
Journal files: 32
...

systemctl daemon-reload is not needed here — nothing about the unit changed. systemctl restart systemd-journald is what makes journald re-read its configuration.

Investigating journal disk pressure

$ df -h /var/log/journal/; du -sh /var/log/journal/; journalctl --header | grep -E 'Max|size' ; journalctl --verify 2>&1 | head

Filesystem      Size  Used Avail Use% Mounted on
/dev/sdc1        20G   18G   2.0G  90%
12G   /var/log/journal

Storage: persistent
Current journal size: 7.8G (max 8.0G, leaving 20% free)
...

Reclaiming the space now

Lowering SystemMaxUse= does not shrink a journal that is already oversized. journald applies the new limit going forward; it does not go back and delete what it has already written. man 5 journald.conf is explicit about the analogous case: once the filesystem fills, journald stops using more space, “but it will not be removing existing files to reduce the footprint again, either”.

The command that actually frees space is journalctl --vacuum-*:

# Look first
journalctl --disk-usage

# Archive the active journal file so it becomes eligible
sudo journalctl --rotate

# Then pick a bound. Any one of these, or several.
sudo journalctl --vacuum-size=2G    # keep at most 2 GB
sudo journalctl --vacuum-time=7d    # discard entries older than 7 days
sudo journalctl --vacuum-files=10   # keep at most 10 journal files

# Confirm
journalctl --disk-usage
df -h /var/log/journal/

Vacuuming buys time; it does not fix the cause. Follow it with the real decision: cap the journal properly with a journald.conf.d drop-in, quieten whatever unit is generating the volume, or grow /var.

Long-term retention with SIEM forwarding

The journal is not a long-term store. For retention beyond the journal cycle, forward to a SIEM:

# /etc/rsyslog.d/40-journal-to-siem.conf
module(load="imjournal")

global(
  DefaultNetstreamDriver="gtls"
  DefaultNetstreamDriverCAFile="/etc/rsyslog.d/tls/ca.pem"
  DefaultNetstreamDriverCertFile="/etc/rsyslog.d/tls/client-cert.pem"
  DefaultNetstreamDriverKeyFile="/etc/rsyslog.d/tls/client-key.pem"
)

action(type="omfwd"
       target="siem.example.com"
       port="6514"
       protocol="tcp"
       StreamDriver="gtls"
       StreamDriverMode="1"
       StreamDriverAuthMode="x509/name"
       StreamDriverPermittedPeers="siem.example.com"
       queue.type="LinkedList"
       queue.filename="siem_queue"
       queue.maxdiskspace="10g"
       action.resumeRetryCount="-1")

This forwards every journal entry to the SIEM. The journal keeps the recent days; the SIEM keeps everything.

Knowledge check