Skip to main content
RunBook Academy

← All break/fix scenarios in Linux

intermediateFilesystems~25 min

Break/Fix: the systemd journal has filled /var and services are failing to log

Reported symptoms

  • ●`/var` is at or near 100% and applications are failing on write
  • ●`journalctl` returns entries that stop abruptly at a timestamp hours ago
  • ●Services log "No space left on device" or exit non-zero at start-up
  • ●The monitoring agent reports the host as healthy because its own checks succeeded before the disk filled
  • ●A colleague has already run `rm` inside /var/log/journal and journalctl now reports fewer entries but df has not moved

Evidence

  • · `df -h /var` shows 100% used with 0 available
  • · `journalctl --disk-usage` reports the archived plus active journal size
  • · `du -xh --max-depth=1 /var | sort -h | tail` attributes the space to /var/log
  • · `journalctl -n 5` shows the newest entry is hours old
  • · `journalctl --verify` reports FAIL on one or more journal files
  • · `systemctl status systemd-journald` shows repeated "Failed to write entry" messages
  • · `grep -rn SystemMaxUse /etc/systemd/journald.conf /etc/systemd/journald.conf.d/` returns nothing, so the 10% default applies to a filesystem shared with everything else
  • · `lsof +L1 /var | head` shows deleted-but-open files still holding blocks
Diagnosis and resolutionclick to reveal

Root cause

The journal was made persistent (/var/log/journal exists) without an explicit size cap. journald's default is SystemMaxUse=10% of the filesystem, capped at 4G, with SystemKeepFree=15% held back - but those percentages are of the whole filesystem, and on a host where /var is shared with application data, package caches and container layers the journal is only one of several growing consumers. journald honours its own cap and stops there; it does not and cannot stop anything else from consuming the remaining space. Once the filesystem is genuinely full, journald cannot write, so the log lines that would have explained the original incident are the first casualty.

Remediation

Reclaim space with journalctl --vacuum-size or --vacuum-time rather than rm, because deleting journal files by hand leaves journald holding the inodes and the space is not returned until the daemon is restarted. Then set an explicit SystemMaxUse and SystemKeepFree in /etc/systemd/journald.conf.d/, restart systemd-journald, and confirm the new cap with journalctl --disk-usage. Give /var/log its own filesystem or LVM volume so a log flood cannot take the rest of /var with it.

Verification

df -h /var shows free space restored and stable. journalctl --disk-usage reports a figure below the configured SystemMaxUse. journalctl -f shows new entries arriving in real time. journalctl --verify passes. A deliberate test message written with systemd-cat appears in journalctl within a second.

Prevention

Set SystemMaxUse and SystemKeepFree explicitly on every host - never rely on the percentage default. Put /var/log on its own volume. Alert on filesystem usage above 80% for /var and /var/log, and alert separately on journal ingestion having stopped, because a full disk silences the very logs that would page you. Forward logs off the host so the evidence survives the filesystem that produced it.

Reported symptoms

At 02:40 an application team reports that their service will not restart. The error is No space left on device. Nobody has deployed anything. The monitoring dashboard is green, because the last successful check landed before the filesystem filled and nothing has scraped since.

You open journalctl to find out what started this and the newest entry is from 21:15 the previous evening.

Evidence provided

Read-only / Safethe filesystem is full and the journal stopped six hours ago
$ df -h /var; journalctl --disk-usage; journalctl -n 1 --no-pager
Filesystem      Size  Used Avail Use% Mounted on
/dev/mapper/vg0-var  40G   40G     0 100% /var
Archived and active journals take up 3.9G in the file system.
Aug 09 21:15:03 web01 kernel: EXT4-fs (dm-2): error count since last fsck: 0
Read-only / Safeattribute the space before you delete anything
$ du -xh --max-depth=1 /var 2>/dev/null | sort -h | tail -5
1.2G	/var/lib
3.9G	/var/log/journal
6.1G	/var/cache
28G	/var/lib/containers
40G	/var
Read-only / Safeno explicit cap is configured; the grep returns nothing
$ systemctl status systemd-journald --no-pager | tail -4; grep -rns 'SystemMaxUse\|SystemKeepFree' /etc/systemd/journald.conf /etc/systemd/journald.conf.d/ 2>/dev/null
Aug 09 21:15:04 web01 systemd-journald[412]: Failed to write entry (23 items, 712 bytes), ignoring: No space left on device
Aug 09 21:15:04 web01 systemd-journald[412]: Failed to write entry (19 items, 604 bytes), ignoring: No space left on device

Root cause

Two facts combine.

First, the journal was made persistent - /var/log/journal exists - and no size cap was ever configured. journald’s built-in default is SystemMaxUse=10% of the filesystem, capped at 4 GiB, with SystemKeepFree=15% reserved. On this 40 GiB /var that resolves to about 3.9 GiB, which matches journalctl --disk-usage exactly. journald is behaving correctly and is inside its budget.

Second, /var/lib/containers grew to 28 GiB. The journal did not fill the disk. It is a bystander - but it is the bystander that stops your ability to investigate, because journald cannot write to a full filesystem and drops entries instead.

That distinction drives the fix. Reclaiming journal space buys you room to work. It does not address why /var filled, and if you stop there the host fills again.

Resolution

  1. Buy working room first, non-destructively. sudo journalctl --vacuum-size=500M asks journald to drop archived journal files until the store is under 500 MiB. It never touches the active file, so no in-flight log line is lost. Confirm with df -h /var.
  2. Verify what survived. journalctl --verify reports PASS or FAIL per file. A FAIL here usually means someone already used rm; note it, because those entries are gone and cannot be part of your timeline.
  3. Find the real consumer. du -xh --max-depth=1 /var | sort -h | tail then descend into the largest entry. In this scenario that is /var/lib/containers. Reclaim it with the owning tool - podman system prune, docker system prune, dnf clean all, apt-get clean - never with a blind rm -rf on a path you have not attributed.
  4. Check for deleted-but-open files. sudo lsof +L1 /var | head lists files with a link count of 0 that a process still holds. Restart the owning service to release them; for journald that is sudo systemctl restart systemd-journald.
  5. Cap the journal explicitly. Create /etc/systemd/journald.conf.d/10-size.conf with a fixed size rather than a percentage, so the cap does not silently change when the filesystem is resized:
  6. ``ini [Journal] SystemMaxUse=2G SystemKeepFree=4G SystemMaxFileSize=256M MaxRetentionSec=30day ``
  7. Apply and confirm. sudo systemctl restart systemd-journald then journalctl --disk-usage should report a figure at or below 2 GiB. systemd-analyze cat-config systemd/journald.conf shows the merged effective configuration including your drop-in.
  8. Prove ingestion has resumed. logger -t breakfix "journal recovery test" then journalctl -t breakfix -n 1. If the line is not there within a second, journald is still not writing and the incident is not over.

Verification

  1. Free space is restored and stable. df -h /var shows headroom, and it is still there thirty minutes later - a filesystem that refills immediately means you reclaimed the bystander and not the cause.
  2. The journal is inside its cap. journalctl --disk-usage reports below the configured SystemMaxUse.
  3. The journal is intact. journalctl --verify returns PASS for every file.
  4. New entries are arriving. journalctl -f shows live traffic, and the test message written with logger is queryable.
  5. The gap is documented. Record the window between the last pre-incident entry and the first post-recovery entry. Anyone reading this timeline later needs to know those logs do not exist rather than concluding nothing happened.

Prevention

  • Set SystemMaxUse and SystemKeepFree explicitly on every host. The percentage defaults are computed from the filesystem, so the same config yields a different cap on every machine, and changes silently when a volume is grown.
  • Give /var/log its own filesystem or LVM volume. Then a log flood fills a volume that only holds logs, and /var/lib stays writable.
  • Alert on /var and /var/log above 80%, before the failure.
  • Alert separately on the journal having stopped ingesting. Disk-usage alerting catches the cause; ingestion alerting catches every other reason your evidence stops arriving.
  • Forward logs off the host. Central logging is what makes this incident recoverable rather than merely survivable - the entries you cannot read locally are already at the collector.