Disk-full: distinguishing data, WAL, temp, archive and backup · PostgreSQL

“The disk is full” is five different incidents with five different responses. Establishing which filesystem filled is the first action, and it takes ten seconds.

Identify first

df -h \
  /var/lib/postgresql/18/main \
  /var/lib/postgresql/18/main/pg_wal \
  /var/lib/postgresql/18/main/base/pgsql_tmp \
  /archive \
  /backup \
  /var/log/postgresql

Filled	Symptom	Response
`pg_wal`	`PANIC`, cluster down and will not restart	Give it space. Never delete WAL
Data	Writes fail; cluster survives	Free space, then find the growth
Temp	Queries fail with a temp-file error	Lower `work_mem`, or fix the query
Archive	Archiving fails; `pg_wal` starts growing	Fix the destination. Lesson XIII-05
Backup	Backups fail silently	Retention, and an alert
Logs	Logging fails; may block backends	Rotation. Lesson XVI-01

The archive row is the one that cascades: a full archive stops archiving, unarchived segments cannot be recycled, and pg_wal fills next. What looks like a WAL problem started somewhere else.

What a full `pg_wal` actually does

Data-loss riskthe filesystem reaching 100%— EXECUTION VALIDATED on 18.6, with pg_wal on a 48 MB tmpfs and rows inserted until it filled.

$ df -h /walspace

tmpfs            48M   48M     0 100% /walspace

Data-loss riskthe server log, verbatim— EXECUTION VALIDATED on 18.6. Read the whole sequence.

$ grep -iE 'PANIC|FATAL|recovery|shut' /tmp/full.log

PANIC:  could not write to file "pg_wal/xlogtemp.61": No space left on device
LOG:  database system was interrupted; last known up at 2026-08-27 23:31:00 UTC
FATAL:  the database system is in recovery mode
LOG:  database system was not properly shut down; automatic recovery in progress
FATAL:  could not write to file "pg_wal/xlogtemp.69": No space left on device
LOG:  shutting down due to startup process failure
LOG:  database system is shut down

The recovery

Destructivemoving pg_wal to a filesystem with room, keeping every segment— EXECUTION VALIDATED on 18.6. The cluster is stopped throughout.

$ ls -la $PGDATA/pg_wal
du -sh /walspace/pg_wal

mkdir -p /var/lib/postgresql/walrescue
cp -a /walspace/pg_wal/. /var/lib/postgresql/walrescue/
rm -f $PGDATA/pg_wal
ln -s /var/lib/postgresql/walrescue $PGDATA/pg_wal

lrwxrwxrwx ... pg_wal -> /walspace/pg_wal
48M	/walspace/pg_wal

lrwxrwxrwx ... pg_wal -> /var/lib/postgresql/walrescue

Read-only / Safeand it comes back— EXECUTION VALIDATED on 18.6.

$ pg_ctl -D $PGDATA start; SELECT count(*) FROM fill;

server started

LOG:  database system was not properly shut down; automatic recovery in progress
LOG:  redo starts at 0/17615F8
LOG:  redo done at 0/3FFFDE8 system usage: CPU: user: 0.02 s, system: 0.00 s, elapsed: 0.03 s
LOG:  checkpoint starting: end-of-recovery immediate wait
LOG:  checkpoint complete: wrote 5094 buffers (31.1%) ... distance=41594 kB
LOG:  database system is ready to accept connections

rows_recovered
----------------
        60000

Every committed row recovered, because no WAL was deleted.

The end-of-recovery checkpoint wrote 5,094 buffers — 31.1% of the pool — over a redo distance of 41,594 kB. That is the work that had accumulated and could not be written, now flushed.

Other ways to create space, in order of preference:

Extend the filesystem. Best, if the storage allows it.
Move pg_wal to another filesystem and symlink, as above.
Delete something else on that filesystem — logs, an old backup.
Fix what was retaining WAL, which is the actual cause: a failing archive, an abandoned slot, wal_keep_size. Lesson XII-03.

Step 4 is the only one that stops it recurring.

Prevention

Separate filesystems. pg_wal, data, and the archive on different volumes means one filling does not take the others with it. This is the single highest-value structural decision in this lesson.

Alert at 75% on pg_wal specifically, not on the data volume — they fill at completely different rates.

Bound what retains WAL. max_slot_wal_keep_size, from lesson XIV-06, converts an unbounded pg_wal into an invalidated slot, which is a task rather than an outage.

Watch pg_stat_archiver. A failing archive is a pg_wal problem that has not happened yet.

A ballast file is a genuinely good idea

# create at build time, on the pg_wal filesystem
fallocate -l 2G /walspace/BALLAST_DELETE_ME_IN_EMERGENCY

A pre-allocated file you can delete instantly buys the space to start the cluster and fix the real problem, at the cost of 2 GB you were never going to use.

It converts “the cluster will not start and I need a storage change” — which may be minutes or hours away — into “delete a file”. Name it so that nobody has to guess what it is for at three in the morning, and document it, because a file nobody knows about is not a recovery mechanism.

Why running out of WAL space is a PANIC rather than an error

Most failures in PostgreSQL are errors: the statement fails, the transaction aborts, other sessions continue. Running out of WAL space is a PANIC, which terminates every backend and restarts the cluster.

The reason is the durability contract from lesson XII-01. WAL must be written before the pages it describes, and a backend that cannot write WAL cannot maintain that ordering. Continuing would mean either acknowledging work that is not recoverable, or leaving shared memory in a state where some pages are protected and others are not.

Neither is acceptable, so the server takes the only safe option: stop everything, and let crash recovery re-establish a known state from the WAL that was successfully written.

Three consequences follow.

The PANIC is correct behaviour. It looks catastrophic and it is what prevents an unrecoverable database.

The blast radius is the cluster. One backend’s inability to write WAL takes down every session, including ones doing nothing but reading, because shared memory is shared.

Recovery needs the same resource that just ran out. Replaying WAL writes WAL — the end-of-recovery checkpoint and the new WAL that follows it — which is why the restart failed in the transcript above. There is no read-only mode to limp along in.

One detail about the capture above is worth stating so it is not over-generalised: pg_wal was a 48 MB tmpfs being written to as fast as the CPU could generate WAL, so it went from healthy to full in seconds with nothing in between. A real filesystem filling over hours gives monitoring a window to fire in, and that window is the whole reason for alerting on pg_wal usage separately. The failure mode is identical; only the warning time differs, and it is not something to count on.

What to take from this

Identify which filesystem filled before doing anything. Five filesystems, five responses.
A full archive cascades into a full pg_wal. The symptom is not the cause.
Measured: a full pg_wal produced a PANIC, recovery then failed for the same reason, and the cluster would not start.
Never delete WAL segments, and pg_resetwal is not a disk-space tool.
The fix is space: extend, move and symlink, or delete something else. Then fix what was retaining WAL.
Measured: after moving pg_wal, all 60,000 committed rows recovered.
Separate filesystems, alert on pg_wal at 75%, and keep a ballast file.

Cross-course references

Linux for Production Sysadmins — Part XIV (Filesystems) and Part XVI (LVM) cover finding what actually filled and whether it can be grown in place, and Part LXXX (Common failure scenarios) covers the reflex responses that make this worse.
Ceph & Distributed Storage — Part LXIV (Nearfull, backfillfull and full) and Part LXV (Why full clusters are dangerous) cover the same failure where the storage system stops accepting writes for everyone.
Observability for Production Sysadmins — Part XVIII (Alerting rules) covers alerting on each filesystem separately, since the response differs per volume.