PostgreSQLXII · WAL, Checkpoints and Crash RecoveryWAL
What retains WAL, and what releases it
What you'll learn
- Enumerate the four mechanisms that retain WAL and query each
- Diagnose a filling pg_wal in one query
- Bound slot retention without breaking replication
- Respond to a full pg_wal filesystem in the correct order
Prerequisites
Verified against PostgreSQL 18.x · PostgreSQL (comparison targets) 17.11, 16.15 · PostgreSQL (support calendar) 18, 17, 16, 15, 14 supported · pgBackRest 2.59.1 · PgBouncer 1.25.2 · Patroni 4.1.5 · Ubuntu (host baseline) 26.04 LTS · 2026-08-27
A full pg_wal filesystem stops the cluster. It is one of the most
common ways a PostgreSQL server goes down, it is entirely preventable,
and there are exactly four things that cause it.
The four mechanisms
| Mechanism | Bounded by | Default bound |
|---|---|---|
| Checkpoint distance | max_wal_size | 1 GB, soft |
| Replication slots | max_slot_wal_keep_size | unlimited |
| Unarchived segments | Nothing | unlimited |
wal_keep_size | Itself | 0 |
Two of those four are unbounded by default. That is the whole problem.
Replication slots
A slot records the position a consumer has reached, and PostgreSQL retains every segment from that position onward. It does this whether or not the consumer still exists.
$ SELECT pg_create_physical_replication_slot('demo_slot', immediately_reserve => true);
SELECT slot_name, active, restart_lsn,
pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS wal_retained
FROM pg_replication_slots; pg_create_physical_replication_slot
-------------------------------------
(demo_slot,0/39247A98) <- an LSN is returned
slot_name | active | restart_lsn | wal_retained
-----------+--------+-------------+--------------
demo_slot | f | 0/39247A98 | 169 MB
standby1 | t | 0/43BC14E0 | 0 bytes169 megabytes pinned by a slot that no standby has ever used, while the
slot with a live standby attached holds nothing at all. Dropping it and
checkpointing twice released 64 MB and four segments — pg_wal went
from 241 MB and 15 segments to 177 MB and 11.
Two checkpoints, because recycling happens at checkpoint time and the first one only establishes the new horizon.
At the 4,359 kB/s generation rate measured in lesson XII-02, an unattended reserving slot accumulates about 367 GB a day, indefinitely.
Bounding it
-- cap what any one slot may retain
ALTER SYSTEM SET max_slot_wal_keep_size = '100GB';
SELECT pg_reload_conf();
The trade is explicit and worth being clear about: a slot that exceeds the cap is invalidated, and the standby that depended on it must be rebuilt from a fresh base backup. That is a real cost, and it is smaller than the primary going down.
Set it to a value that a healthy standby will never approach — large enough to survive a network partition or a long maintenance window, small enough that the filesystem cannot fill.
PostgreSQL 18 adds a second, complementary bound. From the release notes:
Allow inactive replication slots to be automatically invalidated using server variable
idle_replication_slot_timeout
ALTER SYSTEM SET idle_replication_slot_timeout = '7d';
This addresses the abandoned-slot case directly rather than by proxy: a slot with no consumer for seven days is invalidated on time rather than on volume. The two settings answer different questions — one bounds space, the other bounds neglect — and both default to off.
Unarchived segments
With archive_mode = on, a segment cannot be recycled until
archive_command has reported success for it. A failing archive command
therefore fills the disk at the full generation rate, and it does so
whether the failure is loud or silent.
$ psql -U postgres -x -c "SELECT * FROM pg_stat_archiver"archived_count | 0
last_archived_wal |
last_archived_time |
failed_count | 0
last_failed_wal |
last_failed_time |
stats_reset | 2026-08-27 21:01:04.021891+00The two comparisons that matter:
failed_count increasing means the command is failing now. Look at
last_failed_wal and the server log for the command’s stderr.
last_archived_time falling behind means the archiver is not
keeping up even though nothing is failing — the command is too slow for
the generation rate. That is a capacity problem rather than an error,
and it fills the disk just as effectively.
Lesson XIII-05 covers writing an archive_command that fails correctly.
The response, in order
When pg_wal is filling:
1. Establish which mechanism. One query each:
SELECT slot_name, active, pg_size_pretty(pg_current_wal_lsn() - restart_lsn) AS retained, wal_status
FROM pg_replication_slots ORDER BY 3 DESC;
SELECT archived_count, failed_count, last_archived_time, last_failed_time, last_failed_wal
FROM pg_stat_archiver;
SHOW wal_keep_size;
SHOW max_wal_size;
2. Act on the mechanism, not on the symptom. Drop the abandoned
slot; fix or replace the archive command; lower wal_keep_size.
3. Then CHECKPOINT to allow recycling to proceed.
What to take from this
- Four mechanisms retain WAL. Two — slots and unarchived segments — are unbounded by default.
- Measured: an inactive slot with a reserved position pinned 169 MB and would have continued at 367 GB a day.
wal_statusis PostgreSQL’s own summary.lostmeans the slot is unusable and the standby needs rebuilding.- Set
max_slot_wal_keep_size, and in 18 alsoidle_replication_slot_timeout. Both default to off. pg_stat_archiver:failed_countrising is an error;last_archived_timefalling behind is a capacity problem. Both fill the disk.- Never delete files from
pg_wal. It makes the cluster unrecoverable.
Cross-course references
- Observability for Production Sysadmins — Part LIX (Database observability) covers alerting on each of the four retention causes separately, because the response differs and a single “pg_wal is large” alert does not distinguish them.
- Linux for Production Sysadmins — Part XIV (Filesystems) covers
why a separate filesystem for
pg_walturns a cluster-down into a bounded problem.
Quiz
Knowledge check · 6 questions
Q1. A cluster's pg_wal has grown to 400 GB over three weeks. There is one replication slot, showing active = f and a restart_lsn from three weeks ago. What happened?
Q2. A replication slot shows wal_status = 'lost'. What does that mean for the standby using it?
Q3. The pg_wal filesystem is 100% full and the cluster will not start. What must not be done?
Q4. Which mechanisms retain WAL and are unbounded by default? Select all that apply.
Q5. A pg_wal directory that stays at 40 GB for an hour after a bulk load has finished indicates a retention problem that should be investigated.
Q6. pg_wal is filling on a production cluster. Describe your diagnostic sequence and the order of actions.
Passing score: 75%. Answers are checked in this browser.