Skip to main content
RunBook Academy

PostgreSQLXII · WAL, Checkpoints and Crash RecoveryWAL

What retains WAL, and what releases it

Advanced⏱ ~30 minpsql

What you'll learn

  • Enumerate the four mechanisms that retain WAL and query each
  • Diagnose a filling pg_wal in one query
  • Bound slot retention without breaking replication
  • Respond to a full pg_wal filesystem in the correct order

Prerequisites

Verified against PostgreSQL 18.x · PostgreSQL (comparison targets) 17.11, 16.15 · PostgreSQL (support calendar) 18, 17, 16, 15, 14 supported · pgBackRest 2.59.1 · PgBouncer 1.25.2 · Patroni 4.1.5 · Ubuntu (host baseline) 26.04 LTS · 2026-08-27

Not yet marked complete on this device.

A full pg_wal filesystem stops the cluster. It is one of the most common ways a PostgreSQL server goes down, it is entirely preventable, and there are exactly four things that cause it.

The four mechanisms

MechanismBounded byDefault bound
Checkpoint distancemax_wal_size1 GB, soft
Replication slotsmax_slot_wal_keep_sizeunlimited
Unarchived segmentsNothingunlimited
wal_keep_sizeItself0

Two of those four are unbounded by default. That is the whole problem.

Replication slots

A slot records the position a consumer has reached, and PostgreSQL retains every segment from that position onward. It does this whether or not the consumer still exists.

Read-only / Safethe same slot, created with immediately_reserve, then some write traffic
$ SELECT pg_create_physical_replication_slot('demo_slot', immediately_reserve => true);
SELECT slot_name, active, restart_lsn,
     pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS wal_retained
FROM pg_replication_slots;
 pg_create_physical_replication_slot
-------------------------------------
(demo_slot,0/39247A98)                <- an LSN is returned

slot_name | active | restart_lsn | wal_retained
-----------+--------+-------------+--------------
demo_slot | f      | 0/39247A98  | 169 MB
standby1  | t      | 0/43BC14E0  | 0 bytes

169 megabytes pinned by a slot that no standby has ever used, while the slot with a live standby attached holds nothing at all. Dropping it and checkpointing twice released 64 MB and four segments — pg_wal went from 241 MB and 15 segments to 177 MB and 11.

Two checkpoints, because recycling happens at checkpoint time and the first one only establishes the new horizon.

At the 4,359 kB/s generation rate measured in lesson XII-02, an unattended reserving slot accumulates about 367 GB a day, indefinitely.

Bounding it

-- cap what any one slot may retain
ALTER SYSTEM SET max_slot_wal_keep_size = '100GB';
SELECT pg_reload_conf();

The trade is explicit and worth being clear about: a slot that exceeds the cap is invalidated, and the standby that depended on it must be rebuilt from a fresh base backup. That is a real cost, and it is smaller than the primary going down.

Set it to a value that a healthy standby will never approach — large enough to survive a network partition or a long maintenance window, small enough that the filesystem cannot fill.

PostgreSQL 18 adds a second, complementary bound. From the release notes:

Allow inactive replication slots to be automatically invalidated using server variable idle_replication_slot_timeout

ALTER SYSTEM SET idle_replication_slot_timeout = '7d';

This addresses the abandoned-slot case directly rather than by proxy: a slot with no consumer for seven days is invalidated on time rather than on volume. The two settings answer different questions — one bounds space, the other bounds neglect — and both default to off.

Unarchived segments

With archive_mode = on, a segment cannot be recycled until archive_command has reported success for it. A failing archive command therefore fills the disk at the full generation rate, and it does so whether the failure is loud or silent.

Read-only / Safethe archiver's own view of itself
$ psql -U postgres -x -c "SELECT * FROM pg_stat_archiver"
archived_count     | 0
last_archived_wal  |
last_archived_time |
failed_count       | 0
last_failed_wal    |
last_failed_time   |
stats_reset        | 2026-08-27 21:01:04.021891+00

The two comparisons that matter:

failed_count increasing means the command is failing now. Look at last_failed_wal and the server log for the command’s stderr.

last_archived_time falling behind means the archiver is not keeping up even though nothing is failing — the command is too slow for the generation rate. That is a capacity problem rather than an error, and it fills the disk just as effectively.

Lesson XIII-05 covers writing an archive_command that fails correctly.

The response, in order

When pg_wal is filling:

1. Establish which mechanism. One query each:

SELECT slot_name, active, pg_size_pretty(pg_current_wal_lsn() - restart_lsn) AS retained, wal_status
  FROM pg_replication_slots ORDER BY 3 DESC;

SELECT archived_count, failed_count, last_archived_time, last_failed_time, last_failed_wal
  FROM pg_stat_archiver;

SHOW wal_keep_size;
SHOW max_wal_size;

2. Act on the mechanism, not on the symptom. Drop the abandoned slot; fix or replace the archive command; lower wal_keep_size.

3. Then CHECKPOINT to allow recycling to proceed.

What to take from this

  • Four mechanisms retain WAL. Two — slots and unarchived segments — are unbounded by default.
  • Measured: an inactive slot with a reserved position pinned 169 MB and would have continued at 367 GB a day.
  • wal_status is PostgreSQL’s own summary. lost means the slot is unusable and the standby needs rebuilding.
  • Set max_slot_wal_keep_size, and in 18 also idle_replication_slot_timeout. Both default to off.
  • pg_stat_archiver: failed_count rising is an error; last_archived_time falling behind is a capacity problem. Both fill the disk.
  • Never delete files from pg_wal. It makes the cluster unrecoverable.

Cross-course references

  • Observability for Production Sysadmins — Part LIX (Database observability) covers alerting on each of the four retention causes separately, because the response differs and a single “pg_wal is large” alert does not distinguish them.
  • Linux for Production Sysadmins — Part XIV (Filesystems) covers why a separate filesystem for pg_wal turns a cluster-down into a bounded problem.

Quiz

Knowledge check · 6 questions

  1. Q1. A cluster's pg_wal has grown to 400 GB over three weeks. There is one replication slot, showing active = f and a restart_lsn from three weeks ago. What happened?

  2. Q2. A replication slot shows wal_status = 'lost'. What does that mean for the standby using it?

  3. Q3. The pg_wal filesystem is 100% full and the cluster will not start. What must not be done?

  4. Q4. Which mechanisms retain WAL and are unbounded by default? Select all that apply.

  5. Q5. A pg_wal directory that stays at 40 GB for an hour after a bulk load has finished indicates a retention problem that should be investigated.

  6. Q6. pg_wal is filling on a production cluster. Describe your diagnostic sequence and the order of actions.

Passing score: 75%. Answers are checked in this browser.