Replication slots and unbounded WAL retention · PostgreSQL

A replication slot is a promise the primary makes: I will keep every WAL segment you have not confirmed, for as long as it takes. That promise is what makes a standby reliably resumable, and it is also the most direct route to a full disk that PostgreSQL offers.

What a slot is

A slot records a position — restart_lsn — and the primary refuses to recycle any segment from that position onward.

Without a slot, a standby that falls behind further than the primary’s WAL retention gets:

FATAL:  requested WAL segment 000000010000000000000023 has already been removed

and must be rebuilt or fetched from the archive. With a slot, that cannot happen. The cost is that “cannot happen” is implemented by retaining WAL indefinitely.

A slot retains nothing until it reserves a position

Read-only / Safea slot created the ordinary way— EXECUTION VALIDATED on 18.6, followed by three rounds of bulk writes, drops and checkpoints.

$ SELECT pg_create_physical_replication_slot('demo_slot');
SELECT slot_name, active, restart_lsn, wal_status FROM pg_replication_slots;

 pg_create_physical_replication_slot
-------------------------------------
(demo_slot,)                          <- no LSN returned

slot_name | active | restart_lsn | wal_status
-----------+--------+-------------+------------
demo_slot | f      |             |              <- BOTH NULL

pg_wal before: 257M, 16 segments
pg_wal after:  257M, 16 segments        <- no growth

A physical slot begins reserving only when a standby connects to it, or when it is created with immediately_reserve => true. This matters when you are working out whether a slot is what is filling your disk: a slot with a NULL restart_lsn is not.

Read-only / Safea reserving slot with no consumer— EXECUTION VALIDATED on 18.6, after four rounds of bulk write, DROP and CHECKPOINT.

$ SELECT slot_name, active, restart_lsn,
     pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS wal_retained
FROM pg_replication_slots;

 slot_name | active | restart_lsn | wal_retained
-----------+--------+-------------+--------------
demo_slot | f      | 0/39247A98  | 169 MB
standby1  | t      | 0/43BC14E0  | 0 bytes

169 MB pinned by a slot no standby has ever used, while the slot with a live standby attached holds nothing — because that standby is keeping up. A healthy slot retains almost nothing. Retention is the symptom.

At the 4,359 kB/s generation rate from lesson XII-02, an unattended reserving slot accumulates roughly 367 GB a day, indefinitely, until the filesystem fills and the primary stops committing.

Bounding it

ALTER SYSTEM SET max_slot_wal_keep_size = '100GB';
SELECT pg_reload_conf();

Beyond that limit, the slot is invalidated rather than allowed to keep growing.

Read-only / Safesafe_wal_size counting down to invalidation— EXECUTION VALIDATED on 18.6. The standby was stopped; max_slot_wal_keep_size was set to 48MB; WAL was then generated in rounds.

$ SELECT slot_name, active, wal_status, pg_size_pretty(safe_wal_size) AS safe FROM pg_replication_slots;

 slot_name | active | wal_status | safe
-----------+--------+------------+-------
standby1  | f      | reserved   | 56 MB

 round 1:  wal_status: reserved   safe_wal_size: 28 MB
 round 2:  wal_status: lost       safe_wal_size: NULL
 round 3:  wal_status: lost       safe_wal_size: NULL

safe_wal_size is the column to alert on. It is how many bytes of WAL can still be generated before this slot is invalidated. It went 56 MB → 28 MB → NULL, and NULL means it is already too late.

Service impact possiblethe primary, at the moment of invalidation— EXECUTION VALIDATED on 18.6, transcribed from the server log.

$ docker logs rbpg-prim 2>&1 | grep -i invalidat

LOG:  invalidating obsolete replication slot "standby1"
DETAIL:  The slot's restart_lsn 0/1C82A100 exceeds the limit by 8216320 bytes.
HINT:  You might need to increase "max_slot_wal_keep_size".

Service impact possiblethe standby, when it came back— EXECUTION VALIDATED on 18.6, transcribed from the standby's log.

$ grep FATAL /tmp/sb.log

FATAL:  could not start WAL streaming: ERROR:  can no longer access replication slot "standby1"
DETAIL:  This replication slot has been invalidated due to "wal_removed".

The standby could not resume and had to be rebuilt from a new base backup.

max_slot_wal_keep_size does not prevent a failure — it chooses which one

Setting	What you get	What breaks
`-1` (default, unlimited)	The standby can always resume	`pg_wal` fills and the primary stops
A finite limit	`pg_wal` is bounded	The standby is invalidated and needs a rebuild

There is no third option. Choosing the limit is choosing which consequence you would rather have at three in the morning, and for most production systems the answer is clear: a standby that needs rebuilding is a task, while a primary that has stopped committing is an outage.

That makes a finite max_slot_wal_keep_size the right default for most clusters — set generously enough that a standby’s ordinary maintenance window fits inside it, and paired with a WAL archive so a standby that does exceed it can still catch up via restore_command instead of needing a full rebuild. Lesson XIV-03’s slot-plus-archive arrangement is what makes the limit safe to set.

The abandoned slot

The classic outage, and it is always the same sequence:

A standby is built, with a slot.
The standby is decommissioned, fails, or is rebuilt with a new slot name.
Nobody drops the old slot.
WAL accumulates at the full generation rate.
Days or weeks later, pg_wal fills and the cluster stops.

It is silent throughout. Nothing about the primary is unhealthy and nothing reports it.

-- run this on a schedule; alert on any row
SELECT slot_name, slot_type, active, wal_status,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn))
         AS retained,
       pg_size_pretty(safe_wal_size) AS safe_remaining
  FROM pg_replication_slots
 WHERE NOT active
    OR wal_status IN ('extended','unreserved','lost');

Dropping a slot is safe if you are certain nothing needs it

SELECT pg_drop_replication_slot('old_standby');

An inactive slot can be dropped immediately, and the WAL is released at the next checkpoint — two, in practice, since the first establishes the new horizon. Measured: dropping the 169 MB slot above and checkpointing twice took pg_wal from 241 MB and 15 segments to 177 MB and 11.

Being certain is the whole job. A slot belonging to a standby that is merely down for maintenance is doing exactly what it was created to do, and dropping it converts a temporary outage into a rebuild.

Check pg_stat_replication for a live connection, check the slot’s active flag, and check with whoever owns the standby — in that order.

Logical slots, catalog_xmin, and the failure physical slots do not have

Logical replication slots do everything physical slots do and add one thing that is genuinely more dangerous.

A logical slot must be able to decode WAL into row changes, which requires the catalogue as it was when those records were written. So in addition to restart_lsn, a logical slot holds catalog_xmin — an xid horizon below which system catalogue rows must not be vacuumed.

An abandoned logical slot therefore does two things at once. It retains WAL, like a physical slot. And it stops vacuum from cleaning catalogue tables, cluster-wide, forever. The consequence is pg_attribute, pg_class and pg_depend bloating steadily, which slows every query’s planning, and — because catalogue tables are subject to the same wraparound rules as everything else — an eventual anti-wraparound emergency on tables nobody thinks of as tables.

pg_replication_slots.catalog_xmin is the column that shows it, and age(catalog_xmin) is the number to watch:

SELECT slot_name, slot_type, active,
       age(catalog_xmin) AS catalog_xmin_age,
       age(xmin)         AS xmin_age
  FROM pg_replication_slots
 ORDER BY age(catalog_xmin) DESC NULLS LAST;

Two further asymmetries worth knowing.

Physical slots with hot_standby_feedback also hold an xmin, and lesson XIV-07 measures that. But it is the user table horizon, which vacuum reports on and which you are already watching. catalog_xmin is not visible in any table’s vacuum output.

PostgreSQL 18 adds idle_replication_slot_timeout, which invalidates a slot that has been inactive for too long. It is the time-based counterpart to max_slot_wal_keep_size, and it is the better bound for the abandoned-slot case specifically: a slot nobody has connected to in a week is far more likely to be forgotten than legitimately waiting. It makes the same trade — the slot is invalidated, and whatever it belonged to needs a rebuild.

What to take from this

A slot guarantees a standby can resume, by retaining WAL without bound.
A slot with a NULL restart_lsn reserves nothing.
A healthy slot retains almost nothing. Retention is the symptom.
safe_wal_size counts down to invalidation. NULL means it already happened.
max_slot_wal_keep_size chooses which failure you get. A finite limit plus an archive is the arrangement that makes it safe.
Alert on inactive slots and on wal_status leaving reserved.
Logical slots additionally hold catalog_xmin and block catalogue vacuum cluster-wide.

Cross-course references

Observability for Production Sysadmins — Part LIX (Database observability) covers alerting on inactive slots and on wal_status, which is the earliest signal available before the volume fills.
Linux for Production Sysadmins — Part XIV (Filesystems) covers the filesystem this eventually fills, and Part LXXX (Common failure scenarios) covers the class of failure where a safety mechanism is the cause.

What a slot is

Bounding it

The abandoned slot

What to take from this

Cross-course references

Quiz