A replica was decommissioned in March and took the primary down in June

Reported symptoms

db-prod-11 PANICs at 04:52:

PANIC:  could not write to file "pg_wal/xlogtemp.2214": No space left on device

The postmaster shuts down. The cluster does not come back — the startup process fails with the same error and exits.

The filesystem holding the data directory is 100% full. The largest directory in it, by a very wide margin, is pg_wal. Row counts and table sizes are unchanged from a month ago and there has been no unusual write volume.

The only recent change on the host is an OS patch five days earlier. The change calendar has nothing else in the preceding week, and the team starts investigating the patch.

Nobody connects the incident to a read replica that was decommissioned eleven weeks earlier.

Evidence provided

$ du -sh $PGDATA/pg_wal && ls $PGDATA/pg_wal | grep -c '^0000'

689G	/var/lib/postgresql/18/main/pg_wal
44118

Archiving is healthy: no .ready files, failed_count zero, last_archived_time within a minute of the PANIC. That eliminates the other cause of unbounded WAL retention.

Once the cluster is started with pg_wal relocated:

$ psql -c "SELECT slot_name, active, restart_lsn, wal_status, pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained FROM pg_replication_slots;"

     slot_name     | active | restart_lsn  | wal_status |  retained  
-------------------+--------+--------------+------------+------------
replica_analytics | f      | 8C/1A004628  | reserved   | 689 GB

pg_stat_replication is empty and has been for eleven weeks according to the monitoring history. max_slot_wal_keep_size is -1.

Change record CHG-2209, closed 12 March, describes decommissioning the analytics read replica. Its steps are: delete the VM, remove the monitoring entry, update the runbook. There is no step for the slot.

A second production cluster in the same estate carries two inactive slots with restart_lsn values from January, currently retaining 71 GB.

Work the evidence before reading on

1. pg_wal is 86 times max_wal_size and archiving is healthy. What is the remaining cause?
2. The slot’s restart_lsn is dated 12 March. What is significant about that date?
3. max_slot_wal_keep_size is -1. What has the server been instructed to prioritise?
4. The decommissioning change removed the VM and the monitoring entry. Why did that make the problem harder to find rather than easier?

Root cause

The slot outlived the standby, and the primary kept its promise

A replication slot records a position and instructs the primary to retain every WAL segment from that position onwards. The primary honours it unconditionally — across restarts, indefinitely — because it cannot distinguish a standby offline for ten minutes from one deleted in March.

The default made the choice

max_slot_wal_keep_size = -1 means unlimited retention, and it encodes a real decision: the primary will fill its disk rather than break a standby.

That is defensible where standbys matter more than headroom. It is the wrong choice where a slot can be orphaned and nobody notices. What is not defensible is having neither made the choice nor known it was made.

Removing the monitoring entry removed the last reference

The decommissioning change deleted the VM, removed the monitoring entry, and updated the runbook. Every one of those steps removed something that referred to the replica.

The slot was on a different server, was not referred to by any of them, and became invisible on exactly the day it became dangerous. Eleven weeks later there was no artefact anywhere in the estate connecting replica_analytics to anything a human would search for.

Resolution

The cluster is down with a full filesystem. Get space first, cause second.

Relocate pg_wal to a filesystem with room. This is a supported layout and it is reversible:

systemctl stop postgresql@18-main
mv "$PGDATA/pg_wal" /mnt/spare/pg_wal
ln -s /mnt/spare/pg_wal "$PGDATA/pg_wal"
chown -h postgres:postgres "$PGDATA/pg_wal"
systemctl start postgresql@18-main
tail -3 /var/log/postgresql/postgresql-18-main.log

Confirm from the log that the server reached database system is ready to accept connections.

Find the slot, and confirm it genuinely has no consumer:

SELECT slot_name, slot_type, active, active_pid, restart_lsn, wal_status,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained
FROM pg_replication_slots ORDER BY slot_name;

SELECT count(*) FROM pg_stat_replication;

SELECT pg_drop_replication_slot('replica_analytics');
CHECKPOINT;

pg_wal shrinks over the following checkpoints rather than at once, because segments are recycled by renaming.

Before closing the incident, audit the estate. The evidence already shows 71 GB held on another cluster.

Verification

The cluster is running and the log confirms it.

No inactive slots remain, or only ones with a documented, currently offline consumer.

pg_wal falls toward max_wal_size across successive checkpoints.

The pg_wal relocation is scheduled to be reversed in a planned window rather than left as an undocumented permanent layout.

SHOW max_slot_wal_keep_size returns a value somebody chose, with the reasoning recorded.

Prevention

Alert on NOT active in pg_replication_slots. This is the eleven weeks of warning that existed and was not used:

SELECT slot_name, active, wal_status, invalidation_reason,
       pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) AS retained_bytes,
       safe_wal_size
FROM pg_replication_slots
WHERE NOT active OR wal_status <> 'reserved';

Set max_slot_wal_keep_size deliberately, and alert on safe_wal_size falling.

Add the slot drop to the decommissioning runbook, next to deleting the VM. It is missing precisely because the slot lives on a different server from the thing being retired.

Audit slots quarterly across the estate.

Graph pg_wal size against max_wal_size. A ratio persistently above one has two causes and both deserve a page long before the filesystem is involved.