Skip to main content
RunBook Academy

← All break/fix scenarios in PostgreSQL

intermediatepg-disk-full~40 min

The cluster PANICked at 02:14 with no space left on device, and the first suggestion in the channel was to delete WAL files

Reported symptoms

  • ●At 02:14 the cluster stops accepting connections entirely and the application returns connection refused
  • ●The last line in the server log is PANIC: could not write to file "pg_wal/xlogtemp.61": No space left on device
  • ●df reports the data volume at 100 percent used with zero bytes available
  • ●Attempting to start the cluster produces the same PANIC within a second and the postmaster exits again
  • ●pg_wal contains 1.4 GB of segments against a max_wal_size of 1 GB, and 88 files carry a .ready marker in archive_status
  • ●The first suggestion in the incident channel is to delete the oldest files in pg_wal to buy space
  • ●A second suggestion is to set fsync = off so the cluster can start

Evidence

  • · The server log ends with PANIC: could not write to file "pg_wal/xlogtemp.61": No space left on device, followed by the postmaster shutting down all other processes
  • · df -h shows /var/lib/postgresql at 100% with 0 available on a 20 GB volume
  • · du -sh on pg_wal returns 1.4G, against max_wal_size = 1GB
  • · ls pg_wal/archive_status | grep -c ready returns 88
  • · The archive_command is a shell script that writes to an NFS mount, and the mount has been unavailable since 22:40 the previous evening
  • · The server log contains repeated archive command failed with exit code 1 entries beginning at 22:41, roughly every 15 seconds
  • · No monitoring alert was raised for archive failures; the disk-space alert fired at 95 percent at 01:58, sixteen minutes before the PANIC
  • · pg_replication_slots is empty, so WAL retention is not caused by an abandoned slot
Diagnosis and resolutionclick to reveal

Root cause

The archive destination became unreachable at 22:40. PostgreSQL did exactly what it is designed to do: it refused to recycle any WAL segment that had not been successfully archived, retried the `archive_command` roughly every fifteen seconds, and logged every failure. `pg_wal` grew past `max_wal_size` because `max_wal_size` is a soft target for checkpoint pacing, not a cap — unarchived segments are retained regardless of it. At 02:14 the volume filled. PostgreSQL then hit a write it could not complete durably and issued a `PANIC`, which is the correct and deliberate behaviour: a database that cannot durably record a WAL record must stop rather than continue and risk acknowledging transactions it cannot replay. The PANIC is the safety mechanism working, not the failure. The actual failure chain is: archive destination down (22:40) → archive failures logged but not alerted (22:41 onward) → WAL accumulating for three and a half hours → disk alert at 95 percent with only sixteen minutes of headroom (01:58) → PANIC (02:14). Two of the three links are monitoring defects. Archive failures were logged and not alerted, which removed three and a half hours of warning. The disk alert fired at 95 percent, which on a volume filling at this rate left sixteen minutes — enough to acknowledge a page, not enough to act on it. The remaining risk in this incident is not the outage. It is that the first two suggestions in the channel — delete WAL files, disable fsync — would each have converted a recoverable outage into unrecoverable data loss.

Remediation

**Do not delete files from `pg_wal`.** Those segments are the only record of transactions that have been acknowledged to clients but not yet written into the data files. Deleting them destroys the ability to recover, and the cluster will refuse to start once it notices the gap. This is the single most damaging thing anybody can do at this point in the incident. **Do not set `fsync = off`.** It will let the cluster start. It also removes the guarantee that anything committed afterwards survives a crash, on a machine that has just demonstrated it can lose its footing. It converts one outage into permanent silent corruption. Free space from somewhere that is not `pg_wal`. In order of preference: 1. **Grow the volume.** On cloud or LVM storage this is usually minutes and it is always the correct first answer. 2. **Remove non-PostgreSQL files from the same volume** — old server logs, a forgotten dump, a core file. `du -sh /var/lib/postgresql/*` finds these quickly. 3. **Delete the emergency ballast file** if one exists, which is exactly what it is for. With space available, fix the archive. Until the `archive_command` succeeds, PostgreSQL will keep retaining segments and the volume will fill again: ```bash # Verify the destination is reachable and writable as the postgres user sudo -u postgres test -w /mnt/wal-archive && echo writable ``` Restore the mount, or repoint `archive_command` at a destination that works. `archive_command` is reloadable: ```sql ALTER SYSTEM SET archive_command = '/usr/local/bin/archive-wal.sh %p %f'; SELECT pg_reload_conf(); ``` Start the cluster. It will replay from the last checkpoint, then work through the 88 pending segments. Watch `archive_status` drain: ```bash watch -n 5 'ls /var/lib/postgresql/18/main/pg_wal/archive_status | grep -c ready' ``` Only once that count reaches zero and `pg_wal` has fallen back toward `max_wal_size` is the incident over.

Verification

The cluster accepts connections and `pg_is_in_recovery()` returns false. `ls pg_wal/archive_status | grep -c ready` returns 0, and `pg_stat_archiver` shows a recent `last_archived_time` with `failed_count` no longer increasing: ```sql SELECT archived_count, last_archived_wal, last_archived_time, failed_count, last_failed_wal, last_failed_time FROM pg_stat_archiver; ``` `du -sh pg_wal` has fallen back to roughly `max_wal_size` plus normal churn. `df -h` shows meaningful free space — meaningful meaning hours of headroom at the observed WAL generation rate, not a percentage. Force an archive round trip and confirm the file lands: ```sql SELECT pg_switch_wal(); ``` Then verify the resulting segment exists at the destination. Testing the archive is the only way to know it works; a quiet `pg_stat_archiver` also looks quiet when nothing is being archived.

Prevention

**Alert on `pg_stat_archiver.failed_count` increasing.** This is the alert whose absence cost three and a half hours of warning. A failing archive is a countdown to a full volume and it is visible from the first failure: ```sql SELECT failed_count, last_failed_wal, last_failed_time FROM pg_stat_archiver; ``` **Alert on `archive_status/*.ready` file count**, which measures the backlog rather than the failure. A sustained non-zero count is a problem even when individual archive attempts occasionally succeed. **Alert on disk free at a threshold expressed in time, not percent.** Ninety-five percent of 20 GB left sixteen minutes. Alert when free space falls below several hours of measured WAL generation, and page at a level that leaves room to act. **Give `pg_wal` its own filesystem.** A full `pg_wal` still stops the cluster, but it stops it without also taking down anything else that shares the volume, and it makes the growth visible as its own metric. **Keep a ballast file.** A pre-allocated file of a few gigabytes on the data volume, deletable in one command, converts "we are down and cannot free space" into "we have ten minutes of runway". Document where it is and who may delete it. **Write down that WAL is never deleted by hand**, in the runbook, where the person at 02:14 will read it. The suggestion appeared within minutes in this incident because it is the obvious wrong answer, and obvious wrong answers need to be pre-empted in writing rather than argued about during an outage. **Understand that `max_wal_size` is not a cap.** Any team that sizes a WAL volume as `max_wal_size` plus a margin will eventually meet this incident.

Reported symptoms

At 02:14 the cluster stops accepting connections entirely. The application returns connection refused.

The last line in the server log is a PANIC about no space left on device. df reports the data volume at 100 percent, zero bytes available.

Starting the cluster produces the same PANIC within a second and the postmaster exits again.

pg_wal holds 1.4 GB of segments against a max_wal_size of 1 GB, and 88 files carry a .ready marker.

The first suggestion in the incident channel is to delete the oldest files in pg_wal. The second is to set fsync = off so the cluster can start.

Evidence provided

Read-only / Safethe last thing the cluster said before it stopped
$ tail -4 /var/lib/postgresql/18/main/log/postgresql.log
2026-08-28 02:14:07.882 UTC [61] PANIC:  could not write to file "pg_wal/xlogtemp.61": No space left on device
2026-08-28 02:14:07.909 UTC [59] LOG:  checkpointer process (PID 61) was terminated by signal 6: Aborted
2026-08-28 02:14:07.909 UTC [59] LOG:  terminating any other active server processes
2026-08-28 02:14:07.933 UTC [59] LOG:  database system is shut down
Read-only / Safethe volume is genuinely full
$ df -h /var/lib/postgresql
Filesystem      Size  Used Avail Use% Mounted on
/dev/vdb         20G   20G     0 100% /var/lib/postgresql

Illustrative output

Read-only / Safe1.4 GB of WAL against a 1 GB max_wal_size, and 88 segments waiting to be archived
$ du -sh pg_wal && ls pg_wal/archive_status | grep -c ready
1.4G	pg_wal
88

The archive_command writes to an NFS mount that has been unavailable since 22:40 the previous evening. The log has carried archive command failed with exit code 1 roughly every fifteen seconds since 22:41.

No alert was raised for the archive failures. The disk-space alert fired at 95 percent at 01:58 — sixteen minutes before the PANIC.

pg_replication_slots is empty, so this is not an abandoned slot.

Work the evidence before reading on

  1. max_wal_size is 1 GB and pg_wal holds 1.4 GB. Is that a bug?
  2. Three and a half hours passed between the first archive failure and the outage. What consumed that warning?
  3. The disk alert fired at 95 percent. Was that alert useful?
  4. What happens if somebody deletes the oldest 20 files in pg_wal right now?

Root cause

PostgreSQL did the right thing at every step

The archive destination went away at 22:40. PostgreSQL refused to recycle any segment that had not been successfully archived, retried every fifteen seconds, and logged every failure. When the volume filled and it met a write it could not complete durably, it PANICked and stopped.

The failure chain is mostly monitoring

TimeEventWas it alerted?
22:40Archive destination unreachableNo
22:41 →archive command failed every 15sNo
22:41 → 02:14WAL accumulating, 3h33mNo
01:58Disk at 95 percentYes — 16 minutes of headroom
02:14PANIC, cluster downYes

Three and a half hours of warning existed and was written to the log the entire time. The one alert that did fire arrived too late to act on, because 95 percent of a volume filling at this rate is sixteen minutes, not a buffer.

The dangerous part of this incident is the proposed fix

Resolution

Free space from somewhere that is not pg_wal:

  1. Grow the volume. On cloud or LVM storage this is minutes, and it is always the correct first answer.
  2. Remove non-PostgreSQL files from the same volume — old logs, a forgotten dump, a core file. du -sh /var/lib/postgresql/* finds them in seconds.
  3. Delete the ballast file, if one exists. That is exactly what it is for.

Then fix the archive, because until archive_command succeeds the volume fills again:

sudo -u postgres test -w /mnt/wal-archive && echo writable

archive_command is reloadable:

ALTER SYSTEM SET archive_command = '/usr/local/bin/archive-wal.sh %p %f';
SELECT pg_reload_conf();

Start the cluster. It replays from the last checkpoint, then works through the 88 pending segments. Watch the backlog drain:

watch -n 5 'ls /var/lib/postgresql/18/main/pg_wal/archive_status | grep -c ready'

The incident is over when that count reaches zero and pg_wal has fallen back toward max_wal_size — not when the cluster starts.

Verification

The cluster accepts connections and pg_is_in_recovery() returns false.

The archive backlog is zero and the archiver is succeeding:

SELECT archived_count, last_archived_wal, last_archived_time,
       failed_count, last_failed_wal, last_failed_time
FROM pg_stat_archiver;

du -sh pg_wal has returned to roughly max_wal_size plus normal churn.

df -h shows hours of headroom at the observed WAL rate, not a percentage.

Force a round trip and confirm the segment lands at the destination:

SELECT pg_switch_wal();

A quiet pg_stat_archiver looks the same whether the archive is healthy or nothing is being archived at all. Test it.

Prevention

Alert on pg_stat_archiver.failed_count increasing. This is the missing alert that cost three and a half hours of warning.

Alert on the archive_status ready count. It measures the backlog rather than individual failures, so it catches an archive that succeeds intermittently but cannot keep up.

Express the disk alert in time, not percent. Ninety-five percent of 20 GB was sixteen minutes. Alert below several hours of measured WAL generation.

Give pg_wal its own filesystem. A full pg_wal still stops the cluster, but it stops only the cluster, and the growth becomes its own visible metric.

Keep a ballast file — a few pre-allocated gigabytes, deletable in one command. Document where it is and who may delete it.

Write down that WAL is never deleted by hand, in the runbook, where the person at 02:14 will read it. That suggestion arrived within minutes here because it is the obvious wrong answer, and obvious wrong answers must be pre-empted in writing rather than argued about mid-outage.