Runbook: Reclaim Space on a Data Volume Approaching Full

1 · Prerequisites

Confirm every item is in place before any state change.

• Shell access to the database host with enough privilege to read directory sizes and to remove files that are not PostgreSQL data
• A database connection, if the cluster is still running
• The ability to grow the volume, or an agreed alternative source of space
• Knowledge of whether WAL archiving and replication slots are in use, because both retain WAL regardless of any size setting
• Authority to act quickly, since a volume filling at a measured rate has a deadline
• The two prohibitions in this runbook read and understood before starting, because they will be suggested

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

• · Measure where the space is, by directory. du -sh /var/lib/postgresql/18/main/* | sort -h | tail -10. This distinguishes a full pg_wal from a large base from something that is not PostgreSQL at all.
• · **Check pg_wal against max_wal_size.** du -sh <datadir>/pg_wal and SHOW max_wal_size;. A pg_wal much larger than max_wal_size means something is retaining WAL, because max_wal_size is a checkpoint pacing target and not a cap.
• · Check the archive backlog. ls <datadir>/pg_wal/archive_status | grep -c ready and SELECT archived_count, last_archived_time, failed_count, last_failed_wal, last_failed_time FROM pg_stat_archiver;. A failing archive retains every unarchived segment.
• · Check replication slots. SELECT slot_name, active, wal_status, safe_wal_size, restart_lsn FROM pg_replication_slots; An inactive slot retains WAL indefinitely, and wal_status other than reserved says so directly.
• · Check temporary files. du -sh <datadir>/base/pgsql_tmp and SELECT datname, temp_files, pg_size_pretty(temp_bytes) FROM pg_stat_database ORDER BY temp_bytes DESC;. A reporting workload can produce hundreds of gigabytes of temporary files in an hour.
• · Check the server log directory. Logs frequently share the data volume and are frequently the easiest thing to remove safely.
• · Measure the fill rate. Two df readings a minute apart give you the deadline, which is the number that decides whether you have time to grow the volume or need the ballast file now.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1. 1**Free space from somewhere that is not pg_wal.** In order of preference: grow the volume; remove non-PostgreSQL files from the same filesystem; delete the ballast file if one exists.
2. 2Grow the volume if you can. On cloud or LVM storage this is usually minutes, and it is always the correct first answer because it costs nothing else.
3. 3Remove things that are not database data. Old server logs, a forgotten dump, a core file, an unpacked tarball. du -sh /var/lib/postgresql/* and find <datadir>/.. -maxdepth 2 -size +1G find these quickly.
4. 4Delete the ballast file if the estate keeps one. A pre-allocated file of a few gigabytes exists precisely for this moment. Deleting it buys minutes; it does not fix anything.
5. 5Fix the cause of WAL retention, because otherwise the space refills. If the archive is failing, repair the destination or repoint archive_command, which is reloadable. If an abandoned slot is holding WAL, confirm it is genuinely abandoned and drop it.
6. 6Watch the backlog drain rather than assuming it will. watch -n 5 'ls <datadir>/pg_wal/archive_status | grep -c ready'. The incident is over when that count reaches zero and pg_wal has fallen back toward max_wal_size — not when the cluster starts.
7. 7If a reporting workload filled the volume with temporary files, cap it per role. ALTER ROLE reporting SET temp_file_limit = '20GB'; This converts "the volume filled" into "one query failed", which needs no operator.
8. 8**If the cluster has already stopped with a PANIC, free space first and then start it.** It will replay from the last checkpoint and then work through the archive backlog. Do not change any durability setting to make it start.
9. 9**Consider moving pg_wal to its own filesystem** as the durable fix. A full pg_wal still stops the cluster, but it stops only the cluster rather than everything sharing the volume, and its growth becomes a visible metric of its own.
10. 10**Do not reclaim space with VACUUM FULL during business hours.** It takes ACCESS EXCLUSIVE on the table for its whole duration and every reader queues behind it. If bloat is genuinely the cause, measure it with pgstattuple and schedule a rewrite properly.
11. 11Re-measure and record. Where the space went, what was freed, what the fill rate was, and how much headroom now exists expressed in hours rather than percent.

4 · Verification

Confirm the procedure actually fixed the problem.

• ✓df -h shows headroom expressed in hours at the measured write rate, not as a percentage. Ninety-five percent of a 20 GB volume is sixteen minutes.
• ✓ls <datadir>/pg_wal/archive_status | grep -c ready returns zero, and pg_stat_archiver.failed_count has stopped increasing.
• ✓du -sh <datadir>/pg_wal has fallen back to roughly max_wal_size plus normal churn.
• ✓SELECT slot_name, active, wal_status FROM pg_replication_slots; shows no slot with a wal_status other than reserved.
• ✓A forced archive round trip lands at the destination: SELECT pg_switch_wal(); then confirm the segment appears. A quiet archiver looks the same whether it is healthy or doing nothing.
• ✓The cluster accepts connections and pg_is_in_recovery() returns false.
• ✓No durability setting was changed. SELECT name, setting FROM pg_settings WHERE name IN ('fsync','full_page_writes','synchronous_commit'); matches what it was before.
• ✓The alerting gap that let the volume reach this state has been identified and a follow-up action recorded.

5 · Rollback

If verification fails, undo the procedure in reverse order.

• ↶Freeing space is not reversible in a meaningful sense, and does not need to be. The rollback section here covers the changes that accompany it.
• ↶If archive_command was repointed, revert it once the original destination is healthy, and confirm the segments archived to the temporary location are copied to the permanent one. WAL that exists only in a temporary location is a gap in the recovery window.
• ↶If a replication slot was dropped, the standby that used it may no longer be able to catch up. Confirm it reconnected; if the WAL it needed is gone, rebuild it from a base backup.
• ↶If temp_file_limit was set aggressively during the incident, review it afterwards against what the reporting workload actually needs rather than leaving an emergency value in place.
• ↶If the ballast file was deleted, recreate it once the volume has headroom. A ballast file that was used and never restored is not available for the next incident.
• ↶If files were removed and it later turns out something needed them, they are gone. This is why the procedure removes non-PostgreSQL files by name rather than by pattern.

6 · Escalation

When the runbook isn't enough, contact:

• · Somebody proposes deleting files from pg_wal: escalate and stop them. Those segments hold transactions already acknowledged to clients; deleting them destroys the ability to recover and the cluster will refuse to start once it notices the gap.
• · Somebody proposes fsync = off to make the cluster start: escalate and stop them. It removes the guarantee that anything committed afterwards survives a crash, on a machine that has just fallen over, and converts one outage into silent corruption.
• · The archive destination cannot be restored quickly: escalate to whoever owns it. Repointing archive_command at temporary storage is a legitimate emergency action and it creates a follow-up obligation to consolidate the archive.
• · A replication slot is holding WAL and its consumer cannot be identified: escalate rather than dropping it. Dropping a slot whose standby is merely slow forces a rebuild of that standby.
• · The volume cannot be grown and there is nothing safe to delete: escalate to the platform owner immediately with the measured fill rate and the resulting deadline. This is a capacity emergency, not a database one.
• · The cluster has already PANICked and space cannot be freed: escalate. The cluster is down and every minute of a wrong decision — a manual WAL deletion, a durability setting — makes the recovery worse.

References

1. PostgreSQL 18 documentation, WAL Configuration
2. PostgreSQL 18 documentation, Continuous Archiving and Point-in-Time Recovery
3. PostgreSQL 18 documentation, Reliability and the Write-Ahead Log
4. PostgreSQL 18 documentation, pg_replication_slots