pg_basebackup: its role and its limits · PostgreSQL

pg_basebackup is the low-level API from the previous lesson, driven correctly by a program that holds one connection open, copies the right files, excludes the right files, and writes backup_label. It ships with PostgreSQL and it is the right starting point for physical backups.

Taking one

Read-only / Safea base backup, compressed, with its own WAL— Captured on 18.6 against a small cluster.

$ pg_basebackup -D /backup/base -Ft -z -Xstream -P -c fast

waiting for checkpoint
23854/23854 kB (100%), 0/1 tablespace
23854/23854 kB (100%), 1/1 tablespace

Read-only / Safewhat it produced— Captured on 18.6.

$ ls -la /backup/base/

-rw------- 1 postgres postgres  138651 backup_manifest
-rw------- 1 postgres postgres 3162227 base.tar.gz
-rw------- 1 postgres postgres   17091 pg_wal.tar.gz

Three artefacts, and all three matter.

File	What it is	If you lose it
`base.tar.gz`	The data directory	No backup
`pg_wal.tar.gz`	WAL generated during the backup	Backup cannot reach consistency
`backup_manifest`	Checksums of every file	Nothing to verify against

The flags that matter

-X stream (the default) opens a second connection and streams WAL alongside the copy, so the backup contains everything it needs to reach a consistent state on its own. The alternative, -X none, produces a backup that is unusable without the WAL archive. That is a legitimate choice when you have a reliable archive, and a common way to end up with an unrestorable backup when you do not.

-c fast requests an immediate checkpoint. Without it the backup waits for a spread checkpoint, which on a busy cluster with a large checkpoint_timeout can be several minutes of doing nothing. fast costs a burst of I/O, exactly as lesson XII-04 described.

-P shows progress. On a multi-hour backup this is the difference between knowing it is working and hoping.

-Ft -z produces compressed tar files. -Fp produces a plain directory — larger, and the format pg_combinebackup needs.

--checkpoint=spread is the polite option for a cluster where a burst of checkpoint I/O would be noticed. It trades backup start latency for a gentler write pattern.

-R writes standby.signal and the connection settings, turning the backup into a standby. Part XIV uses this.

Verifying

Read-only / Safepg_verifybackup against the manifest— Captured on 18.6. -n skips WAL verification, which is right when checking the base backup alone.

$ pg_verifybackup -n /tmp/vb

backup successfully verified

The manifest lists every file with its size and checksum. Verification detects truncation, corruption in transit, and files that went missing between backup and storage. It does not prove the backup restores — only that the bytes are the bytes that were written. Proving a restore is lesson XIII-08.

Incremental backup

PostgreSQL 17 added incremental base backups, and they are the largest change to native backup capability in years. The prerequisite is WAL summarization.

Configuration changeenabling WAL summarization— Captured on 18.6. summarize_wal has context sighup, so no restart is needed.

$ ALTER SYSTEM SET summarize_wal = on; SELECT pg_reload_conf();

     name      | setting |       source
---------------+---------+--------------------
summarize_wal | on      | configuration file

With summarization on, an incremental backup takes the previous backup’s manifest as its baseline:

Read-only / Safefull, then incremental after updating 2,000 of 400,000 rows— EXECUTION VALIDATED on 18.6. Test table: 400,000 rows, 92 MB.

$ pg_basebackup -D /backup/full -Fp -Xstream -c fast -P
# ... 2,000 rows updated, CHECKPOINT ...
pg_basebackup -D /backup/incr -Fp -Xstream -c fast -i /backup/full/backup_manifest -P

130846/130846 kB (100%), 1/1 tablespace      <- full
5189/131307 kB (3%),   1/1 tablespace      <- incremental

$ du -sh /backup/full /backup/incr
144M    /backup/full
24M     /backup/incr

Three per cent of the cluster sent. Unchanged blocks are replaced by stub files:

Read-only / Safethe stubs an incremental backup leaves behind— Captured on 18.6.

$ find /backup/incr -name 'INCREMENTAL.*' | wc -l

An incremental backup is not a data directory

Service impact possiblestarting an incremental backup directly— Real error from 18.6.

$ pg_ctl -D /tmp/incrtest start

FATAL:  this is an incremental backup, not a data directory
HINT:  Use pg_combinebackup to reconstruct a valid data directory.

The error is unambiguous and the design is deliberate: there is no way to accidentally start a partial backup and get a partial database. It also means every incremental in the chain must survive. Lose one and everything after it is unusable — which is a different retention problem from a series of independent full backups, and one your retention policy has to be written for.

Read-only / Safereconstructing and verifying the chain— EXECUTION VALIDATED on 18.6. The merged directory was started on port 5434.

$ pg_combinebackup /backup/full /backup/incr -o /backup/merged
pg_ctl -D /backup/merged start
psql -p 5434 -c "SELECT count(*), count(*) FILTER (WHERE pad LIKE 'z%') FROM bulk"

$ du -sh /backup/merged
145M    /backup/merged

server started

rows  | updated_rows
--------+--------------
400000 |         2000

All 400,000 rows, and the 2,000 updated after the full backup carry their new values. The chain reconstructed correctly.

What `pg_basebackup` cannot do

Back up a single database or table. It copies the whole cluster.
Deduplicate across backups. Each full backup is a full copy.
Manage retention. No expiry, no rotation, no catalogue.
Parallelise the copy. One connection, one stream. On a very large cluster this is the practical ceiling.
Encrypt. Compression, yes. Encryption is your problem.
Resume. An interrupted backup starts over.
Restore. There is no pg_baserestore. Restoring is extracting the archive and configuring recovery yourself.

That list is the argument for the tools in lesson XIII-08. It is not an argument against pg_basebackup, which is correct, always present, and the right answer for a great many clusters.

How WAL summarization makes incremental backups possible, and what it costs

To send only changed blocks, the server has to know which blocks changed. The naive approach is to read every block and compare checksums, which costs a full read of the cluster and saves only transfer, not I/O.

summarize_wal instead runs a background process — the WAL summarizer — that reads WAL as it is generated and records, per relation, which block ranges were modified. The summaries live in pg_wal/summaries/. An incremental backup reads the summaries covering the range from the baseline manifest’s LSN to now, and sends exactly the blocks they name.

Three consequences worth planning for.

Summaries must cover the whole gap. wal_summary_keep_time defaults to 10 days (14400 minutes). If summaries covering the baseline’s LSN have been removed, the incremental backup cannot be taken and pg_basebackup says so rather than silently falling back to a full copy. A backup schedule whose full-to-incremental gap exceeds the retention window fails on a night when nobody is watching, so the two settings have to be chosen together.

The summarizer costs CPU and I/O continuously, in proportion to WAL generation rate. On the cluster in lesson XII-02, generating 4,359 kB/s, that is a real if small background cost paid all the time in exchange for a cheaper backup window.

Summarization begins when you enable it, not retroactively. The first backup after turning it on must be a full one, because there are no summaries covering anything earlier.

The saving is proportional to the change rate, not the cluster size. A cluster where 3% of blocks change between backups gets the 24 MB against 144 MB measured above. A cluster where a nightly batch rewrites most of a large table gets very little, and the honest way to find out which you have is to take one incremental and look at the percentage pg_basebackup reports.

What to take from this

pg_basebackup drives the low-level API correctly. Prefer it to hand-rolled scripts.
-X stream makes the backup self-sufficient; -X none makes it depend on your archive.
Keep backup_manifest with the backup, and run pg_verifybackup.
Incremental backups need summarize_wal, and sent 3% — 24 MB against 144 MB — in a measured case.
An incremental is not a data directory. pg_combinebackup reconstructs it, and every member of the chain must survive.
No deduplication, no retention, no parallelism, no encryption, no resume, and no restore command.

Cross-course references

Linux for Production Sysadmins — Part XLVII (Backup strategy) covers retention arithmetic, which changes shape once a backup chain has dependent members rather than independent fulls.
Observability for Production Sysadmins — Part XCI (Backup strategy) covers alerting on the age of the oldest usable full, which is the number an incremental chain makes non-obvious.