PostgreSQL on distributed storage

Ceph, EBS, NFS, SAN, a Kubernetes storage class — from PostgreSQL’s point of view they are all “somewhere the writes go”, and the only number that matters is how long an fsync takes.

The ceiling

$ pg_test_fsync
# then, separately:
pgbench -c 1 ...

fdatasync latency:        854 µs
predicted ceiling:      ~1,170 commits/second   (1 / 0.000854)
pgbench measured:          842 commits/second

The arithmetic is unavoidable. A durable commit waits for WAL to reach stable storage, so one session cannot commit faster than one fsync. Whatever the storage layer’s latency is, that is your single-session commit ceiling.

Does it honour fsync?

Everything in Part XII rests on this. If a storage layer acknowledges an fsync before the data is durable, PostgreSQL’s durability guarantee is void and it has no way to know.

The layers that can lie:

• A disk’s volatile write cache, if not battery-backed and not disabled.
• A RAID controller cache without a working battery.
• A virtualisation layer with caching enabled.
• NFS mounted with the wrong options.
• A distributed storage layer configured for throughput over durability.

pg_test_fsync reports what the storage claims. A result far better than physics allows means something is lying:

pg_test_fsync -f /var/lib/postgresql/18/main/pg_wal/testfile

A network storage layer reporting a 20 µs fsync has not written anything to a network in 20 µs.

The only honest test is a power-loss test on the actual hardware — which most people cannot do, so the practical position is: know what your layer claims, know whether the vendor documents durability, and treat crash recovery as the thing that has to work rather than as an edge case.

What replicated storage does and does not do

Distributed storage replicates blocks. It knows nothing about transactions, WAL, or PostgreSQL.

It does address: a disk failing, a node failing, a rack failing — depending on placement.

It does not address:

• A DROP TABLE, which it replicates faithfully. Lesson XIV-01.
• Corruption written by PostgreSQL, replicated to every copy.
• Anything above the block layer — the whole of Parts VII to XI.
• Availability. Storage surviving does not make the database available; something must still start PostgreSQL somewhere with that storage attached, which is Part XV.

What to take from this

• fsync latency is a hard ceiling on single-session commit rate. Measured: 854 µs predicted ~1,170/s and pgbench measured 842.
• An order of magnitude in latency is an order of magnitude in commit rate. Nothing else compensates.
• Concurrency raises throughput by sharing flushes — 32,069 tps at 64 clients — and does not reduce per-transaction latency.
• Establish whether the layer honours fsync. A result better than physics allows means something is lying.
• Replicated storage protects against hardware and replicates every logical error faithfully.
• Put pg_wal on the lowest-latency storage you have. WAL is the worst possible workload for a network round trip.

Cross-course references

• Ceph & Distributed Storage — Part XXXV (RBD architecture), Part XXXVIII (RBD performance) and Part LXVIII (OSD latency) cover what a database workload asks of distributed storage, and Part IV (Failure domains) covers whether your replicas share one.
• Linux for Production Sysadmins — Part XLI (Storage Performance) covers measuring the resulting flush latency rather than accepting a vendor figure.

Quiz