Replication is not high availability · PostgreSQL

Part XIV built a working replica. It is tempting to call that high availability. It is not, and the gap is larger than it looks.

What PostgreSQL gives you

Everything in this part was performed by hand on a live pair, and every step worked exactly as documented:

Capability	Mechanism
Ship changes to another host	Streaming replication
Make a standby writable	`pg_promote()`
Keep divergent histories apart	Timelines and `.history` files
Rejoin a diverged node cheaply	`pg_rewind`

Those are genuinely good primitives. Promotion took under a second in the measurement in lesson XV-02, and pg_rewind rejoined a diverged node by copying 196 MB of a 376 MB cluster.

What it does not give you

Four decisions, and PostgreSQL will never make any of them:

1. Deciding the primary has failed. There is no health check, no quorum, no timeout after which anything happens. A primary that is unreachable to clients but reachable to the standby, or slow rather than dead, or has a full disk, looks identical to a healthy one from inside PostgreSQL.

2. Stopping the old primary from accepting writes. There is no mechanism. This is not a missing feature so much as a category error — a database cannot fence itself, because the situation where fencing is needed is exactly the one where it cannot be trusted to act.

3. Telling clients where the primary now is. A promoted standby does not announce itself. Every connection string still points at the old host.

4. Preventing two primaries. Nothing consults the old primary before a promotion, and nothing informs it afterwards.

Demonstrated: a promotion while the old primary kept running

Data-loss risktwo primaries, neither aware of the other— EXECUTION VALIDATED on 18.6. The standby was promoted; the old primary was deliberately left running. Both were then written to.

$ -- after promoting rbpg-sb, writes were issued to BOTH clusters
SELECT origin, count(*) FROM ha GROUP BY origin;

--- old primary (timeline 1) ---     --- new primary (timeline 2) ---
 origin    | count                  origin    | count
-------------+-------              -------------+-------
OLD-PRIMARY |   120               NEW-PRIMARY  |   300
primary     |   500               primary      |   500

old primary: SELECT count(*) FROM pg_stat_replication;  -> 0
old primary: SELECT pg_is_in_recovery();                -> f
new primary: SELECT pg_is_in_recovery();                -> f

Two clusters, 500 rows in common, divergent tails, both accepting writes, neither aware the other exists.

PostgreSQL raised no error and had no setting that would have prevented this. The promotion is a local decision made by the standby; the old primary is neither consulted nor informed.

Lesson XV-03 covers what this costs — the 120 rows on the losing side were later discarded, permanently, when the split was resolved.

The manual failover, honestly

Doing it by hand is legitimate for many systems. It is worth being clear about what it involves at three in the morning:

Notice the primary is down — and be right about that.
Confirm the standby is caught up, or accept the loss.
Ensure the old primary cannot write. Power it off, revoke its storage, block it at the network. This is the step people skip.
Promote.
Repoint every client.
Rebuild or pg_rewind the old primary.

Steps 1 and 3 are where the mistakes happen, and both involve judgement under pressure with incomplete information. Steps 4 to 6 are mechanical.

Manual failover is a real choice, and it should be a deliberate one

Automated failover is not obviously better. It adds a distributed system to your database, and distributed systems fail in ways single systems do not — most commonly by failing over when nothing was wrong.

A spurious failover on a healthy cluster is an outage caused entirely by the tool that exists to prevent outages, and it is common enough to be the main argument against automation on systems that do not need it.

Choose manual when:

Your RTO is measured in tens of minutes and someone is on call.
Failures are rare and a human can be trusted to look before acting.
The operational cost of running a consensus system exceeds the downtime it saves.

Choose automated when:

Your RTO is minutes and nobody can respond that fast.
You have several clusters, so per-cluster manual procedures do not scale.
The team is willing to operate, monitor and test the HA stack as a production system in its own right — because it is one.

The wrong answer is an automated stack nobody understands, tests, or watches. That is strictly worse than a documented manual procedure.

What availability actually depends on

A useful exercise: list what has to be true for the database to serve traffic, and notice how little of it is PostgreSQL.

The host is up.
Its storage is healthy and not full.
The network path from clients is intact.
DNS or the proxy points somewhere useful.
Connections are available (lesson IV-02).
No single query has taken a lock everything queues behind (Part IX).
pg_wal has not filled (lesson XII-03).
Someone would notice if any of this were false.

A second machine addresses the first two. Replication solves the narrowest of the failure modes that actually cause outages, and the observability in Part XVI addresses the last one, which is frequently the binding constraint.

What an automated failover has to get right, and why consensus is not optional

Every automated failover system faces the same problem, and it is a hard one: from inside the cluster, a failed primary and an unreachable primary look identical.

A standby that cannot reach the primary knows exactly one thing: it cannot reach the primary. It does not know whether the primary crashed, whether the network between them broke, or whether it is itself the partitioned node. Acting on that information alone is how split-brain happens.

The standard answer is consensus. Rather than one node deciding, a majority must agree, and a node that cannot reach a majority must refuse to act.

That is why HA stacks come with a consensus store — etcd, Consul, ZooKeeper — and why the arithmetic matters. Three nodes tolerate one failure. Two nodes tolerate none, because neither one alone is a majority. A two-node HA cluster is not highly available; it is a system that requires a human whenever the two disagree, which is the same as manual failover with more moving parts.

Consensus gives three things that follow from that majority rule:

Leader election. Exactly one node holds the leader lease at a time.

A leader lease with a timeout. A leader that cannot renew its lease must demote itself before the lease expires, so that even a partitioned primary stops accepting writes without anyone reaching it — this is the one form of self-fencing that works, and it works because it fails closed.

A place clients can ask. The consensus store knows who the leader is, which is the routing problem in lesson XV-06.

The cost is that the consensus store becomes part of your availability chain. If etcd is down, the HA stack cannot make decisions, and a well-built one responds by refusing to act rather than by guessing. You have added a system that can itself fail, in exchange for correctness when it does not.

What to take from this

PostgreSQL supplies replication, promotion, timelines and pg_rewind. Good primitives.
It supplies no failure detection, no fencing, no client routing, and nothing preventing two primaries.
Demonstrated: a promotion succeeded while the old primary kept accepting writes, with no error and no way to prevent it.
Manual failover is a legitimate choice. An untested automated stack is worse than a documented manual one.
A two-node HA cluster is not highly available; a majority cannot exist.
Most outages are not the failure mode replication addresses.

Cross-course references

Linux for Production Sysadmins — Part LII (High availability fundamentals) covers the same claim for services generally, and Part LV (Pacemaker and Corosync) covers what a real HA stack supplies beyond a copy of the data.
Proxmox — Part XII (High availability) covers the hypervisor’s own answer, which restarts a VM and knows nothing about the database inside it.