What an HA stack must supply beyond PostgreSQL · PostgreSQL

The previous three lessons established what is missing. This one turns that into a list you can evaluate a tool against.

The six responsibilities

1. Detect. Determine that the primary is unhealthy — and be right. Requires health checks that distinguish down from slow, from unreachable-from-here, from up but full disk.

2. Decide. Determine that a failover should happen. Distinct from detection: a single standby’s opinion is not sufficient evidence, which is where consensus and a majority come in.

3. Fence. Make certain the old primary cannot write, and verify it. Lesson XV-03.

4. Promote. The only one PostgreSQL supplies. pg_promote(), measured at 89 ms.

5. Reroute. Get clients to the new primary. Lesson XV-06.

6. Rejoin. Return the old primary to service as a standby. pg_rewind, lesson XV-07 — PostgreSQL supplies the tool but not the orchestration.

Evaluating a tool

Reputation is a poor guide. These questions are better, and each maps to a responsibility above.

How does it detect failure, and what does it do about ambiguity? Ask specifically what happens when a node is slow rather than dead. A tool that treats “no response in 10 seconds” as death will fail over during a long checkpoint.

How does it decide, and what is the quorum? If a single node can decide, it can decide wrongly during a partition. If a majority is required, count the nodes and confirm a majority can exist — a two-node cluster cannot form one.

How does it fence, and what happens when fencing fails? The important half is the second. A tool that promotes anyway when it cannot fence has traded a recoverable outage for an unrecoverable one. A tool that refuses to promote is behaving correctly and will be blamed for the outage.

How do clients find the new primary, and how fast? DNS TTL, VIP failover time, proxy reconfiguration. This is frequently the largest term in the real RTO and the least examined.

How does it rejoin, and does it require human intervention? A stack that fails over automatically and then needs a person to rebuild the old primary has automated the easy half.

What does it do when it is broken? The consensus store is unreachable; the agent has crashed; the agent is running but its view is stale. The right answer is that PostgreSQL keeps serving and the stack refuses to act. A stack that fails over because it cannot see clearly is worse than no stack.

The failure modes the stack itself introduces

An HA stack is a distributed system running against your database, and it adds failure modes that did not exist before:

Failover when nothing was wrong. The most common HA-caused outage. Some transient condition — a GC pause, a slow disk, a network blip — looks like death, and a healthy cluster is failed over for no reason, with a rebuild and a client-reconnection storm as the reward.

Flapping. Repeated failovers as conditions oscillate around a threshold.

Failing over to a standby that is far behind, silently accepting an RPO nobody agreed to, because “promote” and “promote something useful” are different operations.

The stack itself becoming the outage. Consensus store down, agents unable to agree, and a database that is fine but that the stack has decided to demote.

Configuration drift, where the stack’s idea of the topology and the actual topology diverge — most often after a manual intervention that did not go through the stack.

None of these are arguments against automation. They are the reason an HA stack must be operated, monitored and tested as a production system in its own right, and the reason “we installed Patroni” is a beginning rather than an outcome.

Does the complexity earn its place?

Honest arithmetic beats instinct here.

Estimate your actual availability need. 99.9% is 8.8 hours of downtime a year; 99.99% is 53 minutes. Which one is written down anywhere, and what happens if you miss it?

Estimate manual failover time. Detection plus decision plus fencing plus promotion plus rerouting. Twenty to sixty minutes is typical with a practised team and a written runbook, and it is worse than people estimate because steps 1 to 3 involve judgement.

Count the primary failures you have actually had. Not incidents — primary host failures.

Estimate the cost of a spurious failover, and multiply by how often you think the stack will produce one.

For many systems the arithmetic favours a hot standby, good monitoring and a rehearsed manual procedure. For systems where minutes matter, or where there are enough clusters that per-cluster manual procedures do not scale, automation wins clearly. What almost never wins is automation installed because it seemed like best practice and then left unexercised.

Where the HA stack should run, and the failure that keeps recurring

A detail that decides whether a stack works: the agent should run on the database host itself, not on a separate monitoring machine.

The reason is fencing. An agent co-located with PostgreSQL can stop it, demote it, or refuse to let it start, using local means that need no network. A remote agent can only ask, over the network that may be the thing that failed — and lesson XV-03’s table is unambiguous that mechanisms requiring the failing node’s cooperation are the weak ones.

Co-location gives four things a remote agent cannot have:

Local fencing. The agent demotes its own PostgreSQL without needing anything external. This is the leader-lease self-demotion from lesson XV-03, and it works during a partition precisely because it is local.

Accurate local health. Disk space, memory pressure, whether the postmaster is actually accepting connections on the socket. A remote check sees a TCP port.

Authority over its own node. The agent starts and stops PostgreSQL, which means the topology cannot drift through someone starting the database by hand and the stack not knowing.

No dependency on the network for the decision to stop. Stopping is the safe direction, and it must work when nothing else does.

The recurring failure is running PostgreSQL under a service manager that does not know about the HA stack. systemd restarts a PostgreSQL that the agent had deliberately demoted; the node comes back as a primary; you now have two. This is common enough to be worth stating as a rule: when an HA agent manages PostgreSQL, the service manager must not. Disable the unit, or configure the agent as the only thing that starts it.

The consensus store is the opposite case: it should not be co-located with the databases, because a store sharing failure domains with the nodes it arbitrates between cannot arbitrate a failure of those domains. Three consensus nodes across three failure domains, separate from the database hosts, is the arrangement that works.

What to take from this

Six responsibilities: detect, decide, fence, promote, reroute, rejoin. PostgreSQL supplies one.
Evaluate a tool by what it does with ambiguity and by what it does when it is itself broken.
The stack adds failure modes: spurious failover, flapping, promoting a stale standby, becoming the outage, configuration drift.
Do the arithmetic: availability target, manual failover time, actual failure rate, cost of a spurious failover.
Run the agent on the database host; run the consensus store elsewhere.
If an agent manages PostgreSQL, systemd must not.

Cross-course references

Linux for Production Sysadmins — Part LII (High availability fundamentals), Part LIV (Fencing and STONITH) and Part LVI (Keepalived and VRRP) cover failure detection, fencing and address failover as separate responsibilities, which is the decomposition this lesson uses.
Kubernetes for Production Sysadmins — Part LXVI (etcd) covers the consensus store an HA manager needs, and what it means when that store is the thing that fails.