Patroni as one implementation — concepts first · PostgreSQL

Patroni is the most widely deployed PostgreSQL HA solution. This lesson is about its concepts rather than its installation, because the concepts transfer and the installation instructions do not.

The components

A Patroni agent on each database host. It starts and stops PostgreSQL, checks its health locally, and is the only thing that should start it — lesson XV-04’s rule about systemd.

A distributed configuration store (DCS) — etcd, Consul or ZooKeeper. It holds the leader lease and the cluster’s intended configuration. It runs on its own nodes, in its own failure domains.

A REST API on each agent, which reports whether that node is the leader and is what a proxy or load balancer queries for routing.

Mapping onto lesson XV-04’s six responsibilities:

Responsibility	Patroni
Detect	Agent health-checks its local PostgreSQL
Decide	Leader lease in the DCS; a majority is required
Fence	Leader demotes itself when it cannot renew the lease
Promote	Agent calls PostgreSQL’s promotion
Reroute	REST API endpoints a proxy can query
Rejoin	Agent runs `pg_rewind` or a fresh base backup

All six. That is what “an HA stack” means concretely.

The timing constraint

Four settings govern everything, and the documentation states a relationship between three of them.

Setting	Default	What it is
`ttl`	30 s (min 20)	The TTL on the leader lock. “The length of time before initiation of the automatic failover process.”
`loop_wait`	10 s (min 1)	How long the agent’s loop sleeps between iterations
`retry_timeout`	10 s (min 3)	Timeout for DCS and PostgreSQL retries. “DCS or network issues shorter than this will not cause Patroni to demote the leader.”
`primary_start_timeout`	300 s	How long a primary may take to recover from failures before failover is triggered

loop_wait + 2 × retry_timeout ≤ ttl

That inequality is documented, and understanding why explains most of how safe automated failover works.

The leader agent must renew its lease before it expires. In the worst case it has just started a sleep (loop_wait), then needs to retry a DCS operation (retry_timeout), then retries a PostgreSQL operation (retry_timeout again). If that worst case exceeds ttl, a healthy leader can lose its lease simply because it was slow, and the cluster fails over for no reason.

The defaults satisfy it exactly: 10 + 2 × 10 = 30 = ttl.

The consequence is that these settings cannot be tuned independently. Lowering ttl to fail over faster without lowering loop_wait and retry_timeout alongside it breaks the inequality and produces spurious failovers — which is lesson XV-04’s most common HA-caused outage, arrived at by a change that looked like an improvement.

How the lease produces fencing

This is the mechanism from lesson XV-03, concretely:

The leader holds a lock in the DCS with a TTL of ttl seconds.
It refreshes that lock on every loop iteration.
If it cannot refresh — for any reason — it demotes its own PostgreSQL. It does not wait to be told; it acts on its own inability to confirm it is still the leader.
Other agents cannot acquire the leader lock until the TTL expires.

Step 3 is the fencing, and step 4 is what makes it safe. By the time any other node can take the lock, the old leader has already demoted itself, because it began demoting the moment it lost contact and the TTL has since elapsed.

The guarantee comes from arithmetic on timeouts rather than from reaching the failing node — which is why it holds during a network partition, the case where every cooperation-based mechanism fails.

Choosing a candidate

maximum_lag_on_failover is “the maximum bytes a follower may lag to be able to participate in leader election.”

This is the setting that stops a stale standby being promoted, and it is the point at which an availability decision becomes a data loss decision. A tight value means a failover may not be possible when the only surviving replica is behind; a loose one means promoting a replica that is missing more than you would have accepted.

synchronous_mode (off, on, quorum) makes Patroni manage synchronous_standby_names itself, so that only a standby known to be current can be elected. That is the zero-RPO configuration, and it carries the availability cost lesson XIV-05 measured.

Where the RTO actually goes, and why the database is the small term

It is natural to assume the failover time is dominated by PostgreSQL. The measurements in this part say otherwise.

Promotion itself was 89 ms from request to accepting connections, because a hot standby has already replayed everything.

Everything else is larger, and most of it is not the database:

Detection is bounded below by ttl — 30 seconds by default. Nothing happens before the lease expires, by design, because acting sooner means acting on less evidence.

Decision adds a DCS round trip and the election.

Promotion is the 89 ms.

Rerouting is where the time goes, and it depends entirely on the mechanism: a proxy noticing via health checks is seconds; a DNS change is bounded by TTL plus whatever resolvers and JVMs are caching, which can be minutes; a VIP move is fast when it works.

Application reconnection is frequently the largest term and the least examined. Pools must notice their connections are dead, which many do only on next use; retry logic must not stampede; and a few hundred clients reconnecting simultaneously is itself a load event, on a cluster that has just started cold with an empty buffer pool.

So a realistic automated RTO is something like 30 seconds of detection, a second of election and promotion, some seconds of rerouting, and however long the application takes — plausibly 45 to 90 seconds in total, of which the database’s own contribution is under a tenth of a second.

Two conclusions follow. Tuning ttl downward to improve RTO trades directly against spurious failovers, and the inequality above bounds how far you can go. And if the RTO is not good enough, the profitable place to look is almost always the client side — because that is where the seconds are.

What to take from this

Patroni supplies all six responsibilities: agent per node, DCS holding a leader lease, REST API for routing.
loop_wait + 2 × retry_timeout ≤ ttl. Documented, and the defaults meet it exactly at 10 + 20 = 30.
Tuning ttl alone to fail over faster produces spurious failovers.
Fencing is the leader demoting itself when it cannot renew, plus a TTL that must expire before anyone else can take the lock.
maximum_lag_on_failover is where availability becomes an RPO decision.
Patroni does not solve routing, pooling, backups, or your locks.
Measured: promotion is 89 ms. The RTO is spent almost entirely elsewhere.

Cross-course references

Kubernetes for Production Sysadmins — Part LXVI (etcd) and Part LXVII (etcd quorum) cover the distributed configuration store Patroni depends on, including the failure mode where the store is healthy and the network to it is not.
Linux for Production Sysadmins — Part LV (Pacemaker and Corosync) covers an older stack solving the same problem, which makes the trade-offs easier to see.
Ansible for Production Sysadmins — Part XXXVI (Drift) covers keeping the configuration Patroni owns and the configuration you own from overwriting each other.