Identifying the right cluster pattern - a decision procedure

The previous lessons in this part described the patterns. This one is the procedure for picking between them, because in practice the pattern is not chosen - it is inherited from whoever set up the first node, and then defended.

The procedure is five questions. Answer them about the service in front of you, in order, before opening any documentation about Pacemaker.

Question 1: where does state that must survive a restart live?

Not “does the service have state” - almost everything does. The question is where the state lives that you would miss.

• Nowhere. Every request is self-contained; a restart loses nothing. → Stateless.
• In an external system the service talks to over the network: a database, an object store, a queue. → External state.
• In memory, and it matters. Sessions, caches with no backing store, accumulated counters. → Replicated state or redesign.
• On local disk, in a format the service owns: an embedded database, a mail spool, an upload directory. → Shared or replicated storage.

This one question eliminates most of the option space. Get it wrong - by answering “nowhere” for a service that quietly writes to /var/lib - and every subsequent decision is built on sand. Check rather than assume:

sudo lsof -p "$(pidof myapp)" | awk '$4 ~ /[0-9]+[uw]/ {print $9}' | sort -u

Question 2: can two instances run at the same time?

Run them and find out, in the lab. The failure is rarely subtle.

• Yes, freely. No shared mutable resource, or the shared resource arbitrates. → Active-active is available.
• Yes, with coordination. They can both run if something serialises access - a distributed lock, a cluster filesystem, a database row lock. → Active-active with a lock manager.
• No. A second instance corrupts data, double-processes work, or fails to bind a port or a licence. → Active-passive or leader-elected.

Question 3: what RPO can the service tolerate?

How much recent work may be lost when the active node dies?

• Zero. Every acknowledged write must survive. → Synchronous replication (DRBD protocol C, a synchronous database replica) or genuinely shared storage.
• Seconds. → Asynchronous replication.
• Hours. → Backups and a rebuild. This is a perfectly respectable answer, and it is the right one far more often than cluster designs assume.

Question 4: what RTO can the service tolerate?

How long may it be unavailable?

• Seconds. → Active-active behind a load balancer, or a pre-warmed standby.
• A minute or two. → Active-passive with Pacemaker. This is what a promote, mount and service start actually costs.
• Tens of minutes. → Rebuild from configuration management, restore data, start. No cluster required.

Be honest here, and get the number from whoever owns the service rather than inventing it. “Highly available” is not an RTO. A cluster bought to reduce an outage from ten minutes to one minute, for a service whose users would not notice either, has added a fencing configuration, a split-brain failure mode and a second node to patch, in exchange for nothing.

Question 5: does the client recover on its own?

The cluster’s RTO and the user’s RTO are different numbers.

• Yes - the client retries, reconnects, or is behind a load balancer that does. → The cluster’s failover time is the real RTO.
• No - the client holds a connection, caches a DNS answer for an hour, or needs a human to press retry. → Failover restores the service and not the users. Fix the client, or accept the number.

Mapping answers to patterns

The last row is a real answer. It appears in this table deliberately, because it is the one nobody writes down.

Worked examples

A static content web server. Q1 nowhere. Q2 yes. Q3 not applicable. Q4 seconds. Q5 yes, browsers retry. → Stateless, N instances, load balancer. No cluster manager, no shared storage, no fencing. Adding Pacemaker here makes the service less available, not more, because it adds a component that can fail.

A PHP application storing sessions in /var/lib/php/sessions. Q1 local disk. Q2 no - a user’s session is on one node only. Q4 seconds. The correct move is not a cluster: it is to answer Q1 differently by putting sessions in Redis or the database, which turns the service into the previous example. Clustering the filesystem here solves a problem that should not exist.

PostgreSQL. Q1 local disk. Q2 no, absolutely not. Q3 usually zero. Q4 a minute. → Active-passive. But the tool is PostgreSQL’s own replication and a promotion, not DRBD under the data directory, because the database understands what a committed transaction is and a block device does not. The pattern is active-passive; the mechanism is the application’s.

Redis as a cache. Q1 in memory, and it does not matter - losing the cache costs latency, not data. Q4 seconds. → Treat it as stateless. Run two, let clients fail over, and let the cache refill. A cluster manager protecting a cache is effort spent on the wrong risk.

A batch job scheduler. Q1 local disk (job state). Q2 no - two schedulers run every job twice. Q4 tens of minutes, because a job that starts late is usually fine. → Leader-elected singleton, or an active-passive Pacemaker resource. The RTO answer means this does not need to be fast, only correct.

A vendor licence daemon locked to a MAC address. Q2 no, and it cannot be made yes. Q5 usually no. → Active-passive with the MAC as a cluster-managed resource, if the vendor supports it; otherwise this is a documented single point of failure and the mitigation is a spare licence, not a cluster.

Writing the decision down

The reason to run this procedure explicitly is that the answers are the design rationale, and the next operator will otherwise have to reverse-engineer it from the pcs configuration.

A short block in the service’s runbook is enough:

Service:      billing-api
State:        external (PostgreSQL) - Q1
Concurrency:  safe, no shared mutable local state - Q2
RPO:          zero, owned by the database tier
RTO:          30 s, agreed with service owner 2026-08-11
Client:       retries with backoff, behind HAProxy - Q5
Pattern:      stateless, 3 instances, no cluster manager
Reviewed:     2026-08-11

When someone proposes adding DRBD to this service in two years, that block is the answer. Without it, the proposal wins by default, because nobody can remember why it was not needed.

Knowledge check