When DRBD materially helps HA - and when it does not

DRBD is often chosen because it is the answer to “how do I make this service highly available without buying a SAN”. That is a real and good reason. It is also, on its own, not enough to tell you whether DRBD is the right tool for a given workload. This lesson is the decision.

The one failure DRBD removes

DRBD removes exactly one failure from the picture: the disk that only exists on one node.

Before DRBD, a service pinned to node1’s local storage cannot run anywhere else, because the data is not anywhere else. After DRBD, the same bytes exist on node2, so a cluster manager can start the service there.

That is a large and specific win. Everything else the reader usually hopes for is somebody else’s job:

• Deciding that node1 has failed - Corosync membership.
• Refusing to run the service twice - quorum and fencing.
• Starting the service on node2 - Pacemaker.
• Recovering the application’s own state - the application.
• Protecting against a bad write - backups. DRBD replicates a DELETE FROM or an rm -rf faithfully and instantly.

The cost side: synchronous replication is a latency tax

Protocol C acknowledges a write only after the peer has it on stable storage. Every write therefore pays one network round trip plus the peer’s own commit time.

The arithmetic is simple and worth doing before the deployment, not after:

For a workload doing 200 small synchronous writes per second, 1 ms of added latency is 0.2 seconds of extra wall-clock time per second of work - noticeable but survivable. For a database committing 5000 transactions per second with fsync on each, the same 1 ms is the whole performance budget.

Measure the link before you design around it:

$ ping -c 200 -i 0.2 -q 192.0.2.12

--- 192.0.2.12 ping statistics ---
200 packets transmitted, 200 received, 0% packet loss, time 39840ms
rtt min/avg/max/mdev = 0.104/0.138/0.402/0.031 ms

Protocol A removes the tax by acknowledging before replication, and buys it back as an RPO: the writes in flight when the primary dies are lost. That is a legitimate trade for a cross-site link, and an illegitimate one for a two-node rack pair where the round trip is a tenth of a millisecond anyway.

DRBD against the alternatives

Shared storage (SAN, iSCSI, NFS). One copy of the data, reachable by several nodes. Failover is instant because nothing has to be copied - but the array and the fabric are a shared failure domain, and a serious one. DRBD has no shared component at all: two hosts, two disks, one network link. On the other hand DRBD gives you two copies to keep in agreement, which is a class of problem shared storage does not have.

Choose shared storage when you already own a redundant array with dual controllers and a fabric you trust. Choose DRBD when the storage would otherwise be a single box.

Distributed storage (Ceph, Gluster). Data spread across many nodes with its own replication and recovery. Scales far past DRBD, survives whole-host loss without a cluster manager promoting anything, and is a substantially larger system to run. Below roughly a handful of nodes the operational cost is hard to justify; above a dozen, DRBD’s full-mesh requirement and write latency make it the wrong shape.

Application-level replication. PostgreSQL streaming replication, MySQL replication, Elasticsearch shards, a message broker’s own mirroring. The application understands its own consistency, can replicate a transaction rather than a block, and can serve reads from the replica - which DRBD’s secondary can never do, because it is not mounted.

RPO and RTO, side by side

The RTO column hides a detail worth naming. After an unclean failover, DRBD hands the new primary a filesystem that was never unmounted. XFS replays its journal, which is fast; ext4 does the same; but a filesystem large enough to need a full check adds that check to your RTO, on the worst possible day. Test the failover with a realistic amount of data on it and time what actually happens.

Where DRBD is clearly the right answer

• A two-node or three-node HA pair for a service with no replication of its own: an LDAP server with a file-backed database, a legacy application writing to a local directory, a mail store, a file server.
• An NFS server made highly available: DRBD under the export, Pacemaker owning the filesystem, the export and the virtual IP. This is close to the canonical deployment.
• Virtual machine disks on a small two-host hypervisor pair, with temporary dual-primary for live migration.
• Any case where you want zero RPO and cannot afford or cannot trust a shared array.

Where DRBD is the wrong answer

• Anything that already replicates itself. As above.
• High-throughput write workloads over a slow link. The latency tax is charged on every write and cannot be amortised.
• More than a handful of replicas. DRBD 9 supports up to 32 nodes per resource, but every synchronous replica adds latency and the mesh grows quadratically. Past a few nodes this is Ceph’s problem, not DRBD’s.
• Scale-out read capacity. The secondary is not mounted and serves nothing. If the goal is more read throughput, DRBD delivers none of it.
• Cross-site DR as the only mechanism. Protocol A across a WAN replicates a corruption as faithfully as a good write, and a link outage leaves you with a divergence to reconcile. Use it alongside backups, never instead of them.

Knowledge check