Dual-primary DRBD - what it actually requires

DRBD will let both nodes be Primary at the same time. One option in the net section turns it on, and the option is easy to find. What is harder to find is the rest of the sentence: dual-primary is a block-layer capability, and by itself it changes nothing about the filesystem sitting on top.

What dual-primary does and does not give you

Enabling dual-primary means DRBD accepts writes from both nodes and replicates each node’s writes to the other. Every block written on node1 appears on node2 and vice versa.

That is genuinely all it does. It does not:

• coordinate the two filesystems’ journals,
• invalidate the other node’s page cache when you write,
• serialise metadata updates,
• or stop the two kernels from allocating the same inode.

XFS and ext4 assume they are the only writer on the device. That assumption is baked into their caching and journalling. Put either of them on a dual-primary DRBD device, mount on both nodes, and the first concurrent metadata operation corrupts the filesystem - not eventually, but at once. xfs_repair cannot reconstruct the result; the recovery is a restore from backup.

This is the same failure the course covers for a shared SAN LUN. DRBD does not change it. Two nodes writing one filesystem image need a filesystem designed for two writers, whatever the block layer underneath happens to be.

The three preconditions

Dual-primary is supported, and it is correct, when all three of these hold. None is optional and none substitutes for another.

1. Protocol C. Dual-primary is only permitted with fully synchronous replication. With protocol A or B a write can be acknowledged before the peer has it, so the two nodes can read different data for the same block - which breaks the distributed lock manager’s assumptions immediately.

2. A cluster-aware consumer. Either a cluster filesystem (GFS2 or OCFS2) with the DLM underneath it, or an application that does its own cluster-wide locking and never assumes exclusive ownership of the device. A raw device handed to a clustered database that manages its own locking qualifies. A plain mkfs.xfs does not.

3. Working fencing. Both DRBD-side fencing and cluster STONITH. With two writable copies, a partition means both sides keep taking writes, and the resulting split brain is not a recoverable one-sided divergence - it is two divergent histories on a filesystem that a lock manager believed was coordinated. Fencing is what stops the second writer before it starts. The mechanics belong to the fencing part of this course and to the DRBD in a cluster lesson; the point here is that dual-primary without it is not a degraded configuration, it is an unsafe one.

The configuration

resource shared {
    device    /dev/drbd0;
    disk      /dev/sdb1;
    meta-disk internal;

    net {
        protocol C;
        allow-two-primaries yes;
        after-sb-0pri discard-zero-changes;
        after-sb-1pri discard-secondary;
        after-sb-2pri disconnect;
    }

    on node1 { node-id 0; address 192.0.2.11:7788; }
    on node2 { node-id 1; address 192.0.2.12:7788; }
}

Note after-sb-2pri disconnect. In a dual-primary resource the two-Primary split-brain case is the expected one, and there is no policy that can pick a winner correctly when both sides took coordinated filesystem writes. disconnect leaves both sides StandAlone and waits for a human, which is the only honest answer.

Bring it up and confirm the option actually loaded:

sudo drbdadm adjust shared
sudo drbdadm dump shared | grep -i allow-two-primaries

Then, on both nodes:

sudo drbdadm primary shared
sudo drbdadm status shared

Both nodes should report role:Primary and peer-disk:UpToDate.

Putting a cluster filesystem on top

The DLM and mkfs.gfs2 steps are the same as for a shared LUN, with one difference that catches people: the journal count. GFS2 needs one journal per node that will mount, and adding journals later is a separate operation.

# 0. Fencing first. The DLM will not grant locks in a cluster
#    that cannot fence, so a GFS2 mount hangs at the first
#    membership change.
sudo pcs property set stonith-enabled=true
sudo pcs stonith status

# 1. BOTH nodes: the lock managers, owned by Pacemaker as a
#    cloned group so they start and stop in the right order.
sudo pcs resource create dlm ocf:pacemaker:controld \
op monitor interval=30s on-fail=fence --group locking
sudo pcs resource clone locking interleave=true

# 2. BOTH nodes: DRBD Primary on both sides.
sudo drbdadm primary shared
sudo drbdadm status shared

# 3. ONE node only: the filesystem. -t is clustername:fsname,
#    -j is one journal per node that will ever mount it.
sudo mkfs.gfs2 -p lock_dlm -t mycluster:shared -j 2 /dev/drbd0

# 4. BOTH nodes: mount. In production Pacemaker owns this as a
#    cloned ocf:heartbeat:Filesystem resource, not /etc/fstab.
sudo mount /dev/drbd0 /mnt/shared

In Pacemaker the DRBD resource becomes a promotable clone that is allowed two promoted instances:

sudo pcs resource create drbd_shared ocf:linbit:drbd \
  drbd_resource=shared \
  promotable promoted-max=2 promoted-node-max=1 \
  op monitor interval=20s

promoted-max=2 is what makes this dual-primary at the cluster layer. Leave it at the default of 1 and Pacemaker will demote the second node no matter what allow-two-primaries says.

Temporary dual-primary for live migration

The most defensible use of dual-primary is the narrowest one: migrating a running virtual machine between two hosts. For the few seconds of the handover both hypervisors need write access to the same block device. There is no cluster filesystem involved - the guest owns the device and only one guest exists.

Enable it for the migration and turn it off afterwards, without editing the resource file:

# Before the migration, on either node:
sudo drbdadm net-options --allow-two-primaries=yes vmdisk

# ... perform the live migration ...

# Immediately afterwards:
sudo drbdadm net-options --allow-two-primaries=no vmdisk
sudo drbdadm dump vmdisk | grep -i allow-two-primaries

Resource-level quorum as a second line of defence

DRBD 9 can refuse writes on a node that has lost contact with a majority of the resource’s replicas, independently of the cluster manager. On a resource where two nodes may both write, this is worth having:

resource shared {
    options {
        quorum majority;
        on-no-quorum io-error;
    }
}

quorum majority requires a majority of nodes holding the data to be reachable. on-no-quorum io-error makes the losing side return I/O errors rather than silently accepting writes it can never replicate - the filesystem goes read-only and the application fails loudly, which is what you want.

This does not replace STONITH. A node returning I/O errors is still a node the cluster cannot account for, and a wedged kernel returns nothing at all. It narrows the window; fencing closes it.

When single-primary is the right answer

Most workloads that reach for dual-primary do not need it. Ask what concurrency actually buys:

• Two nodes must write the same files at the same time. This is the real case. GFS2 or OCFS2 on dual-primary DRBD, or a distributed filesystem such as CephFS.
• Two nodes must read the same files, one writes. Not a dual-primary case. A single-primary DRBD with an NFS export, or a read-only mount pattern, is simpler and safer.
• Failover must be fast. Not a dual-primary case either. A single-primary resource that Pacemaker promotes takes seconds, and dual-primary does not make the application’s own recovery any faster.
• Live migration of virtual machines. Temporary dual-primary, as above.

Single-primary DRBD with a normal XFS filesystem, owned by one Pacemaker resource group, has a far smaller failure surface: no DLM, no cluster filesystem, no lock traffic on the cluster network, and a split brain that is recoverable by discarding one side. Reach for it first and make dual-primary justify itself.

Knowledge check