DRBD primary/secondary and split-brain handling

DRBD has a primary (writable) and a secondary (replica). A network partition can create two primaries - split brain. This lesson covers how DRBD detects and recovers.

Primary and secondary

• Primary: the writable node. The application reads and writes to /dev/drbd0.
• Secondary: the replica. DRBD receives writes from the primary and applies them.

Promote a secondary to primary:

sudo drbdadm primary myapp

Demote a primary to secondary:

sudo drbdadm secondary myapp

Role names are lowercase sub-commands. drbdadm Primary and drbdadm Secondary are not valid on DRBD 9 utilities.

For HA, Pacemaker manages these automatically.

How DRBD actually detects split brain

A network partition can leave both nodes writing to their own copy. That is split brain.

DRBD does not notice it while the partition lasts. There is no heartbeat protocol that flags the condition mid-outage. Each node keeps a set of data generation UUIDs describing its write history. While the link is down, neither node can see the other’s UUIDs, so neither knows anything is wrong.

Detection happens on reconnect:

• The link comes back and the two nodes exchange generation UUIDs.
• DRBD compares the histories.
• If each node has writes the other never saw, neither history is an ancestor of the other. DRBD declares split brain.
• DRBD drops the connection. Both sides sit in StandAlone and the resource stops replicating until a human acts.

Split-brain policy handlers

DRBD does not have three escalating recovery policies. It has three handlers, and the numeral is a count of how many nodes were in the Primary role when the split was detected:

• after-sb-0pri - neither node was Primary.
• after-sb-1pri - one node was Primary, one Secondary.
• after-sb-2pri - both nodes were Primary.

Each handler takes a required policy argument. The argument is what actually decides who loses data.

after-sb-0pri accepts disconnect, discard-younger-primary, discard-older-primary, discard-zero-changes, discard-least-changes and discard-node-NODENAME.

after-sb-1pri accepts disconnect, consensus, discard-secondary, call-pri-lost-after-sb and violently-as0p.

after-sb-2pri accepts disconnect, call-pri-lost-after-sb and violently-as0p.

A production-safe starting point:

resource myapp {
    net {
        after-sb-0pri discard-zero-changes;
        after-sb-1pri discard-secondary;
        after-sb-2pri disconnect;
    }
}

discard-zero-changes only resyncs automatically when exactly one side wrote anything; if both wrote, it disconnects. discard-secondary trusts the node that was serving the application. disconnect on the two-Primary case means DRBD never guesses when both sides took writes.

For data you cannot afford to lose, set disconnect on all three. DRBD then stays StandAlone after any split brain and waits for an operator.

Manual split-brain recovery

Recovery always throws away one side’s writes. There is no merge. Your job is to decide which side loses, then tell DRBD explicitly.

Confirm the state first on both nodes:

sudo drbdadm status myapp
sudo drbdadm cstate myapp

StandAlone on both sides, with the kernel log reporting Split-Brain detected, is the signature.

Pick the victim - the node whose divergent writes you are discarding. If the two sets of writes are both valuable, take a block-level copy of the victim’s backing device before going further, because the next command is not reversible.

On the victim:

sudo drbdadm disconnect myapp
sudo drbdadm secondary myapp
sudo drbdadm connect --discard-my-data myapp

On the survivor (only if it is also StandAlone):

sudo drbdadm connect myapp

Then watch the resync run to completion before anyone uses the device:

sudo drbdadm status myapp

You are waiting for UpToDate/UpToDate and a connection state of Connected. The victim’s divergent writes are gone.

Test split-brain on a disposable pair

On node1, the current Primary:

sudo drbdadm primary scratch
sudo mount /dev/drbd0 /mnt/scratch

Break the replication link. Scope the rules to the peer address so the rest of the host keeps working:

# node1
sudo iptables -I INPUT  -s 10.0.0.2 -p tcp --dport 7788 -j DROP
sudo iptables -I OUTPUT -d 10.0.0.2 -p tcp --dport 7788 -j DROP

Now make both sides diverge. On node2, force a promotion and write:

sudo drbdadm primary --force scratch
sudo mount /dev/drbd0 /mnt/scratch
sudo dd if=/dev/urandom of=/mnt/scratch/node2.bin bs=1M count=8 conv=fsync

And write something different on node1:

sudo dd if=/dev/urandom of=/mnt/scratch/node1.bin bs=1M count=8 conv=fsync

Unmount and demote node2, then restore the link:

# node2
sudo umount /mnt/scratch
sudo drbdadm secondary scratch

# node1
sudo iptables -D INPUT  -s 10.0.0.2 -p tcp --dport 7788 -j DROP
sudo iptables -D OUTPUT -d 10.0.0.2 -p tcp --dport 7788 -j DROP

Watch the split brain appear on reconnect, not before:

sudo drbdadm status scratch
sudo journalctl -k | grep -i split-brain

Recover with the manual procedure above, choosing node2 as the victim.

Cleanup

# node1 and node2: remove any leftover filter rules
sudo iptables -D INPUT  -s 10.0.0.2 -p tcp --dport 7788 -j DROP 2>/dev/null
sudo iptables -D OUTPUT -d 10.0.0.2 -p tcp --dport 7788 -j DROP 2>/dev/null
sudo iptables -S | grep 7788   # expect no output

# node2: confirm it is Secondary and unmounted
sudo umount /mnt/scratch 2>/dev/null
sudo drbdadm secondary scratch

# both: confirm the pair is healthy again
sudo drbdadm status scratch   # expect Connected, UpToDate/UpToDate

Remove the test files from the surviving copy once the resync has finished.

Knowledge check