DRBD in a cluster - integration with Pacemaker for HA

DRBD provides replicated block storage; Pacemaker provides cluster management. Together they form a complete HA solution for two-node clusters. This lesson covers the integration.

The architecture

Host A (often primary):
  /dev/drbd0 (DRBD primary)
  /var/lib/myapp (filesystem)
  Application

Host B (often secondary):
  /dev/drbd0 (DRBD secondary)

Pacemaker:
  Resource: myapp-drbd-clone (promotable, one instance per node)
  Resource: myapp-fs
  Constraint: fs colocated with the Promoted DRBD instance
  STONITH: IPMI per host

The application depends on the filesystem, which depends on DRBD. Pacemaker manages the promotion and constraints.

Fence DRBD, not only the nodes

Fencing has two halves and both are mandatory. Pacemaker fences nodes with STONITH. DRBD fences its own replication link, and that half is configured in the DRBD resource file.

Add it to /etc/drbd.d/myapp.res before you touch Pacemaker:

resource myapp {
    net {
        protocol C;
        fencing resource-and-stonith;
    }
    handlers {
        fence-peer   "/usr/lib/drbd/crm-fence-peer.9.sh";
        unfence-peer "/usr/lib/drbd/crm-unfence-peer.9.sh";
    }
    # on host-a { ... } and on host-b { ... } as before
}

Apply and confirm the parameter is really in effect:

# On both nodes
sudo drbdadm adjust myapp
sudo drbdadm dump myapp | grep -i fencing

What each piece does:

• fencing resource-and-stonith makes a Primary that loses its peer freeze all I/O and call the fence-peer handler, instead of carrying on alone.
• crm-fence-peer.9.sh asks Pacemaker to outdate or fence the peer and writes a location constraint that forbids promotion there.
• crm-unfence-peer.9.sh removes that constraint once the resync has finished.

The default policy is dont-care: no fencing at all. With the default, a disconnected Primary keeps writing while Pacemaker promotes the other node. Both nodes then hold Primary over an ext4 filesystem, and the volume is destroyed by the first pair of concurrent writes. This is the single most common way a DRBD cluster loses data.

Configure DRBD in Pacemaker

STONITH first. The fence-peer handler has nothing to call if node fencing is not configured and enabled:

pcs property set stonith-enabled=true
pcs stonith status

DRBD must be a promotable clone: one instance running on each node, exactly one of them Promoted. A plain primitive never reaches Primary, so /dev/drbd0 is never writable.

# Create the DRBD resource as a promotable clone
pcs resource create myapp-drbd ocf:linbit:drbd \
    drbd_resource=myapp \
    drbd_conf=/etc/drbd.d/myapp.res \
    op monitor interval=20s role=Unpromoted \
    op monitor interval=10s role=Promoted \
    promotable promoted-max=1 promoted-node-max=1 \
        clone-max=2 clone-node-max=1 notify=true

# Prefer promotion on host-a, but do not force it
pcs constraint location myapp-drbd-clone rule role=Promoted \
    score=100 '#uname' eq host-a

Three details that bite:

• pcs names the clone myapp-drbd-clone. Every constraint below refers to the clone id, not to myapp-drbd.
• The two monitor operations need different intervals. Pacemaker rejects two monitors with the same interval on one resource.
• Use a finite promotion score, not INFINITY. INFINITY drags the Promoted role back to host-a the moment it returns, possibly mid-resync.
• pcs 0.12 flags this command form as legacy. With pcs resource create --future the clone options are written as promotable meta promoted-max=1 ... instead.

Quote '#uname' in the rule. Unquoted, the shell treats the rest of the line as a comment and the constraint is created with no expression.

Configure the filesystem on top

# Create the filesystem
pcs resource create myapp-fs ocf:heartbeat:Filesystem \
    device=/dev/drbd0 \
    directory=/var/lib/myapp \
    fstype=ext4 \
    op monitor interval=30s

# Colocation: fs on the node holding the Promoted DRBD instance
pcs constraint colocation add myapp-fs with Promoted myapp-drbd-clone INFINITY

# Order: promote DRBD, then start the filesystem
pcs constraint order promote myapp-drbd-clone then start myapp-fs

Both keywords go in front of the resource they qualify. A role written after the target (with-rsc-role=Master) and an action written after the resource (then myapp-fs promote) are rejected by current pcs, and Master no longer exists as a role name. It is Promoted and Unpromoted.

Configure the application

# Application resource (e.g. systemd)
pcs resource create myapp systemd:myapp \
    op monitor interval=30s

# Same node as FS
pcs constraint colocation add myapp with myapp-fs INFINITY

# After FS
pcs constraint order myapp-fs then myapp

Test failover

Standby is the controlled test: Pacemaker moves everything off the node and you watch it land. Maintenance mode is the opposite, and is for editing a running cluster without it reacting, so do not use it here.

# Baseline before you touch anything
sudo pcs status --full

# Move resources off host-a
sudo pcs node standby host-a

# Verify placement
sudo pcs status --full

# Give it back
sudo pcs node unstandby host-a

Verify on host-b:

sudo drbdadm status myapp    # host-b Primary, host-a Secondary
findmnt /var/lib/myapp       # mounted on host-b only
systemctl is-active myapp

Then prove you have left nothing behind. A test that ends with a node still drained is a cluster with one node of capacity and nobody aware of it:

sudo pcs status nodes                    # no node Standby, none Maintenance
sudo pcs property config maintenance-mode  # must report false or be unset

Test STONITH

This hard-resets host-a with no shutdown. Run it on a test pair, or during an agreed window with the service already failed over.

# Fence host-a
pcs stonith fence host-a

# Verify the cluster continues on host-b
pcs status

STONITH is what makes promotion safe. Without it, the DRBD fence-peer handler cannot confirm the peer is down, I/O stays frozen, and the service does not come back.

After the fenced node returns and the resync completes, check that the handler removed its own constraint:

pcs constraint location config --full | grep drbd-fence-by-handler

Empty output is the expected result. A constraint left behind pins the Promoted role to one node and the next failover will not happen. That is the unfence handler telling you it was never configured, or never ran.

Production discipline

For every cluster:

• STONITH for every node, and stonith-enabled=true.
• fencing resource-and-stonith plus the fence-peer and unfence-peer handlers in the DRBD resource.
• DRBD configured for sync replication (protocol C).
• DRBD as a promotable clone, with the filesystem colocated with the Promoted instance.
• Test failover quarterly.
• Document the procedure.

Knowledge check