DRBD architecture - the distributed replicated block device

DRBD (Distributed Replicated Block Device) replicates block devices between two hosts. It is the standard Linux solution for block-level replication and is often used with Pacemaker for HA.

What DRBD does

DRBD mirrors a block device across two hosts:

• Primary: the writable node.
• Secondary: the replica, not writable.
• Synchronous or asynchronous replication.

The secondary has a current copy of the primary’s data. On primary failure, the secondary can be promoted.

Architecture

Host A (primary):
  /dev/sda1
  DRBD layer: /dev/drbd0
  Application: writes to /dev/drbd0

Host B (secondary):
  DRBD layer: /dev/drbd0
  Disk: receives writes from A

The application writes to /dev/drbd0 on the primary. DRBD replicates the writes to the secondary.

Configure DRBD

# Install (RHEL)
sudo yum install drbd kmod-drbd90

# Configure
sudo tee /etc/drbd.d/global_common.conf <<EOF
global {
    usage-count no;
}
common {
    net {
        protocol C;    # sync replication
    }
}
EOF

# Resource config
sudo tee /etc/drbd.d/myapp.res <<EOF
resource myapp {
    on host-a {
        device /dev/drbd0;
        disk /dev/sda1;
        meta-disk internal;
        address 10.0.0.10:7788;
    }
    on host-b {
        device /dev/drbd0;
        disk /dev/sda1;
        meta-disk internal;
        address 10.0.0.11:7788;
    }
}
EOF

# Initialise
sudo drbdadm create-md myapp
sudo drbdadm up myapp

# On host-b, the same
sudo drbdadm create-md myapp
sudo drbdadm up myapp

# Promote host-b as secondary
sudo drbdadm secondary myapp

# On host-a, make primary
sudo drbdadm primary myapp --force

The DRBD device /dev/drbd0 is now active on host-a.

Use with Pacemaker

DRBD is often used with Pacemaker for HA. DRBD is added as a promotable clone: one instance per node, one of them Promoted. A plain primitive has no Promoted role, so DRBD would stay Secondary on both nodes and /dev/drbd0 would never be writable.

# Fencing first. A DRBD cluster without STONITH will corrupt itself.
pcs property set stonith-enabled=true

# Create the DRBD resource as a promotable clone
pcs resource create myapp-drbd ocf:linbit:drbd \
    drbd_resource=myapp \
    op monitor interval=20s role=Unpromoted \
    op monitor interval=10s role=Promoted \
    promotable promoted-max=1 promoted-node-max=1 \
        clone-max=2 clone-node-max=1 notify=true

# Create a filesystem on top
pcs resource create myapp-fs ocf:heartbeat:Filesystem \
    device=/dev/drbd0 \
    directory=/var/lib/myapp \
    fstype=ext4

# Colocation: FS on the node holding the Promoted instance
pcs constraint colocation add myapp-fs with Promoted myapp-drbd-clone INFINITY

# Order: promote DRBD, then start the FS
pcs constraint order promote myapp-drbd-clone then start myapp-fs

pcs names the clone myapp-drbd-clone, and the constraints refer to that id.

pcs also needs the resource file to hand fencing decisions to the cluster. Add this to /etc/drbd.d/myapp.res once Pacemaker manages the resource — not before, because with no cluster to call, a disconnected Primary would freeze I/O with nothing able to release it:

resource myapp {
  net {
    fencing resource-and-stonith;
  }
  handlers {
    fence-peer          "/usr/lib/drbd/crm-fence-peer.9.sh";
    after-resync-target "/usr/lib/drbd/crm-unfence-peer.9.sh";
  }
}

fencing resource-and-stonith makes a Primary that loses its peer suspend I/O and call the fence-peer handler, which asks Pacemaker to fence the other node before anything is promoted. The default is dont-care, which takes no action at all — and that is how a partition ends with two Primaries and a corrupted ext4 filesystem.

On host-a failure, Pacemaker demotes DRBD on host-a, promotes on host-b, starts the filesystem, and the application continues. That is only safe with both halves in place: STONITH enabled in the cluster, and DRBD-side fencing in the resource file. The DRBD in a cluster lesson works through both in full.

DRBD modes

• Protocol A: async. Write acked before replication. Fastest, lowest consistency.
• Protocol B: semi-sync. Write acked after primary, before secondary.
• Protocol C: sync. Write acked after both confirm. Most consistent, slowest.

For most production, protocol C (sync) is the right choice for HA. The latency cost is acceptable.

When to use DRBD

• Block-level replication between two hosts.
• HA database (single writer, replicated to standby).
• HA shared storage (with Pacemaker and a clustered FS).

Not for:

• Very wide replication. DRBD 9 supports up to 32 nodes accessing one resource, but every additional synchronous replica adds write latency and LINBIT does not recommend more than about five. Beyond a handful, use a distributed store such as Ceph.
• High write throughput (sync replication is slow).
• Object storage or POSIX FS without cluster FS.

Knowledge check