DRBD resync and online verification - reading the replication state

A DRBD pair that reports Connected is not necessarily a pair that holds the same bytes. Replication can be established while one side is still filling in, and it can be established while both sides silently disagree about a range of blocks. This lesson is about reading the state precisely enough to tell the difference, and about the one command that proves the copies match.

The three state axes

DRBD 9 reports three independent things per resource. Operators who collapse them into “is it up” miss the case that matters.

• Role - Primary or Secondary. Who is allowed to write.
• Disk state - what the local backing device holds.
• Connection state and, per peer, a replication state - whether the link is up and what it is currently doing.

drbdadm status prints all of them:

$ sudo drbdadm status myapp

myapp role:Primary
disk:UpToDate
node2 role:Secondary
  peer-disk:UpToDate

During a resync the peer section grows a replication state and a progress figure:

myapp role:Primary
  disk:UpToDate
  node2 role:Secondary
    replication:SyncSource peer-disk:Inconsistent done:34.28

Disk states, in the order they matter

• UpToDate - consistent, and DRBD knows it is current. This is the only state you should accept for a copy you plan to fail over to.
• Consistent - the data is a valid filesystem image, but DRBD has not yet compared generation UUIDs with the peer, so it does not know whether it is current. Seen after a restart before the connection comes up.
• Outdated - consistent, and known to be stale. This is the state a fencing handler sets on a peer that was cut off, and it is deliberately not promotable.
• Inconsistent - not a usable image at all. A sync target is Inconsistent for the whole duration of the resync, because it holds a mixture of old and new blocks.
• Diskless - the backing device is gone or was detached. The node can still serve I/O from the peer over the network.
• DUnknown - the peer’s disk state, when there is no connection to ask. Not a fault by itself.

Connection and replication states

Connection state covers the link: Connected, Connecting, Unconnected, StandAlone, Disconnecting, plus failure states such as NetworkFailure, BrokenPipe and ProtocolError.

Replication state covers what the link is doing for a given volume: Established (steady state), SyncSource / SyncTarget (a resync is running), VerifyS / VerifyT (an online verification is running), PausedSyncS / PausedSyncT, and Ahead / Behind when protocol A has let the secondary fall behind.

StandAlone is the state to alert on. It means DRBD has deliberately stopped replicating - after a split brain, or after an operator disconnect - and every write since then exists on one node only.

sudo drbdadm cstate myapp
sudo drbdadm dstate myapp
sudo drbdadm role myapp

For the full picture, including per-peer byte counters:

sudo drbdadm status --verbose --statistics myapp

The counter to read there is out-of-sync. It is the number of kibibytes DRBD knows differ between the two copies. In a healthy steady state it is 0.

sudo drbdsetup events2 --now myapp

Tuning the resync, correctly

A resync copies only the blocks the activity log and the bitmap say are stale, but on a large device after a long outage that is still a lot of data. Two knobs look like the throttle. Only one of them is connected.

DRBD 8.4 introduced a dynamic resync controller and it is on by default in DRBD 9. While it is on, the static resync-rate setting is ignored. The controller lives in the disk section:

resource myapp {
    disk {
        c-plan-ahead 20;
        c-min-rate   10M;
        c-max-rate   200M;
        c-fill-target 10M;
    }
}

• c-plan-ahead enables the controller. Setting it to 0 disables it and hands control back to resync-rate.
• c-max-rate is the ceiling the controller will not exceed.
• c-min-rate is the throughput below which DRBD stops throttling itself out of the way of application I/O. Set it to 0 to tell DRBD never to throttle the resync for the sake of application writes.
• c-fill-target is how much data the controller tries to keep in flight on the link.

Change it at runtime without editing the file:

sudo drbdadm disk-options --c-max-rate=200M myapp
sudo drbdadm dump myapp

There is a real trade-off underneath this. A resync running at line rate on a shared link is exactly the storage burst that delays Corosync tokens and gets a healthy node fenced. Cap c-max-rate below the point where the replication link starves the heartbeat, or give replication its own path.

Online verification: proving the copies match

Replication guarantees that DRBD sent every write to the peer. It does not guarantee the peer’s disk still holds it. Silent corruption on the secondary - a bad sector, a firmware bug, a controller that lied about a flush - is invisible until the day you fail over onto it.

Online verification walks both devices, compares block digests, and marks any difference. It needs a digest algorithm, which is not configured by default:

resource myapp {
    net {
        verify-alg crc32c;
    }
}

Apply it and start a verification from the node you want to be the verification source:

sudo drbdadm adjust myapp
sudo drbdadm verify myapp

The replication state becomes VerifyS on that node and VerifyT on the peer. Progress appears in drbdadm status, and the result is logged by the kernel:

sudo drbdadm status myapp
sudo journalctl -k | grep -i 'Online verify'

Schedule verification off-peak - it reads both devices end to end - and alert on the out-of-sync counter rather than on the log line.

The two invalidate commands

When you already know which copy is right, you can force a full resync in either direction. Both commands throw data away, and which node you run them on decides whose data dies.

• drbdadm invalidate myapp marks the local disk Inconsistent and pulls a full copy from the peer. The local data is discarded.
• drbdadm invalidate-remote myapp marks the peer’s disk Inconsistent and pushes a full copy to it. The peer’s data is discarded.

A verification routine that holds up

# 1. Confirm a healthy starting point on both nodes.
sudo drbdadm status myapp

# 2. Confirm a digest algorithm is actually loaded.
sudo drbdadm dump myapp | grep -i verify-alg

# 3. Start verification from the Primary, off-peak.
sudo drbdadm verify myapp

# 4. Watch it finish. Expect the replication state to return
#    to Established.
sudo drbdadm status myapp

# 5. Read the result, not the connection state.
sudo drbdadm status --verbose --statistics myapp | grep out-of-sync

A non-zero out-of-sync after step 5 is a repair job, and a question about why the two devices diverged in the first place.

Knowledge check