What a replicated copy is consistent with

Advanced⏱ ~18 minrbdradosgw-admin

What you'll learn

Classify the consistency each mechanism delivers
Identify where a replica is safe to mount and where it is not
Recognise the multi-image consistency gap
Order writes so a partial replica remains usable

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

A replica that is behind is usable; a replica that is inconsistent is not, and the two are indistinguishable from a status command.

The four consistency levels

Level	Meaning	Recoverable by
Application-consistent	the application flushed and quiesced first	nothing; just start it
Crash-consistent	as if power was cut at one instant	journal replay or fsck
Snapshot-boundary consistent	crash-consistent only at completed snapshots	rolling back to the snapshot
Eventually consistent	each item arrives independently, unordered	nothing — there is no instant to roll to

What each mechanism delivers

Mechanism	Level	The instant it represents
RBD journal-based	crash-consistent, continuously	primary state minus the unreplayed entries
RBD snapshot-based	snapshot-boundary	the last completed mirror snapshot
CephFS snapshot mirroring	snapshot-boundary, per directory tree	the last snapshot that finished syncing
RGW multi-site	eventual, per object	no single instant across objects

# journal mode replays in order, so the replica is always crash-consistent
rbd mirror image status rbd/vm-101-disk-0 | grep -E 'state|description'

# snapshot mode is consistent only at a completed mirror snapshot
rbd mirror image snapshot rbd/vm-204-disk-0
rbd snap ls rbd/vm-204-disk-0 --all

Promotion in snapshot mode rolls the non-primary image to the last
complete mirror snapshot. Data written after it is discarded, which is
what makes the promoted image mountable rather than half-written.

The multi-image gap

# a group gives one consistent instant across several images, locally
rbd group create rbd/db-cluster
rbd group image add rbd/db-cluster rbd/db-data
rbd group image add rbd/db-cluster rbd/db-log
rbd group snap create rbd/db-cluster@pre-change
rbd group snap ls rbd/db-cluster

Mirroring is configured per image and each image takes its own mirror
snapshots on its own schedule. Two images mirrored from the same cluster
can therefore be promoted to two different instants — a database whose
data and log volumes are separate images has no guaranteed common
recovery point at the peer.

Layout	Common recovery point at the peer
One image per workload	yes, by construction
Data and log on separate images	no
LVM spanning several mirrored images	no — and the result is unmountable
Filesystem in RGW plus disks in RBD	no

Ordering writes so a partial replica is usable

Write the referenced object before the reference. An index entry
pointing at missing content is a detectable error; content with no index
entry is merely orphaned.

radosgw-admin bucket sync status --bucket=acme-index
radosgw-admin bucket sync status --bucket=acme-content

# verify at the peer before trusting it: promote a clone, not the image
rbd snap create rbd/vm-204-disk-0@verify
rbd clone rbd/vm-204-disk-0@verify rbd/vm-204-verify

Quiz

Knowledge check · 4 questions

Q1. Why is a promoted snapshot-based mirror image mountable rather than half-written?

The rbd-mirror daemon flushes the in-flight delta to the peer before it allows the promotion to proceed Promotion rolls the image to the last complete mirror snapshot and discards anything written after it Snapshot mode replicates synchronously, so the peer image is never behind the primary by more than one write The image is quiesced at the primary before each snapshot, so every mirror snapshot is application-consistent
Q2. Two RBD images mirrored from the same cluster can be promoted to two different points in time.

True False
Q3. Assess whether a mirrored database can be brought up at the peer.

A database VM has its data files on rbd/db-data and its write-ahead log on rbd/db-log. Both are mirrored in snapshot mode on a 15-minute schedule. The primary site has been lost.
Q4. Why does an eventually consistent object replica have no point-in-time to roll back to?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Record what instant each replicated unit represents before promoting anything — a status of healthy says nothing about whether two images share a recovery point. Where a workload spans volumes, put it on one image or accept that the peer copy requires application-level repair, and decide which before the event rather than during it.

Cross-course references

Kubernetes: a restored PVC and a restored ConfigMap are not from the same instant either
Linux: crash consistency is a property of an ordered write stream, not of a healthy link