Failure domains in distributed storage - where the replicas actually land

Distributed storage keeps several copies of every object. That only buys you anything if the copies fail independently. Three replicas on three disks in the same chassis survive three disk failures and no power supply failures at all.

A failure domain is a set of components that fail together because they share infrastructure: a chassis, a rack with one pair of top-of-rack switches and one PDU pair, a room, a site. The placement rule’s job is to put each copy in a different one. The rest of this lesson is about telling the storage system what your failure domains really are, and checking that it believed you.

The hierarchy Ceph places into

CRUSH describes the cluster as a tree of buckets. The leaves are OSDs; the interior buckets are host, chassis, rack, row, pdu, room, datacenter and root, among others.

$ sudo ceph osd tree

ID  CLASS  WEIGHT    TYPE NAME       STATUS  REWEIGHT  PRI-AFF
-1        43.65916  root default
-3        14.55305      host ceph1
0    ssd  7.27652          osd.0      up   1.00000  1.00000
1    ssd  7.27652          osd.1      up   1.00000  1.00000
-5        14.55305      host ceph2
2    ssd  7.27652          osd.2      up   1.00000  1.00000
3    ssd  7.27652          osd.3      up   1.00000  1.00000
-7        14.55305      host ceph3
4    ssd  7.27652          osd.4      up   1.00000  1.00000
5    ssd  7.27652          osd.5      up   1.00000  1.00000

That tree is the whole of Ceph’s knowledge about your physical layout. It is not discovered - it is asserted. If all three hosts are in one rack, CRUSH does not know and cannot warn you.

A replicated rule names the bucket type it spreads across:

sudo ceph osd crush rule ls
sudo ceph osd crush rule dump replicated_rule

The line that decides everything is the chooseleaf step, which in the default rule reads chooseleaf_firstn 0 type host: pick enough leaves, never reusing a host.

Making CRUSH aware of racks

Buckets have to be created and hosts moved into them. This changes data placement, so it triggers a rebalance - do it during a window, and read the recovery lesson first.

# Create the rack buckets and attach them to the root.
sudo ceph osd crush add-bucket rack1 rack
sudo ceph osd crush add-bucket rack2 rack
sudo ceph osd crush add-bucket rack3 rack
sudo ceph osd crush move rack1 root=default
sudo ceph osd crush move rack2 root=default
sudo ceph osd crush move rack3 root=default

# Move each host into the rack it is physically in.
sudo ceph osd crush move ceph1 rack=rack1
sudo ceph osd crush move ceph2 rack=rack2
sudo ceph osd crush move ceph3 rack=rack3

# A rule that spreads replicas across racks.
sudo ceph osd crush rule create-replicated rack_rule default rack

# Apply it to the pool.
sudo ceph osd pool set mypool crush_rule rack_rule

Confirm the tree looks like the datacentre:

sudo ceph osd tree
sudo ceph osd crush rule dump rack_rule

size and min_size decide two different things

• size is how many copies the pool wants. It sets durability: how many independent failures the data survives.
• min_size is how many copies must be available for the pool to accept I/O. It sets availability, and it is the guardrail against writing to a copy that is about to be the only one.

sudo ceph osd pool get mypool size
sudo ceph osd pool get mypool min_size
sudo ceph osd pool set mypool min_size 2

With size 3, min_size 2: lose one failure domain and I/O continues on two copies while Ceph makes a third. Lose two and I/O stops - deliberately, because continuing would mean every new write exists once.

Monitors are their own failure domain problem

Ceph’s data survives host loss because of CRUSH. The cluster survives host loss because of the monitors, and they have a separate rule: a strict majority must be reachable, or the cluster stops serving even though every byte is intact.

• Three monitors tolerate one loss. Five tolerate two.
• Spread them across the same failure domains as the data. Three monitors in one rack means a rack failure stops a cluster whose data survived perfectly.

sudo ceph mon stat
sudo ceph quorum_status --format json-pretty

Gluster: the failure domain is brick order

Gluster has no CRUSH map. Replica sets are formed by the order bricks are listed on the command line: for replica 3, the first three bricks are one replica set, the next three are the second, and so on.

sudo gluster volume create gv0 replica 3 \
  server1:/data/brick1 \
  server2:/data/brick1 \
  server3:/data/brick1

That places one copy per server. Reorder the list so two bricks from the same server land in one set, and both copies of that data live on one machine.

Gluster does check for the obvious case and refuses, telling you that multiple bricks of a replicate volume are on the same server - and then offers force. The refusal is the feature; force is how it gets defeated, usually by somebody who read that the command “needs” it.

Read back what you built:

sudo gluster volume info gv0

The brick list in that output is in replica-set order. Count in threes and check each group spans three servers.

What to verify, and how often

• The tree matches the building. Walk the racks, or read the DCIM export, and compare against ceph osd tree. A host moved between racks and not moved in CRUSH is a silent loss of a failure domain.
• Every pool uses the rule you think it does. ceph osd pool ls detail.
• min_size is 2 everywhere it should be, including after an incident.
• The monitors are spread, and there is an odd number of them.
• Power and network match the labels. Two “racks” fed from one PDU pair and one pair of switches are one failure domain wearing two names. CRUSH will happily believe the labels.

Knowledge check