Recovery and rebalancing - what a distributed store does after a failure

A distributed store repairs itself. That is the whole promise, and it is the reason these systems tolerate host loss without a cluster manager promoting anything. What the promise does not say is that the repair competes with your application for the same disks and the same network, and that it will win unless you tell it not to.

This lesson is about reading the repair, controlling it, and the one case it cannot resolve on its own.

Degraded and misplaced are not the same problem

Ceph reports two conditions that both look like “not clean” and mean completely different things.

• Degraded: fewer copies of some objects exist than size asks for. This is a durability problem. Another failure in the wrong place loses data.
• Misplaced (also shown as remapped): the right number of copies exist, but not where the CRUSH rule says they belong. This is a tidiness problem. Nothing is at risk; data is being moved.

A topology change - adding a host, editing a rule, reweighting an OSD - produces misplaced objects and a lot of I/O. A failed disk produces degraded objects. Only one of those should keep you at your desk.

$ sudo ceph -s

  cluster:
  health: HEALTH_WARN
          Degraded data redundancy: 12841/385230 objects degraded (3.333%)

services:
  mon: 3 daemons, quorum ceph1,ceph2,ceph3
  osd: 6 osds: 5 up (since 4m), 6 in (since 3w)

data:
  pools:   2 pools, 129 pgs
  objects: 128.41k objects, 489 GiB
  pgs:     107 active+clean
           22  active+undersized+degraded

The other states worth recognising: undersized (a PG has fewer OSDs than it should), peering (the OSDs are agreeing on history - brief, and a problem if it persists), incomplete (not enough history to proceed, which needs a human), and stale (no report from the primary OSD).

sudo ceph health detail
sudo ceph pg stat
sudo ceph pg dump_stuck

Down is not out

An OSD that stops answering is marked down. It is not immediately marked out, and only out changes data placement.

The delay is mon_osd_down_out_interval, 600 seconds by default. The reasoning is sound: a host rebooting for a kernel update comes back in three minutes, and rebalancing 8 TB of data because of a three-minute absence would be far more damaging than the absence.

sudo ceph config get mon mon_osd_down_out_interval

Everything in that ten-minute window is degraded but not recovering. Cross the boundary and the cluster starts manufacturing new copies, which is when the disks get busy.

Planned maintenance without a rebalance

# Confirm the cluster is healthy before touching anything.
sudo ceph -s

# Ask whether stopping this OSD is safe right now.
sudo ceph osd ok-to-stop osd.3

# Stop the cluster from marking absent OSDs out.
sudo ceph osd set noout

# ... do the maintenance on the host ...

# Bring the OSDs back and confirm they are up and in.
sudo ceph osd tree

# Last line of the procedure, always.
sudo ceph osd unset noout
sudo ceph -s

ceph osd ok-to-stop is the check worth building the habit around: it answers whether stopping that OSD right now would take a PG below min_size, which is the question the operator is actually asking.

Related flags, for larger interventions:

• norebalance - stop moving misplaced data, still repair degraded data. The right flag for adding capacity during business hours.
• nobackfill and norecover - stop the corresponding work entirely. Blunt; use them to buy time during an incident, not as a configuration.
• noscrub and nodeep-scrub - stop consistency scrubbing. Useful during a recovery storm, and dangerous to leave set, because scrubbing is what finds silent corruption.

Every one of these is cluster-wide and every one of them is easy to forget. ceph -s prints set flags in the health section; treat any flag that has been set for more than a day as an incident of its own.

Throttling recovery on a modern cluster

Recovery I/O and client I/O share the same OSDs. The classic throttles were osd_max_backfills and osd_recovery_max_active, and every runbook written before 2022 tunes them.

Since the Quincy release the default OSD scheduler is mClock, and mClock allocates I/O by profile rather than by those counters. It overrides them.

The classic settings still apply on clusters using the WPQ scheduler, and on mClock clusters only if the override is explicitly enabled - which the Ceph documentation discourages.

Gluster: self-heal and the case it cannot heal

Gluster’s equivalent of recovery is the self-heal daemon, glustershd. When a brick returns after an outage, entries that changed while it was away are marked and healed in the background.

sudo gluster volume heal gv0 info summary
sudo gluster volume heal gv0 info
sudo gluster volume heal gv0
sudo gluster volume heal gv0 statistics heal-count

A steadily shrinking heal count is a healthy heal. A count that does not move is a stuck one, and the first thing to check is whether glustershd is running on every node:

sudo gluster volume status gv0

The case self-heal cannot resolve is a file-level split brain: the same file was modified on two bricks while they could not see each other, so neither version is an ancestor of the other. This is exactly the DRBD situation from the previous part, one layer up, and it has exactly the same resolution - somebody names a survivor and the other version is discarded.

sudo gluster volume heal gv0 info split-brain

Resolve per file, or for the whole volume from one brick:

# Keep the copy with the most recent modification time.
sudo gluster volume heal gv0 split-brain latest-mtime /path/to/file

# Keep the larger copy.
sudo gluster volume heal gv0 split-brain bigger-file /path/to/file

# Keep this brick's copy of one file.
sudo gluster volume heal gv0 split-brain source-brick server1:/data/brick1 /path/to/file

# Keep this brick's copy of every split-brain file in the volume.
sudo gluster volume heal gv0 split-brain source-brick server1:/data/brick1

The structural fix is an arbiter: replica 3 arbiter 1 keeps two full copies and one metadata-only copy, and the arbiter breaks the tie so file-level split brain cannot form in the first place. It costs almost no space and is the default recommendation for three-node Gluster deployments.

The operational habits that matter

• Alert on degraded, not on HEALTH_WARN. A cluster rebalancing after a planned change is warned and fine; a cluster with degraded objects is not.
• Know your recovery time. How long does the cluster take to rebuild one full OSD? That figure is your real exposure window after a disk failure, and it is measurable on the lab cluster.
• Unset flags in the same change that set them. A noout from last month is a cluster that has silently stopped self-healing.
• Never leave scrubbing off. It is the only mechanism that finds corruption nobody has read yet.

Knowledge check