Blocked by insufficient available copies

Why this matters in production

A PG below min_size blocks I/O to the data it holds. It is the most common cause of a genuine Ceph outage and the response involves a durability decision.

Recognising it

ceph -s
ceph health detail | grep -E 'PG_AVAILABILITY|PG_DEGRADED'
ceph pg dump pgs | awk '$10 !~ /active/ {print $1, $10, $15}' | head

pg 3.1f is stuck inactive+undersized+degraded, acting [12,2147483647,2147483647]

The 2147483647 entries are the NONE sentinel: those positions in the acting set have no OSD. One surviving copy on a min_size=2 pool means the PG cannot serve I/O.

ceph osd pool get rbd-vms size
ceph osd pool get rbd-vms min_size

Establishing the scope

# which PGs?
ceph pg dump pgs | awk '$10 ~ /inactive|down|incomplete/ {print $1}' | wc -l

# which pools?
ceph pg dump pgs | awk '$10 ~ /inactive|down|incomplete/ {split($1,a,"."); print a[1]}' | \
  sort -u | while read id; do
    ceph osd pool ls detail | grep "^pool $id " | awk '{print $3}'
  done

# which OSDs would restore them?
ceph pg 3.1f query | python3 -c '
import sys,json; d=json.load(sys.stdin)
print("up:", d.get("up"), "acting:", d.get("acting"))
for s in d.get("recovery_state", [])[:1]:
    print(s.get("name"), s.get("blocked_by"))'

blocked_by names the OSDs the PG is waiting for, which is the shortest path to restoring it.

The options

# preferred: bring the OSDs back
FSID=$(ceph fsid)
systemctl start "ceph-$FSID@osd.31"
ceph osd in 31

Lowering min_size

ceph osd pool set rbd-vms min_size 1
# ... I/O resumes ...
# and afterwards, without fail:
ceph osd pool set rbd-vms min_size 2

The trade is explicit: I/O resumes immediately, and the surviving copy has no redundancy — a read error on it loses that data with nothing to recover from, and a scrub inconsistency has no majority.

Record before doing it:
  pool, previous min_size, time changed, who authorised, restoration condition

Marking OSDs lost

ceph osd lost 31 --yes-i-really-mean-it

This tells the cluster the OSD’s data is gone permanently, allowing PGs to peer without it. Any writes that existed only on that OSD are lost. It is a last resort and is not reversible.

Quiz

Production discipline

Restore a missing OSD rather than lowering min_size where the OSD can be recovered in an acceptable time — it resumes I/O with no durability trade. When lowering is necessary, record the pool, the previous value, the authorisation, and the restoration condition before making the change.

Cross-course references

• Kubernetes: refusing to serve from a quorum-less replica set follows the same reasoning
• Linux: a RAID array refusing to assemble degraded beyond tolerance is the analogue