Multiple rules on one cluster

Intermediate⏱ ~15 minceph

What you'll learn

Design a coherent rule set for a mixed-workload cluster
Understand how pools sharing rules share OSDs
Name and document rules so intent survives
Audit a cluster rule set for drift

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

A real cluster serves several workloads with different requirements. Rules are how those differences are expressed, and a rule set that grew by accretion is one of the harder things to reason about during an incident.

A coherent rule set

For a cluster with NVMe, SSD, and HDD across four racks:

ceph osd crush rule create-replicated nvme_rack default rack nvme
ceph osd crush rule create-replicated ssd_rack  default rack ssd
ceph osd crush rule create-replicated hdd_rack  default rack hdd
ceph osd erasure-code-profile set ec42_rack \
    k=4 m=2 crush-failure-domain=rack

Then assignment expresses policy:

Pool	Rule	Why
`cephfs_metadata`	`nvme_rack`	latency critical, small
`rgw_index`	`nvme_rack`	omap heavy, latency critical
`rbd_vms`	`ssd_rack`	mixed workload, capacity matters
`rgw_data`	`ec42_rack`	large objects, capacity dominant
`backups`	`hdd_rack`	sequential, cheap

Five pools, four rules, and each assignment has a stated reason.

Pools sharing a rule share OSDs and therefore share performance

rbd_vms and a second pool on ssd_rack sit on the same OSDs. A heavy job on one consumes device IOPS the other needs, regardless of their separate quotas and sizes.

Quotas bound capacity. Nothing about separate pools bounds performance.

Where two workloads genuinely must not interfere, they need disjoint OSDs:

ceph osd crush rule create-replicated ssd_tenant_a rack-a host ssd
ceph osd crush rule create-replicated ssd_tenant_b rack-b host ssd

which trades isolation for capacity ceilings and reduced failure tolerance per tenant. That trade should be made explicitly, because both halves are real.

Naming

A rule name should state root, failure domain, and class:

nvme_rack       class nvme, rack failure domain, default root
hdd_host        class hdd, host failure domain
ec42_rack       EC 4+2, rack failure domain
racka_ssd_host  restricted to rack-a, ssd class, host domain

Names like rule-1, replicated_rule, and ssd carry no information and are what an inherited cluster typically has.

Auditing

for p in $(ceph osd pool ls); do
  r=$(ceph osd pool get $p crush_rule -f json | jq -r .crush_rule)
  s=$(ceph osd pool get $p size -f json 2>/dev/null | jq -r .size)
  printf '%-20s %-16s size=%s\n' "$p" "$r" "$s"
done
ceph osd crush rule dump | jq -r '.[] | "\(.rule_name)\t\(.steps)"'

Produce the table, compare each row against the design document, and flag mismatches. Assignments made during incidents are the usual source of drift.

Why unused rules are worth removing and why removing them is risky

Rules accumulate. A cluster three years old typically has rules no pool uses, left from migrations and experiments.

They cost almost nothing at runtime, and they cost real clarity during an incident when someone has to determine which of seven rules a pool actually uses.

Removing them is safe only if genuinely unused:

ceph osd crush rule ls
for p in $(ceph osd pool ls); do
  ceph osd pool get $p crush_rule -f json | jq -r .crush_rule
done | sort -u

The difference between those two lists is the removal candidates.

The risk is rule ids: removing a rule does not renumber the others in current Ceph, but a subsequent hand-edit of the map might, and a pool referencing an id whose rule changed meaning will silently move all its data. Remove unused rules deliberately, verify every pool’s crush_rule afterwards, and never combine removal with other map edits.

Keeping it comprehensible

One rule per distinct policy, not one per pool.
Names that state root, class, and failure domain.
A design document mapping pool to rule to the failure it survives.
An audit after any CRUSH work.

Quiz

Knowledge check · 4 questions

Q1. Two pools use the same CRUSH rule. What does that mean for their performance isolation?

They are isolated, because pools have independent quotas and their own PG counts They are isolated only where their sizes differ, since size decides the OSD spread They share the same OSDs and therefore compete for the same device IOPS They share capacity but not performance, since Ceph schedules OSD work per pool
Q2. Unused CRUSH rules should be left in place, since removing them risks renumbering the rules pools depend on.

True False
Q3. A cluster has 14 pools and 9 rules, several named rule-N. Design an audit and cleanup.

Four-year-old cluster, three operators over its life, no design document. Rules include replicated_rule, rule-1, rule-3, ssd, ssd2, nvme, ec_pool_rule, hdd, and rack. Fourteen pools serve RBD, CephFS, and RGW. Some pool-to-rule assignments were made during incidents. The team wants to know what protection each pool actually has.
Q4. Describe a coherent rule set for a cluster with NVMe, SSD, and HDD across four racks, and state a pool assignment for each.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Name rules for their root, class, and failure domain, and keep one rule per distinct policy rather than one per pool. Maintain a document mapping each pool to its rule and to the failure it is meant to survive, because nothing in the cluster records intent. Audit that mapping after any CRUSH work — assignments made during incidents are the usual source of drift. And remember that pools sharing a rule share OSDs, so performance isolation is a disjoint-OSD decision rather than a consequence of being separate pools.

Cross-course references

Ceph: Part XVI (Device Classes) for the class-based tiers.
Ceph: Part XVII (Pools) for the pool settings that accompany rules.
Ceph: Part CIV (Multi-Tenancy in Practice) for isolation requirements.