Running a dedicated cluster as a service

Intermediate⏱ ~18 mincephcephadm

What you'll learn

Define the operational practices a storage service needs
Establish change management appropriate to the risk
Build the runbook set
Set and meet service commitments

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

A dedicated cluster is a service with consumers, and the practices that make a service dependable are specific and mostly independent of the technology.

The practice set

Practice	Purpose
Runbooks for the recurring incidents	consistent, fast response
Change windows with consumer notice	no surprises
Capacity planning with lead time	expansion lands before it is needed
Alerting reviewed periodically	coverage stays accurate
Baselines and benchmarks	performance changes are detectable
Documented service commitments	expectations are measurable
Post-incident reviews	recurring causes are addressed
Rehearsed recovery procedures	the unfamiliar becomes routine

The runbook set

runbooks/ceph/
  osd-down.md
  osd-replace.md
  pg-inconsistent.md
  slow-ops-triage.md
  blocked-ops-sweep.sh
  capacity-nearfull.md
  monitor-quorum-loss.md
  cluster-upgrade.md
  host-maintenance.md
  pool-creation.md
  onboarding-a-consumer.md

Eleven documents covering everything that recurs. Each should state the symptom, the diagnostic commands, the decision points, and the rollback.

# runbooks live with the monitoring configuration
git add runbooks/ceph/

Change management

Change	Process
Pool creation for a new consumer	ticket, capacity check, quota
Configuration change	review, tested in staging, recorded
Upgrade	change window, consumer notice, rollback plan
Hardware replacement	scheduled, capacity verified first
CRUSH rule change	estimated with crushtool, scheduled
Emergency change	recorded after the fact with the reason

The last row matters: emergency changes happen and the practice is recording them, not preventing them.

# what changed, and when
ceph config log | head -20
ceph config dump > config-$(date +%F).txt

Service commitments

Ceph storage service commitments
  Availability:   99.95% per pool, measured as PGs active
  Latency:        p99 write under 10 ms for NVMe pools, under 30 ms for HDD
  Capacity:       expansion triggered at 70% of usable, before consumers notice
  Recovery:       degraded PGs restored within 24 hours of a single failure
  Notice:         5 working days for planned work affecting latency
  Support:        24/7 for availability, business hours otherwise

Each commitment must be measurable and monitored, or it is an aspiration.

Meeting them

# availability, measured
# in Prometheus, over the reporting period:
#   avg_over_time((ceph_pg_active / ceph_pg_total)[30d:])

# latency against commitment
#   histogram_quantile(0.99, sum(rate(ceph_osd_op_w_latency_bucket[5m])) by (le))

- alert: LatencyCommitmentBreached
  expr: |
    histogram_quantile(0.99,
      sum(rate(ceph_osd_op_w_latency_bucket[5m])) by (le)) > 0.010
  for: 30m
  labels: { severity: page }

Quiz

Knowledge check · 4 questions

Q1. Why does rehearsing recovery procedures matter more on a dedicated cluster?

Dedicated clusters fail more often, since their OSD count is higher and every device adds its own probability of failing The cluster runs itself, so the team may go months without performing consequential procedures and lacks the familiarity frequent exposure provides The procedures are more complex on a dedicated cluster, because the monitors and OSDs sit on separate hosts each with their own access paths and credentials Consumers require it, since a shared service has to show that its recovery procedures have been exercised recently
Q2. Writing the monitoring query for a service commitment is what forces the commitment to be specific enough to mean anything.

True False
Q3. Establish operational practice for a new storage service.

A dedicated Ceph cluster is entering production, serving three consumer teams. No runbooks, commitments, or change process exist yet.
Q4. What should each runbook state?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Write a monitoring query alongside every service commitment — without one the commitment cannot be verified and exists only as a claim. Schedule rehearsal of the rare procedures; a stable dedicated cluster means the team lacks the familiarity that frequent exposure would provide.

Cross-course references

Kubernetes: platform teams operate clusters as services with the same practice set
Linux: any infrastructure service needs runbooks, commitments, and rehearsal