Recording rules for Ceph

Advanced⏱ ~17 minprometheus

What you'll learn

Write recording rules for common Ceph queries
Name rules according to convention
Decide when a rule is justified
Avoid the common recording rule mistakes

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Some Ceph queries are expensive enough that a dashboard with several of them is slow, and an alert evaluating them every fifteen seconds is wasteful. Recording rules compute them once.

Rules worth having

groups:
  - name: ceph.rules
    interval: 30s
    rules:
      - record: cluster:ceph_osd_utilisation:max
        expr: max(ceph_osd_stat_bytes_used / ceph_osd_stat_bytes)

      - record: cluster:ceph_osd_utilisation:avg
        expr: avg(ceph_osd_stat_bytes_used / ceph_osd_stat_bytes)

      - record: cluster:ceph_osd_utilisation:spread
        expr: |
          max(ceph_osd_stat_bytes_used / ceph_osd_stat_bytes)
            - avg(ceph_osd_stat_bytes_used / ceph_osd_stat_bytes)

      - record: cluster:ceph_op_rate:sum
        expr: sum(rate(ceph_osd_op_r[5m])) + sum(rate(ceph_osd_op_w[5m]))

      - record: cluster:ceph_write_latency_p99:histogram
        expr: |
          histogram_quantile(0.99,
            sum(rate(ceph_osd_op_w_latency_bucket[5m])) by (le))

      - record: cluster:ceph_pgs_inactive:count
        expr: ceph_pg_total - ceph_pg_active

      - record: osd:ceph_apply_latency:topk
        expr: topk(10, ceph_osd_apply_latency_ms)

Naming

The convention is level:metric:operation:

cluster:ceph_osd_utilisation:max
pool:ceph_stored_bytes:rate5m
osd:ceph_apply_latency:topk

Part	Meaning
`level`	the aggregation level — cluster, pool, osd, host
`metric`	what is being measured
`operation`	how it was derived

Consistent naming means a rule can be found without reading the rules file, which is the whole point of having them.

When a rule is justified

Situation	Rule worth it?
A query used in several dashboards	yes
A query used by an alert evaluated frequently	yes
An expensive aggregation over many series	yes
A query used once, on one panel	no
A simple gauge lookup	no

# measure a query's cost before deciding
# the expression you are considering recording; URL-encode it if it has spaces
EXPR='sum(rate(ceph_osd_op_r[5m]))'

curl -sg "http://prometheus:9090/api/v1/query?query=$EXPR" \
  -w '\n%{time_total}s\n' -o /dev/null

Common mistakes

Mistake	Consequence
Recording a rule with `rate` over a short window	precomputed noise
Rule interval longer than the query window	gaps
Preserving all labels	no cardinality reduction
Recording a rule nobody uses	pure cost
Alerting on a rule with a stale interval	delayed alerts

# the rule interval should be <= the rate window
- name: ceph.rules
  interval: 30s          # with rate(...[5m]) this is fine

Quiz

Knowledge check · 4 questions

Q1. Why does a recording rule that preserves all source labels save nothing?

Prometheus refuses to load a rule whose recorded series carry the same label set as the source series, so the group never evaluates Recorded series are written to a separate block that the label index does not cover, so queries against them scan every sample An alerting rule may only reference a recorded series that aggregates, so an unaggregated rule cannot serve the alert it was written for Query cost is dominated by series count, so recording the same number of series doubles storage without speeding queries
Q2. A rule evaluation interval longer than the rate window is acceptable if the rule is only used for graphs.

True False
Q3. Speed up a slow Ceph dashboard.

A Ceph dashboard takes 20 seconds to load. It has 15 panels, several of which compute histogram quantiles over 500 OSDs and several of which repeat the same aggregation.
Q4. What is the recording rule naming convention and why does it matter?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Create recording rules only where they aggregate — an unaggregated rule doubles storage and speeds nothing. Follow the level:metric:operation convention and keep the evaluation interval at or below the rate window, or the recorded series carries systematic coverage gaps.

Cross-course references

Kubernetes: the standard kube-prometheus rules follow this same convention
Linux: precomputing expensive aggregates is a general monitoring practice