Reconstructing the partition from evidence

Intermediate⏱ ~17 mincephjournalctl

What you'll learn

State the questions a partition post-mortem must answer
Collect the timeline from cluster and daemon evidence
Correlate two sides whose clocks disagreed
Convert findings into specific configuration changes

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

The evidence that explains a partition ages out of logs on a schedule nobody chose, and the number stakeholders actually want — how long data was unavailable — is not in any dashboard. Both have to be extracted while the cluster still remembers.

The five questions

Question	Where the answer lives
When did the path fail	first heartbeat_check timestamp, OSD logs
Which side held quorum	monitor election entries in the cluster log
Which OSDs went down, and when	OSDMap epoch changes, cluster log
How long was any PG inactive	PG_AVAILABILITY appearance and clearance
How long did recovery take	degraded count reaching zero

The fourth is the customer-visible outage. The others explain it.

Collecting it before it rotates

ceph log last 20000 info cluster > /secure/incident/cluster.log
ceph crash ls
ceph report > /secure/incident/report.json

journalctl -u ceph-$FSID@mon.ceph-mon-a.service --since '2026-08-18 09:00' \
  > /secure/incident/mon-a.log
journalctl -u ceph-$FSID@osd.42.service --since '2026-08-18 09:00' \
  > /secure/incident/osd-42.log

ceph report captures the current OSDMap, monmap, and PG summary in one JSON document — worth taking even when you think you will not need it, because it is the only artefact that stops being reproducible.

Reading the monitor side

2026-08-18T09:13:59 mon.ceph-mon-a calling monitor election
2026-08-18T09:14:04 mon.ceph-mon-a is new leader, mons ceph-mon-a,ceph-mon-b
                    in quorum (ranks 0,1)
2026-08-18T09:14:22 osdmap e91846: 36 total, 24 up, 36 in
2026-08-18T09:24:23 osdmap e91903: 36 total, 24 up, 24 in

The election entry establishes which side was authoritative and from what moment. The 36 in becoming 24 in ten minutes later is the out interval firing, and it is the line that explains the size of the recovery.

Correlating across a clock boundary

Two sides of a partition may not agree on the time, and if the isolated side lost its NTP source the drift grows for the length of the outage.

chronyc tracking
chronyc sources -v

Anchor both sides to the OSDMap epoch sequence instead. Epochs are globally ordered and assigned by the quorum, so an event on the isolated side that references epoch 91846 provably happened after the quorum side committed it, regardless of what either host’s clock says.

Turning it into changes

Finding	Change
`noout` never set	add it as step one of the partition runbook
Monitors share a failure domain	redistribute the monmap
Even monitor count	move to an odd count
Recovery starved clients	set an `osd_mclock_profile` policy in advance
Flap ran unchecked	document `nodown` with an owner and expiry
Nobody knew which side had quorum	put `ceph mon stat` first in the runbook

Quiz

Knowledge check · 4 questions

Q1. Which artefact should be captured first after a partition, because it stops being reproducible?

`ceph report`, which snapshots the current OSDMap, monmap, and PG summary `ceph osd tree`, whose per-OSD up and in flags are overwritten as recovery proceeds The dashboard performance graphs, which the manager discards when it fails over to a standby `ceph df`, because pool usage at the time of the incident cannot be recovered afterwards
Q2. OSDMap epochs order events across the two sides of a partition more reliably than log timestamps do.

True False
Q3. Produce a post-mortem timeline for a partition that has just resolved.

A 40-minute rack isolation ended two hours ago. Recovery is complete. The incident review is tomorrow and asks for the duration of data unavailability.
Q4. Why is the isolated side's log a poor basis for the incident timeline?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Capture ceph report, the cluster log, and the per-daemon journals before you leave the incident — they are the only artefacts that stop being reproducible. Report unavailability from the PG_AVAILABILITY window rather than from alert timestamps, and anchor any cross-side ordering to OSDMap epochs, because a partition is exactly the event that breaks clock agreement.

Cross-course references

Kubernetes: event objects expire on a TTL and are gone before the review meeting
Linux: journald rotation is sized, not time-bounded, so a chatty incident evicts its own evidence