Skip to main content
RunBook Academy

← All break/fix scenarios in Ceph

advancedceph-clock~20 min

Time skew breaks MON election - clock skew warning

Reported symptoms

  • ●ceph -s shows clock skew between MONs
  • ●A MON fails to acknowledge a lease and steps down
  • ●Election cycles repeatedly

Evidence

  • · ceph -s reports clock skew on the affected MON
  • · chronyc tracking on the affected MON shows an offset
  • · chronyc sources on the affected MON shows a failed NTP source
Diagnosis and resolutionclick to reveal

Root cause

Time synchronisation has failed on one MON. The MON cannot maintain its Paxos lease.

Remediation

Restore NTP or chrony on the affected host; verify the offset returns to < 100ms; verify the MON rejoins quorum.

Verification

ceph -s reports no clock skew; all MONs are in quorum; election is stable.

Prevention

Run chrony or systemd-timesyncd on every host; alert on NTP source failure; monitor cluster clock skew.

Architecture

The student examines the broken architecture diagram and identifies the responsible component. The cluster is running Ceph Tentacle (20.2.x); the incident is one a production operator must diagnose from the evidence presented.

Symptoms

  • ceph -s shows clock skew between MONs
  • A MON fails to acknowledge a lease and steps down
  • Election cycles repeatedly

Evidence

  • ceph -s reports clock skew on the affected MON
  • chronyc tracking on the affected MON shows an offset
  • chronyc sources on the affected MON shows a failed NTP source

Student investigation

The student follows the methodology: define the symptom, determine the impact, gather evidence, identify the component, form a hypothesis, test safely, restore, validate.

Progressive hints

  • Hint 1: check the cluster state with ceph -s first.
  • Hint 2: read ceph health detail and identify the affected PGs / OSDs / daemons.
  • Hint 3: use ceph osd tree, ceph pg dump, or ceph mds stat as the next-level diagnostic.
  • Hint 4: the root cause is documented in the frontmatter root_cause field.

Validation

The student runs the verification steps and confirms the symptom cleared.

Root cause

See the frontmatter root_cause field.

Remediation

Apply the fix from the frontmatter remediation field.

Prevention

Apply the prevention measures from the frontmatter prevention field.