Configuring time synchronisation for Ceph

Why this matters in production

Ceph’s tolerance is 50 milliseconds by default, which is tighter than most services require and tighter than a default installation reliably achieves. Configuring it deliberately and verifying the result is what prevents a class of confusing failures.

Configuration

# /etc/chrony.conf
server ntp1.example.com iburst
server ntp2.example.com iburst
server ntp3.example.com iburst

# step the clock if the offset is large, but only in the first few updates
makestep 1.0 3

# allow the local clock to be disciplined quickly
maxupdateskew 100.0
rtcsync

systemctl enable --now chronyd

Three or more sources so one bad server is outvoted. The same sources on every Ceph host so they converge on the same time rather than independently on nearby ones.

Verifying it works

Installing chrony is not the same as being synchronised:

chronyc tracking

Reference ID    : C0A80101 (ntp1.example.com)
Stratum         : 3
System time     : 0.000012 seconds fast of NTP time
Leap status     : Normal

chronyc sources -v
chronyc sourcestats

A source marked ^* is the selected one; ^? means unreachable.

Across the cluster

ceph time-sync-status
ceph -s | grep -i clock

# a direct comparison
for h in ceph-mon-01 ceph-mon-02 ceph-mon-03; do
  printf '%-16s %s\n' "$h" "$(ssh "$h" date -u +%s.%N)"
done

Virtualised hosts

Virtual machines have clocks that drift differently — the hypervisor may pause them, and time can jump on migration.

# check the clocksource
cat /sys/devices/system/clocksource/clocksource0/current_clocksource

kvm-clock or tsc are appropriate. Ensure the hypervisor is not also synchronising the guest clock independently, which fights chrony and produces oscillation.

Monitoring

chronyc tracking | awk '/System time/{print $4}'

Alert on the offset exceeding a fraction of the Ceph tolerance — perhaps 20 ms — so drift is caught before it causes symptoms rather than after.

Quiz

Production discipline

Verify chronyc tracking on every host rather than checking that the service is running; installed and synchronised are different states and only one of them is sufficient. Configure makestep so a rebooted host converges in seconds, since the slow-correction default produces warnings that appear after every reboot.

Cross-course references

• Kubernetes: certificate and token validity depend on the same clock discipline
• Linux: time synchronisation verification is standard practice with a tighter tolerance here