A time skew incident, worked through

Advanced⏱ ~18 mincephchrony

What you'll learn

Follow a realistic incident from symptom to cause
Recognise the investigative wrong turns
Apply the correct diagnostic sequence
Derive preventive measures from an incident

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Time skew incidents are instructive because the symptoms point everywhere except at the cause. Walking one through shows both the wrong turns and the check that ends it.

09:14 — first symptom

Monitoring alerts on S3 error rate. A fraction of requests return SignatureDoesNotMatch. The gateway team begins investigating credentials.

[WRN] Client SignatureDoesNotMatch: 4.2% of requests

09:22 — the second symptom

Monitors begin electing.

ceph -s
#   cluster:
#     health: HEALTH_WARN
#             clock skew detected on mon.ceph-mon-02
#   services:
#     mon: 3 daemons, quorum ceph-mon-01,ceph-mon-03 (age 12s)

Two teams are now investigating two problems.

09:31 — the wrong turn

The gateway team rotates the S3 credentials, reasoning that a signature failure means a bad key. The error rate is unchanged, because the credential was never the problem.

The monitor team considers raising mon_clock_drift_allowed to stop the warning.

09:38 — the check that ends it

Someone runs the clock comparison:

for h in ceph-mon-01 ceph-mon-02 ceph-mon-03 ceph-rgw-01 ceph-rgw-02; do
  printf '%-16s %s\n' "$h" "$(ssh "$h" date -u +%s.%3N)"
done

ceph-mon-01      1755504718.412
ceph-mon-02      1755504720.187      ← 1.8 s ahead
ceph-mon-03      1755504718.409
ceph-rgw-01      1755504718.415
ceph-rgw-02      1755504720.191      ← 1.8 s ahead

Two hosts are 1.8 seconds ahead. Both symptoms have one cause.

09:41 — the root cause

ssh ceph-mon-02 chronyc sources -v
# ^? ntp1.example.com     0   6   0    -     +0ns[   +0ns] +/-    0ns
# ^* ntp-legacy.example   1   6  377   34  +1802ms[+1802ms] +/-  4.1ms

Both hosts had a legacy NTP server in their configuration that the others did not, and it had begun serving incorrect time. The hosts dutifully followed it.

09:44 — recovery

ssh ceph-mon-02 'sed -i "/ntp-legacy/d" /etc/chrony.conf'
ssh ceph-mon-02 'systemctl restart chronyd && chronyc makestep'
ssh ceph-rgw-02 'sed -i "/ntp-legacy/d" /etc/chrony.conf'
ssh ceph-rgw-02 'systemctl restart chronyd && chronyc makestep'

ceph -s
ceph time-sync-status

Both symptoms clear within a minute of the clocks converging.

The lessons

Lesson	Action
Symptoms appearing together suggest one cause	correlate before investigating separately
Signature failures are often clocks	check time before rotating credentials
Configuration drift across a fleet is the underlying fault	enforce time configuration through configuration management
The skew warning was suppressible	never raise the tolerance to silence it
The offset was detectable before it caused symptoms	alert on offset, not on Ceph’s warning

Quiz

Knowledge check · 4 questions

Q1. What single check would have identified this incident's cause at 09:14 rather than 09:38?

Rotating the S3 access credentials Comparing clocks across all hosts Checking the bucket policies on RGW Restarting both gateway daemons
Q2. If every host in the fleet had followed the bad time server, monitor consensus would still have been disrupted.

True False
Q3. Derive preventive measures from the incident.

Following this incident, the team is asked what would prevent a recurrence. The immediate fix — removing the bad server from two hosts — has been applied.
Q4. Why should the incident checklist include a clock comparison before credential rotation?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Add a fleet-wide clock comparison to the first-response checklist and alert on offset well below Ceph’s own threshold — the warning arrives after symptoms begin. Enforce time configuration through configuration management; divergence between hosts is what turns a bad time server into a cluster incident.

Cross-course references

Kubernetes: configuration drift between nodes produces the same class of partial failure
Linux: post-incident review distinguishing trigger from root cause is general practice