What breaks in Ceph when time is wrong

Why this matters in production

Time skew produces symptoms across authentication, consensus, and the gateway that look unrelated to each other and unrelated to time. Knowing the list means one check resolves an investigation that otherwise spans several subsystems.

The five mechanisms

The second and third are the ones that mislead: both look like credential problems and neither is.

Monitor consensus

ceph config get mon mon_clock_drift_allowed      # 0.05
ceph -s | grep -i clock
ceph time-sync-status

Monitors grant each other time-limited leases. A monitor whose clock runs fast may consider its lease expired while the leader still considers it valid, which can trigger an election. Repeated elections destabilise the cluster.

cephx

Tickets are issued with an expiry. A client whose clock differs from the monitors’ by more than the tolerance presents tickets the monitors consider invalid, or considers valid tickets expired.

The signature is characteristic: authentication works, and then stops after a period — because the initial ticket was accepted and the renewal is not.

S3 signing

RequestTimeTooSkewed: The difference between the request time and the
current time is too large

The timestamp is part of the signed canonical request and is checked against a tolerance. A gateway with a drifting clock rejects the fraction of requests that reach it.

TLS

Certificates carry not-before and not-after times. A host whose clock is far in the past sees a valid certificate as not yet valid; one far in the future sees it as expired. Both present as certificate errors with no indication that the clock is the cause.

Log correlation

Without synchronised clocks, reconstructing an incident from logs across hosts is guesswork — events appear out of order and causality cannot be established. This costs nothing until an incident, and then costs a great deal.

Quiz

Production discipline

Check clocks before investigating any credential or certificate problem that appeared without a configuration change; the symptoms wear credential clothing and the check costs seconds. Treat several apparently-unrelated failures appearing together as a signal to look for a shared dependency, and time is the most common one.

Cross-course references

• Kubernetes: certificate rotation and token validity fail identically under skew
• Linux: Kerberos, TLS, and any time-limited credential share this sensitivity