Certificates, msgr2, and time

Why this matters in production

A certificate error on a host whose clock is wrong is indistinguishable from a genuine certificate problem, and the response to each is completely different. Checking the clock first turns a certificate investigation into a one-command diagnosis.

Certificate validity

Every certificate carries a validity window:

openssl x509 -in /etc/ceph/rgw.pem -noout -dates
# notBefore=Aug 18 00:00:00 2026 GMT
# notAfter=Aug 18 00:00:00 2027 GMT

A host whose clock is before notBefore reports the certificate as not yet valid; one after notAfter reports it as expired. Both are correct from that host’s perspective and both mean the clock, not the certificate.

Distinguishing the two

# what does this host think the time is?
date -u

# what does the certificate say?
openssl x509 -in /etc/ceph/rgw.pem -noout -dates

# what does a correctly-synchronised host think?
ssh known-good-host date -u

# verify from this host's perspective
openssl s_client -connect rgw.example.com:443 -servername rgw.example.com </dev/null 2>&1 | \
  grep -E 'Verify return code|notBefore|notAfter'

That last pair is the quickest discriminator: a genuine expiry affects everyone at once, while a clock problem affects one host.

Certificate rotation

Newly-issued certificates commonly have a notBefore of the issuance moment. A host whose clock is even slightly behind sees a brand-new certificate as not yet valid.

# issue with a backdated notBefore where the CA allows it
openssl req -x509 -newkey rsa:4096 -nodes \
  -keyout rgw.key -out rgw.crt -days 365 \
  -subj "/CN=rgw.example.com"

Many CAs backdate by an hour for exactly this reason. Where yours does not, verify clock synchronisation across the fleet before deploying a new certificate.

msgr2 secure mode

msgr2’s secure mode uses AES-GCM with keys derived through the cephx exchange rather than X.509 certificates, so it has no certificate validity window. It remains time-sensitive through cephx: the tickets underpinning the key exchange are time-limited.

ceph config get global ms_cluster_mode
ceph -s | grep -i clock

So a clock problem breaks msgr2 secure mode through authentication rather than through certificate validation — the same cause with a different error.

Quiz

Production discipline

Check whether a certificate error affects one host or all of them before investigating the certificate; the answer distinguishes a clock problem from a certificate problem immediately. Verify clock synchronisation across the fleet before deploying a newly-issued certificate, since a notBefore of the issuance moment is unforgiving of hosts running slightly behind.

Cross-course references

• Kubernetes: kubelet certificate rotation fails the same way on hosts with skewed clocks
• Linux: TLS validation is clock-dependent in every context it appears