Time and name resolution as deployment prerequisites

Why this matters in production

Time skew and name resolution failures produce Ceph symptoms that look like Ceph problems — authentication failures, monitor warnings, hosts that will not join. Both are trivially verifiable and both are frequently assumed rather than checked.

Time

Ceph requires monitors to agree on time closely. The default tolerance is tight:

ceph config get mon mon_clock_drift_allowed      # 0.05 seconds

# chrony on every host
systemctl enable --now chronyd
chronyc tracking
chronyc sources -v

# /etc/chrony.conf
server ntp1.example.com iburst
server ntp2.example.com iburst
server ntp3.example.com iburst
makestep 1.0 3

Three or more sources so a single bad one is outvoted. Every Ceph host on the same sources so they converge together.

ceph -s | grep -i clock
ceph time-sync-status

Symptoms of skew: MON_CLOCK_SKEW warnings, cephx authentication failures that appear after a period of working, S3 RequestTimeTooSkewed errors, and monitor elections.

Name resolution

# forward
getent hosts ceph-osd-01
dig +short ceph-osd-01.example.com

# reverse — frequently the missing one
dig +short -x 10.20.0.21

Both directions must work, from every host. Reverse resolution is the one usually forgotten and it affects daemon identification and log clarity.

hostnamectl status
hostname -f

The hostname must match what the cluster knows the host as. A mismatch between hostname -s and the name used in ceph orch host add produces a host that appears added and never receives daemons.

Verification before deployment

for h in ceph-mon-01 ceph-osd-01 ceph-osd-02; do
  echo "== $h"
  ssh "$h" 'hostname -f; chronyc tracking | head -3; getent hosts $(hostname -f)'
  dig +short -x "$(getent hosts "$h" | awk '{print $1}')"
done

Running this across every intended host before bootstrap catches both classes of problem while they are cheap to fix.

Symptom mapping

Quiz

Production discipline

Verify time synchronisation and both directions of name resolution across every host before bootstrap; both produce symptoms that read as Ceph faults and both are confirmed in one loop. Point every Ceph host at the same three or more NTP sources so they converge together rather than independently.

Cross-course references

• Kubernetes: certificate validity and etcd leases have the same clock sensitivity
• Linux: Kerberos and TLS both fail in the same characteristic way under clock skew