Lab: Configure NTP on a small cluster and verify synchronisation

This lab sets up chrony as both NTP server and NTP client on a small cluster, then verifies that every host is synchronised. By the end you will have a working NTP topology and a baseline that you can compare against.

Objective

By the end of this lab, you can:

• Configure chrony as an internal NTP server.
• Configure chrony as an NTP client on other cluster hosts.
• Verify synchronisation across the cluster.
• Demonstrate the impact of clock skew on TLS.

Architecture

You need at least 2 hosts (or VMs) on the same network:

• ntp-server (10.0.0.1): runs chrony as NTP server and client.
• cluster-node-1 (10.0.0.10): runs chrony as client.
• cluster-node-2 (10.0.0.11): runs chrony as client.

If you have more nodes, add them as additional clients.

Tasks

Task 1: Set up the NTP server

On ntp-server:

# /etc/chrony.conf (or /etc/chrony/chrony.conf)
pool time.cloudflare.com iburst maxsources 4
allow 10.0.0.0/24
local stratum 10
driftfile /var/lib/chrony/drift
makestep 1.0 3
rtcsync

# The unit is chronyd.service on RHEL/Fedora/SUSE and
# chrony.service on Debian/Ubuntu. This lab uses chronyd
# throughout; on a Debian-family host substitute chrony,
# and use /etc/chrony/chrony.conf for the config above.
sudo systemctl enable --now chronyd
sudo systemctl restart chronyd
sleep 30
chronyc tracking

Verify that chronyc tracking shows a small offset (<100 ms) and that the system clock is synchronised.

Open the port as well as the ACL:

# firewalld (RHEL, Fedora, SUSE)
sudo firewall-cmd --permanent --add-service=ntp
sudo firewall-cmd --reload
sudo firewall-cmd --list-services      # confirm 'ntp' is listed

# nftables (Debian, Ubuntu, or any default-deny baseline)
sudo nft add rule inet filter input ip saddr 10.0.0.0/24 udp dport 123 accept
sudo nft list chain inet filter input  # confirm the rule is above any drop

# ufw
sudo ufw allow from 10.0.0.0/24 to any port 123 proto udp

Confirm chronyd is actually bound and listening:

sudo ss -ulnp | grep ':123'
# UNCONN 0 0 0.0.0.0:123 0.0.0.0:* users:(("chronyd",pid=...,fd=...))

Task 2: Set up the clients

On each cluster node:

# /etc/chrony.conf
# Internal server(s) ONLY. See the callout below for why.
server 10.0.0.1 iburst
driftfile /var/lib/chrony/drift
makestep 1.0 3
rtcsync

sudo systemctl enable --now chronyd
sudo systemctl restart chronyd
sleep 30
chronyc tracking
chronyc sources -v

Verify:

• chronyc tracking shows small offset.

• chronyc sources shows ^* 10.0.0.1 (the server selected as primary).

• The Reach column for 10.0.0.1 climbs to 377. 377 is an octal bitmask of the last eight polls; all eight bits set means eight consecutive replies. A Reach of 0 means not one reply has ever arrived.

• The stratum is exactly one higher than the server’s. Stratum counts hops from the reference clock, so a client is always server stratum + 1 — never the same, never lower. If ntp-server syncs to a stratum-2 public pool it reports stratum 3, and these clients report stratum 4. That is the relationship recorded in the Task 4 baseline.

  chronyc tracking | grep -E 'Reference ID|Stratum'

  A client reporting the same stratum as the server, or one that is not server + 1, is not syncing to the server you think it is. Run chronyc sources -v and confirm the ^* marker sits on 10.0.0.1.

Troubleshooting: reach is 0 and no source is selected

This is the failure the previous step exists to prevent, so work the transport first. Suspect the firewall before you suspect chrony.

# On the CLIENT, watch the wire while chrony polls
sudo tcpdump -i any -nn 'udp port 123 and host 10.0.0.1'

chronyc -N sources and journalctl -u chronyd -n 50 fill in the rest. Note that a client with an unreachable source logs almost nothing, which is why the packet capture is the fast path.

Task 3: Verify cluster-wide synchronisation

From any cluster node:

for host in 10.0.0.1 10.0.0.10 10.0.0.11; do
    echo "=== $host ==="
    ssh user@$host "date -u; chronyc tracking | grep -E 'Reference|System time|Last offset'"
done

All hosts should show:

• Same time (within 1 second).
• Small offset (<100 ms).
• The same reference (the NTP server).

If a host is more than 1 second off, NTP has not yet caught up. Wait a few minutes and recheck.

Task 4: Document the baseline

Capture the state of every host for the inventory:

CLUSTER NTP BASELINE
====================
Date: ...

ntp-server (10.0.0.1)
- Sources: time.cloudflare.com
- Stratum: 3
- System time offset: <50 ms
- Tracking: ...

cluster-node-1 (10.0.0.10)
- Sources: 10.0.0.1 (only)
- Stratum: 4
- System time offset: <20 ms
- Tracking: ...

cluster-node-2 (10.0.0.11)
- ...

Save this for future comparison. A drift in any host is a diagnostic.

Task 5: Simulate clock skew

Pick one of the two safe paths:

1. Preferred: run this task on ntp-server or a spare VM that is not a Pacemaker/corosync member. Nothing to stop, nothing to fence.
2. If you must use a cluster node: put it into standby and stop cluster services first with pcs node standby cluster-node-2 followed by pcs cluster stop cluster-node-2, and confirm with pcs status that resources have moved off it.

Record the healthy baseline before you break anything:

chronyc tracking > /tmp/tracking-before.txt
chronyc sources -v

Now step the clock:

sudo date -s "1 hour ago"

The system clock is now 1 hour behind. Verify:

chronyc tracking

chrony sees the new offset and begins slewing - adjusting the rate at which the clock advances rather than jumping it. Slewing is bounded by maxslewrate, which defaults to 83333.333 ppm. That is about 1 second of correction per 12 seconds of wall time, so a 1-hour offset would take roughly 12 hours to slew away.

That number is the operational decision you will face in a real skew incident: wait, or force a step? Slewing preserves monotonic forward time, so nothing that timestamps or holds a lease is surprised, but it is far too slow for a large offset. A step is instant but moves the clock discontinuously, which is exactly what breaks clustered software.

makestep 1.0 3 in the config only permits a step for the first three updates after chronyd starts. Well past that point, the runtime command is the tool:

sudo chronyc makestep      # correct the clock by stepping immediately
sudo chronyc tracking      # confirm the offset is back under 100 ms

Test the impact of the skew (do this while the clock is still wrong, before makestep):

curl -I https://example.com    # may fail with certificate error

A 1-hour skew causes TLS validation failures because the clock is before some certificates’ notBefore.

Restore the node:

sudo chronyc makestep
sleep 5
chronyc tracking
diff /tmp/tracking-before.txt <(chronyc tracking)

# Only once the clock is correct, bring the node back
sudo pcs cluster start cluster-node-2
sudo pcs node unstandby cluster-node-2
sudo pcs status

Task 6: Test the impact on TLS

Generate a certificate with a notBefore in the future (if you have a CA available):

# Generate a cert with notBefore in the future
openssl req -x509 -newkey rsa:2048 -keyout key.pem -out cert.pem -days 365 -nodes -subj "/CN=test"
# notBefore will be "now"; manipulate if needed

Or use any certificate with a future notBefore. With a 1-hour skew, certificate validation may fail.

For a real test:

# On the skewed host:
curl -vI https://www.example.com 2>&1 | grep -E 'certificate|verify|date'

# Restore and retest:
sudo chronyc makestep
sleep 5
curl -vI https://www.example.com 2>&1 | grep -E 'certificate|verify|date'

Task 7: Configure monitoring (if available)

If you have Prometheus and node_exporter, the offset is exposed as node_timex_offset_seconds. Add an alert:

- alert: ClockSkew
  expr: abs(node_timex_offset_seconds) > 0.1
  for: 5m
- alert: ClockSkewCritical
  expr: abs(node_timex_offset_seconds) > 5
  for: 1m
- alert: ClockNotSynchronised
  expr: node_timex_sync_status == 0
  for: 10m

Prove the rule rather than trusting it. While the node is still an hour behind from Task 5, check that the expression evaluates truthy:

# from the Prometheus host
curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=abs(node_timex_offset_seconds) > 5' | jq '.data.result'

An empty result array while the node is visibly skewed means the rule is one-sided or the exporter is not being scraped. Fix that before moving on.

For hosts without Prometheus, set up a cron job:

# Every 5 minutes, log the offset
*/5 * * * * chronyc tracking | grep 'System time' >> /var/log/ntp-offset.log

Validation

• The NTP server is configured and serving time.
• UDP/123 is open on the server’s packet filter, verified from a client with chronyc sources -v showing Reach 377, not from the server itself.
• All clients sync to the server with <100 ms offset.
• The cluster-wide baseline is documented.
• A simulated skew is corrected with chronyc makestep, and the node was outside the cluster (or had cluster services stopped) while it was skewed.
• The skew alert uses abs(node_timex_offset_seconds) and was observed to fire while the node was an hour behind. A one-sided > rule cannot fire in that direction, so it is not a valid pass.

Cleanup

Correct any time changes and return the node to the cluster:

sudo chronyc makestep
chronyc tracking          # offset must be back under 100 ms

sudo pcs cluster start cluster-node-2   # only if you stopped it
sudo pcs node unstandby cluster-node-2
sudo pcs status

What you learned

• chrony works as both client and server.
• allow is an application ACL; the host packet filter must permit UDP/123 separately, and the failure is silent.
• Cluster nodes must share the same set of time sources so that they agree with each other. Inter-node agreement matters more than absolute accuracy, and a node with a private fallback source is free to drift away from its peers.
• Stratum increases by exactly one at each hop, so a client’s stratum is a cheap check on whether it is really following the server you configured.
• A baseline is essential for diagnosing future drift.
• Slewing is bounded by maxslewrate and is far too slow for a large offset; chronyc makestep is the deliberate fix.
• A backward clock step on a cluster member can get it fenced.
• Clock skew causes real production failures (TLS).