NTP fleet design - hierarchy, falsetickers and leap seconds · Linux

Every lesson in this part so far has configured one host. A fleet raises a different question, and it is not “how accurate is each host”. It is consistency: a thousand hosts that are all 40 ms away from true time interoperate perfectly, and two hosts that differ from each other by five minutes break Kerberos between them no matter how close either is to a public server.

Consistency comes from structure. Hosts that share a source share its error; hosts that each pick their own from the public pool do not.

The shape

stratum 0    GPS / radio / atomic reference
                       |
stratum 1    public or vendor stratum-1 servers
                       |
stratum 2    ntp1..ntp4.example.com     <- YOUR internal servers
                       |
stratum 3    every other host in the estate

Two tiers, and the reasons for the internal tier are all operational rather than technical:

One outbound dependency instead of a thousand. Four hosts need UDP/123 to the internet; the rest need it only to the internal servers. That is a firewall rule you can actually audit.
Shared error. Every host derives its time from the same four servers, so their errors correlate. That is what makes timestamps comparable across the fleet.
One place to monitor. The internal servers are the four hosts whose time health you watch closely.
A working answer when the internet is unavailable. The internal servers keep serving from their own disciplined clocks, and the fleet stays consistent with itself even while drifting away from UTC.

Isolated networks stop at the internal tier and use local stratum 10 as linux-chrony-architecture describes, which makes the fleet internally consistent and honest about not being traceable to UTC.

Why the number of sources is the design decision

This is the part that gets skipped, and it is the whole reliability argument.

Sources	What you can conclude when they disagree
1	Nothing. There is no disagreement to observe. A single wrong source is followed silently.
2	That one of them is wrong. Not which.
3	A majority of two can outvote one, but losing any source drops you to the 2-source case.
4	One source can be wrong and a clear majority of three remains.

Four is the standard recommendation and it is not superstition — it is the smallest number that tolerates one bad source and one unreachable source at the same time.

The failure mode with a single source deserves stating plainly, because it is the one people build by accident. A host with one NTP server follows that server wherever it goes. If the server is misconfigured, or its own upstream fails and it starts serving from a drifting local clock, the client tracks it faithfully into the wrong time and reports itself perfectly synchronised the whole way.

chrony lets you refuse to act on insufficient agreement:

# /etc/chrony.conf on a fleet host
server ntp1.example.com iburst
server ntp2.example.com iburst
server ntp3.example.com iburst
server ntp4.example.com iburst

# Do not discipline the clock unless at least 3 sources agree
minsources 3

# Reject a source whose estimated maximum error exceeds this
maxdistance 1.0

minsources defaults to 1, which is the permissive setting. Raising it means a host with only one reachable source stops adjusting rather than following it — the clock free-runs on its last known frequency, which is more accurate over hours than chasing a source you cannot corroborate.

maxdistance defaults to 3 seconds. Lowering it excludes servers whose accumulated root delay and dispersion make their time too uncertain to be useful — typically a server that has itself lost its upstream and is coasting.

Falsetickers

A falseticker is a source that is reachable, responsive, and wrong. chrony identifies them by intersecting the confidence intervals of all sources and finding the ones that do not overlap the majority.

Read-only / Safefalseticker detection— Both commands are read-only. The x state means chrony believes this source may be in error: its interval does not intersect the interval agreed by the others. selectdata shows why - the Interval columns for ntp4 sit entirely outside the range the other three share. This host is fine, because three good sources outvote one bad one. With only two sources configured there would be no majority and chrony would refuse to select either.

$ chronyc sources -v
chronyc selectdata -v

  .-- Source mode  '^' = server, '=' = peer, '#' = local clock.
/ .- Source state '*' = current best, '+' = combined, '-' = not combined,
| /             'x' = may be in error, '~' = too variable, '?' = unusable.
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^* ntp1.example.com              2   6   377    35    +12us[  -23us] +/-  850us
^+ ntp2.example.com              2   6   377    34    -45us[  -67us] +/-  920us
^+ ntp3.example.com              2   6   377    33   +120us[  +98us] +/- 1100us
^x ntp4.example.com              2   6   377    32    +2.1s[ +2.1s] +/-  980us

S Name/IP Address        Auth COpts EOpts Last Score     Interval  Leap
=======================================================================
* ntp1.example.com          N ----- -----    2   1.0 -1846us +1305us  N
+ ntp2.example.com          N ----- -----    1   1.0 -1381us +1355us  N
+ ntp3.example.com          N ----- -----    3   1.0 -1502us +1611us  N
x ntp4.example.com          N ----- -----    2   1.0  +2.09s  +2.10s  N

Illustrative output

An x on a fleet host is a finding about the server, not about the client. One client seeing ntp4 as a falseticker is noise; every client seeing it is ntp4 being broken, and it should page. Alert on the count of sources in the x state across the fleet, aggregated per server name — that turns a per-host curiosity into a signal that names the faulty server.

Pointing the whole fleet at one internal server produces a fleet that is consistently wrong

The internal tier is supposed to make the fleet consistent. A single internal server does that perfectly and provides no protection at all: when it is wrong, every host is wrong by the same amount, agreeing with each other, with no source of disagreement anywhere in the estate to notice.

That is worse than the alternative it replaced. A thousand hosts each using the public pool are inconsistent, but a single bad public server affects a few of them and the disagreement is visible.

The internal tier needs to be redundant at both levels:

Four internal servers, on different hosts, ideally in different failure domains, each configured into every client.
Four upstream sources on each internal server, so an internal server can itself detect a bad upstream.
Different upstream providers across the internal servers, so a single provider’s error does not reach all four the same way.

Two internal servers is the minimum that survives a reboot; it is not enough to resolve a disagreement, because a client seeing two servers differ cannot tell which to believe.

Leap seconds

A leap second is an extra second inserted into UTC to keep it close to solar time: the last minute of June or December gets a 23:59:60. Unix time has no representation for it — every day has 86400 seconds by definition — so something has to give.

chrony’s leapsecmode directive chooses what:

Mode	Behaviour
`system`	The kernel steps the clock back one second at 00:00:00 UTC. The default where the system supports it.
`step`	The same, but chronyd does the stepping rather than the kernel.
`slew`	The clock is corrected by slewing from 00:00:00 UTC. On Linux with the default slew rate this takes about 12 seconds.
`ignore`	No correction; the one-second error is absorbed by normal operation later.

The reason this is a fleet-level decision rather than a host one is the same reason everything else in this lesson is: the modes disagree with each other. A cluster where some nodes step backwards and others slew has nodes that differ by up to a second for the duration, which is enough to reorder timestamps in a distributed database and enough to trip cluster timers.

A backward step is the specific hazard. It invalidates corosync token timers and Pacemaker lease deadlines exactly as linux-chrony-client-config describes for makestep, and it breaks any application that assumes CLOCK_REALTIME moves forwards.

Smearing

The alternative is to have the servers absorb the leap second so the clients never see one. The server slews its served time across many hours, so no client ever receives a leap indicator and no client ever steps.

man 5 chrony.conf gives the recommended server configuration directly:

# On the internal NTP servers ONLY
leapsecmode slew
maxslewrate 1000
smoothtime 400 0.001024 leaponly

That smear starts at 00:00:00 UTC and takes about 17 hours to complete, with the served time deviating by a maximum of about half a second in the middle. The leaponly option is what makes the duration constant, which is what allows several servers to smear identically.

A client must never mix a smearing and a non-smearing source

A smearing server is deliberately not serving its best estimate of the true time. For the seventeen hours of the smear it is up to half a second away from UTC, on purpose.

A client configured with two smearing internal servers and one public pool line therefore sees a half-second disagreement. Depending on which side has the majority, chrony either marks the smearing servers as falsetickers and follows the public one — undoing the entire point of the smear and delivering the leap second the clients were being protected from — or marks the public server a falseticker and generates an alert nobody can explain.

If you smear, smear everywhere:

Every internal server smears, with identical smoothtime parameters.
No client has any non-smearing source configured. Remove the public pool lines from the fleet hosts; the internal servers are their only sources.
Clients of smearing servers must not set leapsectz, per the chrony documentation — that directive tells a host to expect a leap second, and a client of a smearing server must not expect one.

The same rule applies to cloud provider time services, several of which smear by default. A host using both a smearing cloud service and a non-smearing internal server has the problem already, on an ordinary day, waiting for the next leap second to express itself.

No leap second has been announced for the near future, and an international decision has been taken to stop inserting them at some point in the next decade. That is not a reason to leave the question undecided: the configuration is three lines, the consequences of a split fleet are a whole-estate incident, and the announcement period for a leap second is only about six months.

Monitoring from the server end

Client-side checks — chronyc tracking, the offset and sync-status alerts from linux-time-skew-operational-impact — tell you about one host. The server side tells you about the hierarchy.

# On an internal NTP server

# How much work is it doing, and is it dropping requests?
chronyc serverstats

# Which clients are actually using it? Empty output on a server
# you believe is serving the fleet is the finding.
sudo chronyc clients

# Are any of its own upstreams in a bad state?
chronyc sources -v
chronyc activity

chronyc clients is the one that catches the failure described in linux-chrony-architecture: allow is configured, chronyd is healthy, and the host packet filter is dropping UDP/123 so no client ever reaches it. The server looks perfect from its own side and has zero clients.

chronyc activity reports how many sources are online, offline and unresolved — a server whose four upstreams have quietly become one is a server about to become a falseticker for the whole fleet.