Monitoring cluster network and time - the leading indicators

Quorum loss and membership changes are the alerts everyone configures. They are also lagging indicators: by the time either fires, the partition has already happened and resources are already frozen.

The interconnect and the clock are where the warning was, and both fail in ways that produce no alert at all unless somebody asked for one specifically.

Link health is per link

# corosync-cfgtool -s

Local node ID 1, transport knet
LINK ID 0 udp
addr	= 192.0.2.11
status:
	nodeid:          1:	localhost
	nodeid:          2:	connected
	nodeid:          3:	connected
LINK ID 1 udp
addr	= 198.51.100.11
status:
	nodeid:          1:	localhost
	nodeid:          2:	disconnected
	nodeid:          3:	disconnected

The output format differs between corosync 2 and 3 and between transports, so parse what your version prints rather than the string in any document. What does not change is the shape of the problem: status is reported per link and per peer, and a cluster-level “is it quorate” check summarises all of that into one bit that stays green.

sudo corosync-cfgtool -s
sudo corosync-cfgtool -n

Encode the expected count rather than an abstract notion of health. A three-node cluster with two links has a known number of link-peer relationships that should be connected; anything less is the alert. The literal number belongs in the rule, next to the cluster it describes.

If you use the ClusterLabs exporter, check its metrics specification for the current ring and link series names before writing a rule against them - a PromQL expression naming a metric that no exporter produces is valid and permanently empty, which is the trap linux-cluster-monitoring-design documents.

Retransmits are the leading indicator

Before a link fails, it degrades. Corosync logs retransmits when the token does not complete its circuit and has to be resent, and a cluster that is retransmitting is a cluster approaching a failure it has not had yet.

# The signal, on every node
journalctl -u corosync --since '-24h' | grep -i 'retransmit'

# Rate over a shorter window, for a check script
journalctl -u corosync --since '-1h' | grep -ci 'retransmit'

A steady low rate under load is not necessarily alarming; a rate that is rising, or that appears at a new time of day, is the thing to act on. Correlate it with what else happens at that hour - the backup window, a batch job, a snapshot - because the cause is usually that something started sharing the interconnect with corosync.

That correlation is also the argument for monitoring interface counters on the cluster links specifically, rather than only in aggregate:

ip -s -s link show dev eth1

Errors, drops and a growing queue on the interconnect are a capacity problem that presents as a cluster problem.

Time: compare the nodes to each other

Every guide says to monitor NTP. Most monitor the wrong thing.

# chronyc -n tracking

Reference ID    : C0000214 (192.0.2.20)
Stratum         : 3
Ref time (UTC)  : Tue Aug 11 09:14:22 2026
System time     : 0.000284712 seconds slow of NTP time
Last offset     : -0.000091043 seconds
RMS offset      : 0.000210776 seconds
Frequency       : 12.083 ppm slow
Skew            : 0.094 ppm
Root delay      : 0.002418 seconds
Root dispersion : 0.001204 seconds
Leap status     : Normal

That output says node1 agrees with its upstream. It says nothing about whether node1 agrees with node2, and relative skew between nodes is what breaks a cluster:

• Kerberos rejects authentication beyond its tolerance, conventionally five minutes, measured between the two machines involved.
• TLS certificate validity is evaluated against each node’s own clock, so a certificate can be valid on one node and not yet valid on another.
• Cross-node log correlation becomes unreliable, and correlating timestamps across nodes is the primary technique for diagnosing a cluster incident.

Two nodes each three seconds off in opposite directions are six seconds apart while both report themselves synchronised. No per-node check catches that, because per-node is the wrong frame.

# The cluster-level question, from a management host
for n in node1 node2 node3; do
  printf '%s: ' "$n"
  ssh "$n" "chronyc -n tracking | awk -F': ' '/^Last offset/ {print \$2}'"
done

The useful alert is on the spread of those values, not on any one of them. Bound it well below the tightest consumer - if Kerberos tolerates five minutes, alert at a few seconds, so there is time to act before anything actually breaks.

The -n is deliberate: it stops chronyc resolving server addresses, so the time check does not itself depend on DNS. During a DNS outage, a monitoring command that hangs on resolution is a monitoring command that has joined the incident.

Two more signals from the same output:

• Leap status: Not synchronised is the state that matters. It means chrony has lost confidence in its sources and the clock is free-running. node_timex_sync_status from the node_exporter timex collector expresses the same thing as a metric.
• A stratum that has climbed means the upstream chain degraded, which is a warning before the offset moves.

# a node is not synchronised
node_timex_sync_status == 0

# absolute offset beyond a few seconds - tighten to your tightest consumer
abs(node_timex_offset_seconds) > 2

Which lane

The pattern is the same one the part has used throughout: the signal that the service is failing pages, the signal that the protection against failure is gone raises a ticket, and nothing is left with no lane at all.

Knowledge check