Mapping cluster dependencies - finding what you depend on

The previous three lessons each took one dependency and showed what its failure does. This one asks the prior question: how do you know what your cluster depends on at all?

The answer is not “ask the team”. A cluster acquires dependencies through package defaults, PAM stacks, resource agents and someone’s convenient hostname, and none of those get written down. They have to be read off the running system.

A cluster removes internal single points of failure and adds external ones

That is the trade the design made, whether or not anyone said it out loud. Three nodes means no single node failure takes the service down. It also means three machines that all resolve the same names, trust the same CA, authenticate against the same directory, and get their time from the same source.

The internal redundancy is visible and celebrated. The external dependencies are invisible until one of them fails on all three nodes in the same second - at which point the diagnosis is linux-failure-cluster-and-dependencies, and the lesson is that simultaneity was the clue.

Discovering them from the running system

Six places, each of which reliably hides something.

1. What the node is actually talking to

The most honest inventory is the set of connections the node holds in steady state:

# ss -tunp state established

Netid Recv-Q Send-Q  Local Address:Port    Peer Address:Port  Process
tcp        0      0  192.0.2.11:41022      192.0.2.30:389     users:(("sssd_be",pid=1188,fd=17))
tcp        0      0  192.0.2.11:55310      192.0.2.40:3260    users:(("iscsid",pid=901,fd=9))
tcp        0      0  192.0.2.11:39944      203.0.113.9:443    users:(("rsyslogd",pid=1042,fd=8))
udp        0      0  192.0.2.11:123        192.0.2.20:123     users:(("chronyd",pid=812,fd=6))

Run it on every node, over a full day, and compare. Connections that appear only during the backup window or only at certificate renewal time are exactly the ones missing from the documentation.

2. Name resolution and identity

# Which sources answer which lookups
grep -E '^(hosts|passwd|group|shadow|netgroup):' /etc/nsswitch.conf

# Where DNS actually goes
resolvectl status || cat /etc/resolv.conf

# How the cluster resolves its own peers - the critical one
getent hosts node1 node2 node3

If getent hosts for a peer answers from DNS rather than from /etc/hosts, cluster membership depends on DNS. That is a finding, and linux-dns-cluster-failure-impact explains what it costs.

3. Time

chronyc sources
chronyc tracking

Note whether every node points at the same single upstream. A shared time source is a shared dependency, and one that fails by drifting rather than by stopping - which is worse, because nothing alerts.

4. What systemd thinks has to exist first

# What must be running before this unit
systemctl list-dependencies --after pacemaker.service

# What breaks if this unit fails
systemctl list-dependencies --reverse corosync.service

Ordering directives encode dependencies that nobody documented elsewhere. A unit with After=nss-lookup.target is telling you it expects name resolution to work.

5. What the cluster configuration names

# Peer addresses or names in corosync
sudo corosync-cmapctl | grep nodelist

# Hostnames, URLs and paths embedded in resource agents
sudo pcs resource config
sudo pcs stonith config

Fence devices are the ones people forget. A fence_ipmilan pointing at a BMC hostname makes fencing - the last line of defence against corruption - depend on DNS.

6. Certificates and their chains

# Expiry of everything in a directory
for f in /etc/pki/tls/certs/*.crt; do
  printf '%s: ' "$f"; openssl x509 -noout -enddate -in "$f"
done

A certificate is a dependency on a CA, on a renewal process, and on a clock. All three can fail independently.

Classifying what you find

Three classes, and the class determines what you do about it:

The bootstrap class is where the surprises live. A dependency that is only needed at start is invisible during normal operation and absolutely required during a recovery - which is the moment you are most likely to be missing it. A cluster that cannot form without its config management server is a cluster that cannot be rebuilt during that server’s outage.

Recording it

The output of this exercise is a register with one row per dependency, and four columns that make it actionable:

Time to impact is the column that earns its place. It converts a list into a priority order: anything with immediate impact and a fleet-wide blast radius is where redundancy or removal is worth paying for, and anything measured in days can wait.

Blast radius answers the question that matters during an incident. A dependency listed as “all nodes” is one whose failure will not look like a dependency failure at all - it will look like the cluster died.

Two things to look for specifically

Shared fate. A dependency hosted on the cluster that depends on it. The DNS server running as a cluster resource, the certificate authority on a node, the config management server that manages itself. These work perfectly until a cold start, when nothing can begin because everything is waiting for something else. Circular dependencies are covered in linux-designing-around-dependency-failures.

Single instances behind a name. A resolver, an NTP server or an LDAP host that looks redundant because it is reached by a hostname, and resolves to one address. The name creates an impression of abstraction that the deployment does not have. getent hosts and chronyc sources settle it in one command each.

Knowledge check