Designing around dependency failures - remove, cache, degrade, break glass

You have the register from linux-mapping-cluster-dependencies. Every row is something that can be gone tomorrow. This lesson is what to do about each row, and the strategies are in preference order for a reason: the first is permanent, the last is a person in a hurry at three in the morning.

1. Remove it from the critical path

The strongest fix is not to depend on the thing.

# Cluster peers resolved without DNS
cat /etc/hosts

192.0.2.11  node1 node1.cluster.example.com
192.0.2.12  node2 node2.cluster.example.com
192.0.2.13  node3 node3.cluster.example.com
192.0.2.51  node1-bmc
192.0.2.52  node2-bmc
192.0.2.53  node3-bmc

Three names are worth doing this for, and they are the three people forget:

• Cluster peers. Membership must not wait on a resolver.
• BMC addresses used by fence devices. Fencing is the last defence against corruption, and it runs during exactly the incidents where infrastructure is unreliable. Configure fence devices with addresses, or with names in /etc/hosts.
• Storage targets. An iSCSI portal or NFS server named by DNS makes the data path depend on name service.

The cost is real and worth stating: /etc/hosts entries are configuration that drifts. They belong in configuration management with a check that they match reality, or you have traded a dependency failure for a stale-address failure.

2. Cache, and fail static

Where the dependency cannot be removed, keep the last known good answer and keep serving it when the source is gone.

“Fail static” is the precise term and the important idea: when the source of truth is unreachable, continue with the last answer rather than with no answer. A resolver that returns SERVFAIL has converted an upstream outage into a local outage; one that serves a slightly stale record has not.

# unbound: keep answering from expired cache when upstream is unreachable
server:
    serve-expired: yes
    serve-expired-ttl: 86400

# /etc/sssd/sssd.conf - survive an LDAP outage
[domain/example.com]
cache_credentials = true
offline_credentials_expiration = 7
ldap_network_timeout = 5
ldap_opt_timeout = 6

cache_credentials lets people who have logged in before log in again while the directory is down. offline_credentials_expiration bounds that in days, which is the security decision - and it is a decision, not a default to accept.

3. Degrade deliberately

Decide in advance what the service does without the dependency, and make it do that on purpose rather than by accident.

• A service that cannot reach its identity provider serves anonymous read traffic and refuses writes.
• A cluster that cannot reach its metrics backend keeps running and drops metrics rather than blocking on the send.
• An application that cannot reach a non-essential downstream disables that feature and returns a clear error for it, not a timeout for everything.

The difference between deliberate degradation and accidental degradation is that the deliberate one is a code path someone wrote, tested and can describe. The accidental one is a timeout, and timeouts do not degrade gracefully - they queue.

4. Break glass

Every fleet needs at least one path that uses none of the shared dependencies:

• A local account on every node with a password in a physical safe or a sealed secret store, not in the directory.
• Its SSH key in /root/.ssh/authorized_keys, placed by configuration management, not resolved through LDAP.
• Console access through the BMC or a serial concentrator, reachable without the cluster network.
• A documented, rehearsed procedure for using them, because an unrehearsed break-glass path is a hypothesis.

# Verify the local path exists and is usable, before you need it
getent passwd -s files breakglass
sudo passwd -S breakglass
grep -E '^(passwd|sudoers):' /etc/nsswitch.conf

The nsswitch.conf check matters: if passwd: lists the directory before files, a directory that is slow rather than down makes even the local account take the NSS timeout to log in. Put files first.

Fail open or fail closed, per dependency

This is the decision that cannot be made once for the whole system. It is made per dependency, and the answers genuinely differ:

The authentication row and the break-glass section are in tension on purpose. Fail-closed authentication is correct and it locks the operators out during exactly the incident they need to fix. Break glass is how both can be true.

Bound every retry and every timeout

An unbounded wait is how a soft dependency becomes a hard one.

The canonical case: identity is down, nsswitch.conf consults the directory first, and every sudo, every ls -l and every SSH login blocks for the full NSS timeout on every node at once. Nothing has crashed. The service is still serving traffic. No operator can log in to find that out.

The rules:

• Every network call gets a timeout, and the timeout is shorter than the patience of whatever called it. A 30-second LDAP timeout inside a 10-second HTTP request is a guaranteed failure with a confusing error.
• Retries are bounded and backed off. Unbounded retry against a dead dependency is a denial of service you inflict on yourself, and on the dependency when it comes back.
• Fast failure beats slow success during an incident. A request that fails in 200 ms leaves the operator a working shell; one that hangs for 30 seconds does not.

Circular dependencies and the cold start

Every strategy above assumes the cluster is running. The failure mode that ignores all of them is the bootstrap deadlock:

The DNS server runs as a resource on the cluster.
The cluster resolves its peers by name.
Power is restored after a site outage.
  -> corosync cannot resolve its peers
  -> the cluster does not form
  -> the DNS resource never starts
  -> corosync still cannot resolve its peers

The same shape appears with a certificate authority hosted on the cluster it issues certificates for, a config management server that manages itself, and a monitoring system whose storage is the cluster it monitors.

None of these are detectable during normal operation. They are only visible from the question: if everything were off, in what order would it come up, and is that order acyclic?

Two rules keep it acyclic:

• A dependency of the cluster should not be hosted on the cluster. Where that is unavoidable - and it sometimes is - break the cycle at one point with a static answer: /etc/hosts for the peers, an address instead of a name, a certificate on local disk.
• Write the cold-start order down and rehearse it. The order is the deliverable. A recovery plan that assumes DNS is up is not a plan for the outage that took DNS down.

Knowledge check