Cluster IR: node unreachable - the most common incident

A node-unreachable incident is the most common cluster event. This lesson covers the diagnostic, the recovery, and the prevention.

Symptoms

• Cluster shows one node as OFFLINE.
• Resources have migrated to other nodes (or not, if quorum is lost).
• The unreachable node is not responding to ping, SSH, or the management network.

Diagnostic

The BMC checks below read the password from /etc/ipmi/bmc.pw, a mode 0600 file that root owns. Never use -P <password> during an incident: the whole command goes into /proc/<pid>/cmdline and into the shell history of whatever jump host the on-call engineer is on.

# Substitute your own values before running:
UNREACHABLE_NODE=node3
BMC_IP=192.0.2.50

# Check the node from other nodes
ping -c 3 "$UNREACHABLE_NODE"

# Check via management network (BMC)
ipmitool -I lanplus -H "$BMC_IP" -U admin -f /etc/ipmi/bmc.pw chassis power status

# Check Corosync
corosync-cmapctl | grep members

# Check Pacemaker
pcs status

The diagnostic:

• No response to ping, SSH, BMC: the host is down (hardware, OS, or power).
• Responds to BMC but not to OS: the OS is hung (kernel panic, resource exhaustion, network issue).
• Responds to everything but is missing from cluster: the cluster communication is broken (Corosync, firewall).

Recovery

For the most common case (host down):

# Substitute your own values before running:
BMC_IP=192.0.2.50
NODE=node2

# 1. Confirm the host is down
ipmitool -I lanplus -H "$BMC_IP" -U admin -f /etc/ipmi/bmc.pw chassis power status
# Returns: "Chassis Power is off" or similar

# 2. Confirm the cluster has finished with it BEFORE powering it on.
#    Fencing must show completed, not pending, and the resources
#    must already be running somewhere else.
sudo pcs quorum status
sudo pcs stonith history show "$NODE"
sudo pcs status

# 3. Only once the node is fenced and its resources have recovered
#    elsewhere, bring the hardware back
ipmitool -I lanplus -H "$BMC_IP" -U admin -f /etc/ipmi/bmc.pw chassis power on

# 4. Wait for the host to come back
sleep 120

# 5. Check if it rejoined the cluster
pcs status

For the OS-hung case, the node is alive enough to still hold locks and still write to shared storage. Ask the cluster to fence it; do not reach for the BMC yourself:

# Substitute your own values before running:
NODE=node2

# 1. Cluster-initiated fence. The cluster records the fence, then
#    releases the node's resources for recovery elsewhere.
sudo pcs stonith fence "$NODE"

# 2. Confirm the fence completed and resources recovered
sudo pcs stonith history show "$NODE"
sudo pcs status

# 3. Wait for the host, then verify it rejoined
sleep 120
pcs status

For the cluster-communication-broken case the node is healthy and running resources. Restarting Corosync underneath it drops and re-adds membership, which its peers may read as a node failure and fence. Release the resources first:

# Substitute your own values before running:
NODE=node2
THIS_NODE=node1

# 1. Check the firewall against ALL the cluster ports, not just
#    corosync. 5404-5412/UDP is corosync/knet (a port per link),
#    2224/TCP is pcsd, 3121/TCP is pacemaker_remote, 21064/TCP is
#    DLM, and IP protocol 112 is VRRP if keepalived is in play.
#    A ruleset that permits only 5405 keeps membership alive while
#    every remote pcs operation times out.
sudo nft list ruleset | grep -E '5404|5405|2224|3121|21064'
sudo iptables -L -n

# Confirm what is bound locally, and reach it from the PEER -
# nc -z against UDP reports success whenever no ICMP comes back,
# which a DROP rule guarantees, so it is not evidence on its own.
sudo ss -lntup | grep -E 'corosync|pcsd|pacemaker'
corosync-cfgtool -s                      # per-link status
sudo timeout 10 tcpdump -ni any udp portrange 5404-5412 -c 5

# 2. The cluster must be quorate before you touch the membership
#    layer. If it is not, fix quorum first - restarting Corosync
#    from a non-quorate partition makes the outage larger.
sudo pcs quorum status

# 3. Release this node's resources so a membership flap cannot
#    take a live service down with it
sudo pcs node standby "$THIS_NODE"
sudo pcs status                 # confirm resources have moved

# 4. Now restart Corosync
sudo systemctl restart corosync

# 5. Return the node to service and verify
sudo pcs node unstandby "$THIS_NODE"
pcs status

Prevention

• Monitor host health (IPMI, node_exporter).
• Alert on host offline.
• Test BMC access regularly.
• Stage the BMC credential file on every jump host before the incident, so nobody types a password mid-incident.
• Verify network redundancy (dual NICs, dual switches).
• Test the fence device itself, not just the BMC login: pcs stonith status and a scheduled pcs stonith fence against a drained node. An untested fence agent is discovered during the incident it was meant to resolve.
• Deploy a quorum device (corosync-qdevice) on two-node clusters, or set wait_for_all, so a membership flap cannot produce two survivors.
• Document the recovery procedure.

Knowledge check