Break/Fix: a node vanished from the cluster — network, fencing, or split brain?

A node disappears from the cluster; diagnose whether the cause is network, fencing, or split brain, and fix it. The discipline is to apply the methodology to a cluster failure.

Before you start

This drill partitions a live Corosync ring. On a cluster with fencing armed, that is precisely the event STONITH exists to answer: the survivors stop trusting the node they can no longer see, and they power-cycle it. Read the preconditions before you type anything.

Every one of these preconditions is required:

• A disposable three-node lab cluster. Nodes you can rebuild, carrying no real workload, on a network you own. Never a shared cluster, never production.
• Never a two-node cluster. The reason is below, and it is not a detail.
• Fencing disarmed for the duration. From a node that will stay in the majority partition, set stonith-enabled=false before injecting the failure, with sudo pcs property set stonith-enabled=false. If site policy forbids that, disable the target node’s fence device instead with sudo pcs stonith disable fence-node3. Either way, do it first: once the ring is broken, the isolated node can no longer receive a configuration change.
• Out-of-band console access to the target node — IPMI, iDRAC, iLO, or the hypervisor console. If your firewall rule turns out wider than you intended, that console is the only way back in.
• The current value recorded, so you can restore it in Task 4: pcs property config stonith-enabled.

Tasks

Task 1: Inject the failure

Blast radius: the disposable three-node lab cluster, and within it only the node you nominate as the target. The injection is a single firewall rule, which Task 4 removes again.

Confirm fencing is disarmed, then partition the target node:

# On a majority node: prove fencing is off before you break anything
pcs property config stonith-enabled

# On the target node only: drop inbound Corosync
sudo iptables -A INPUT -p udp --dport 5405 -j DROP

# Safety net: undo it automatically in 10 minutes, whatever happens
sudo systemd-run --on-active=10min --unit=corosync-drill-restore \
  /usr/sbin/iptables -D INPUT -p udp --dport 5405 -j DROP

The timer is not optional. You are deliberately cutting a node off from the cluster and you may also cut yourself off — a wider rule than intended, a management interface on the same path, an SSH session that dies mid-command. A self-restoring injection means the worst case is a ten-minute outage on a lab node rather than a trip to the console. When you finish Task 3 early, cancel it:

sudo systemctl stop corosync-drill-restore.timer

One rule, inbound only. That is enough. Knet keeps a link up only while a ping and its pong both arrive, so dropping what comes in also collapses the peers’ view of the link. The tempting second rule on OUTPUT buys no extra isolation and doubles the rollback: delete the INPUT rule, forget the OUTPUT rule, and the node still refuses to rejoin while you hunt a fault that is your own leftover. One rule in, one rule out, in a known order.

Check the port your ring actually uses before assuming 5405. It is mcastport in corosync.conf, and a cluster with a second link uses a second port:

sudo corosync-cmapctl | grep -E 'mcastport|linknumber'

Within a few seconds the survivors report the node lost. With fencing disarmed they stop there rather than resetting it, so the node stays up and reachable over SSH. That is what makes the rest of the drill possible — and it is the one difference between this exercise and the outage it models.

Task 2: Diagnose

Work from both sides of the partition. The isolated node and the survivors see different truths, and the difference is the diagnosis.

# Substitute your own values before running:
OTHER_NODE=node2

# Cluster membership and quorum
pcs status
pcs quorum status

# Corosync link state and current members
sudo corosync-cfgtool -s
sudo corosync-cmapctl | grep members

# Is corosync even listening, and on which address and port
sudo ss -lunp | grep corosync

# Is anything arriving from the peers
sudo timeout 10 tcpdump -ni any udp port 5405 -c 5

# General path to a survivor (ICMP only says the host is up)
ping -c 3 "$OTHER_NODE"

# Fencing configuration, device health, and what has already fired
pcs stonith status
pcs stonith config
sudo pcs stonith history show

Use pcs stonith status and pcs stonith config, not the old pcs stonith show, which pcs 0.11 removed.

Read the three causes apart:

• Network: the survivors still hold quorum, the isolated node reports its links down, and pcs stonith history show records no fencing action. Something on the path is dropping Corosync.
• Fencing: the node was reset. The survivors log a completed fence action, and the node’s own journal simply stops at the moment of the reset with no shutdown sequence.
• Split brain: both partitions claim quorum and both run the same resources. On a correctly built cluster this means quorum was configured away, most often two_node: 1 without a working fence device.

Task 3: Fix

Depending on the cause:

• Network: restore the path (unblock the port).
• STONITH: verify the device is reachable and its credentials are valid.
• Split brain: stop the cluster on the losing partition, then bring it back and let it rejoin. Never let both partitions keep running.

# On the target node: remove the one rule you added
sudo iptables -D INPUT -p udp --dport 5405 -j DROP

# Cancel the safety-net timer so it cannot fire later
sudo systemctl stop corosync-drill-restore.timer

# Prove nothing is left behind
sudo iptables -S INPUT | grep 5405 || echo "no residual Corosync rules"

# Watch the ring recover rather than guessing at a sleep
sudo corosync-cfgtool -s
pcs status

Rejoin is normally seconds, not a minute; the token timeout decides it. Poll pcs status until the node reads Online. If it does not rejoin, the rule you deleted was not the only block — check iptables -S in full, and check the survivors’ firewalls too.

Task 4: Restore the cluster

Do not leave the lab disarmed. Fencing off is a broken cluster that merely looks healthy, and a cluster that cannot fence cannot safely recover a resource.

# 1. Confirm all three nodes are back and quorate
pcs status
pcs quorum status

# 2. Re-arm fencing (or re-enable the device you disabled)
sudo pcs property set stonith-enabled=true
# sudo pcs stonith enable fence-node3

# 3. Confirm the property really took
pcs property config stonith-enabled

# 4. Confirm the fence devices are Started, which means their
#    recurring monitor is passing and the agent can reach the
#    hardware; a configured device that cannot be reached is
#    worse than no device, because the cluster trusts it
pcs stonith status

# 5. Clear any failed actions left by the drill
sudo pcs resource cleanup
sudo pcs stonith history cleanup

There is no dry run for fencing. pcs stonith fence node3 really resets node3, so it belongs in a deliberately scheduled test on the lab cluster, never in a restoration checklist. A passing monitor operation is the non-destructive evidence you have here.

The cluster is restored when all three nodes are Online, stonith-enabled is true, the fence devices are Started, and pcs status shows no failed actions.

Task 5: Document

TROUBLESHOOTING REPORT: Node Disappears
Date: 2026-08-09
Symptom: Cluster shows node3 OFFLINE
Root cause: Firewall blocked Corosync port 5405 inbound
Fix: Removed the iptables rule; node rejoined; fencing re-armed
Prevention: Documented Corosync ports in firewall rules
         Added monitoring for cluster membership
         Alert if stonith-enabled is ever false

Knowledge check