Fencing devices and agents - the practical implementation

Fencing devices are the hardware (or software) that forcibly stops a node. This lesson covers the common options and how to configure them.

Common fencing devices

For physical servers, IPMI/iLO/iDRAC is the standard. For VMs, the hypervisor. For shared storage, SCSI reservation or SBD.

IPMI-based fencing

IPMI is the standard for x86 servers. Each server has a baseboard management controller (BMC) that can power on/off the server independently of the OS.

Store the BMC password once, in a root-only file. Never pass it as -P <password>: arguments are readable by every local user in /proc/<pid>/cmdline and are written to shell history.

# Install ipmitool
sudo apt install ipmitool

# One-time: root-only credential file
sudo install -d -m 0700 /etc/ipmi
sudo install -m 0600 /dev/null /etc/ipmi/bmc.pw
read -rs -p 'BMC password: ' BMC_PW; echo
printf '%s' "$BMC_PW" | sudo tee /etc/ipmi/bmc.pw >/dev/null
unset BMC_PW

# Configure IPMI (one-time)
ipmitool -I lanplus -H <bmc-ip> -U admin -f /etc/ipmi/bmc.pw lan set 1 ipsrc static

# Test power status
ipmitool -I lanplus -H <bmc-ip> -U admin -f /etc/ipmi/bmc.pw chassis power status

# Power off (fencing)
ipmitool -I lanplus -H <bmc-ip> -U admin -f /etc/ipmi/bmc.pw chassis power off

# Power on (recover)
ipmitool -I lanplus -H <bmc-ip> -U admin -f /etc/ipmi/bmc.pw chassis power on

In Pacemaker, the equivalent problem is passwd=. That parameter writes the password into the CIB in clear text, where pcs stonith config, every CIB backup, and every support bundle will carry it. Use password_script= instead: the agent runs the script and reads the password from its standard output.

# One-time, on every node - any node may execute the fence
sudo install -d -m 0700 /etc/pacemaker
sudo install -m 0600 /dev/null /etc/pacemaker/bmc-pw
read -rs -p 'BMC password: ' BMC_PW; echo
printf '%s' "$BMC_PW" | sudo tee /etc/pacemaker/bmc-pw >/dev/null
unset BMC_PW

sudo tee /etc/pacemaker/bmc-pw.sh >/dev/null <<'EOF'
#!/bin/sh
# Command substitution strips the trailing newline, so a stray
# newline in the file cannot corrupt the password.
printf '%s' "$(cat /etc/pacemaker/bmc-pw)"
EOF
sudo chmod 0700 /etc/pacemaker/bmc-pw.sh

# Configure IPMI fencing
pcs stonith create mynode1 fence_ipmilan \
    pcmk_host_list="node1" \
    ip="10.0.0.100" \
    username="admin" \
    password_script="/etc/pacemaker/bmc-pw.sh" \
    lanplus=1

The cluster uses this to fence node1 when needed. ip= and username= are the current parameter names; ipaddr= and login= are deprecated aliases for the same thing.

iLO and iDRAC

For HP iLO and Dell iDRAC, similar agents exist. The same credential rule applies:

# HP iLO - match the agent to the iLO generation
pcs stonith create node1_ilo fence_ilo5 \
    pcmk_host_list="node1" \
    ip="10.0.0.100" \
    username="admin" \
    password_script="/etc/pacemaker/bmc-pw.sh"

# Dell iDRAC
pcs stonith create node1_idrac fence_idrac \
    pcmk_host_list="node1" \
    ip="10.0.0.100" \
    username="admin" \
    password_script="/etc/pacemaker/bmc-pw.sh"

Redfish-based fencing

Redfish is the DMTF replacement for IPMI: the same power control, exposed as an HTTPS REST API instead of RMCP+ over UDP. linux-redfish-and-ipmi-apis covers the API itself. In Pacemaker the agent is fence_redfish, shipped in fence-agents-redfish:

sudo dnf install fence-agents-redfish      # apt: fence-agents-redfish

pcs stonith create node1_redfish fence_redfish \
    pcmk_host_list="node1" \
    ip="10.0.0.100" \
    username="stonith" \
    password_script="/etc/pacemaker/bmc-pw.sh" \
    systems_uri="/redfish/v1/Systems/1" \
    ssl_insecure=0

Two parameters differ from the IPMI agents:

• systems_uri= is the path of the system resource inside the BMC. It is vendor-specific: /redfish/v1/Systems/System.Embedded.1 on Dell iDRAC, /redfish/v1/Systems/1 on many others. Read it from the BMC rather than guessing: curl -sk -u stonith: https://10.0.0.100/redfish/v1/Systems.
• ssl_insecure= controls certificate validation. Leave it at 0 and install the BMC certificate. ssl_insecure=1 makes fencing trust any certificate on that address, which defeats the point of fencing over an authenticated channel.

Verify the agent before trusting it, exactly as with IPMI:

fence_redfish --ip=10.0.0.100 --username=stonith \
  --password-script=/etc/pacemaker/bmc-pw.sh \
  --systems-uri=/redfish/v1/Systems/1 --action=status

Telling the agent which node is which

Every fence device needs to know how to translate a cluster node name into whatever the device calls that machine - an outlet number, a BMC address, a VM name. Two parameters do this, and a device configured without either is a device Pacemaker cannot use.

Hypervisor fencing almost always needs pcmk_host_map, because the cluster knows the node as node1 while vCenter or libvirt knows the same machine as vm-node1.

VM-based fencing

For VMs, fence via the hypervisor. The hypervisor credential is as powerful as the BMC one, so it goes in a script too.

# VMware over the SOAP API
pcs stonith create vmware_fence fence_vmware_soap \
    ip="vcenter.example.com" \
    username="stonith@vsphere.local" \
    password_script="/etc/pacemaker/vsphere-pw.sh" \
    ssl=1 ssl_insecure=0 \
    pcmk_host_map="node1:vm-node1;node2:vm-node2;node3:vm-node3" \
    pcmk_monitor_timeout=60s

# KVM/libvirt, driven over SSH to the hypervisor
pcs stonith create kvm_fence fence_virsh \
    ip="kvmhost.example.com" \
    username="stonith" \
    identity_file="/etc/pacemaker/id_stonith" \
    pcmk_host_map="node1:vm-node1;node2:vm-node2"

Verify the agent can see the targets before wiring it into the cluster. Run the agent directly:

# Ask the device what it can fence - the names on the left of
# this list are what pcmk_host_map must map to.
fence_vmware_soap --ip=vcenter.example.com \
    --username=stonith@vsphere.local \
    --password-script=/etc/pacemaker/vsphere-pw.sh \
    --ssl --action=list

fence_virsh --ip=kvmhost.example.com --username=stonith \
    --identity-file=/etc/pacemaker/id_stonith --action=list

# And confirm the parameter names the installed agent actually
# accepts, rather than trusting any document including this one:
fence_virsh -o metadata | grep '<parameter name'

Test fencing

For each fence device:

1. Configure the device.
2. Trigger a fencing manually.
3. Verify the node is actually stopped.
4. Verify the cluster can still operate.

# Test fencing manually
pcs stonith fence node1

The node should be powered off (or VM stopped). The cluster should continue with the remaining nodes.

Knowledge check