Maintenance mode and drain - the safe cluster update · Linux

Maintenance mode and drain are the tools for safely updating cluster nodes. The cluster manager stops sending work to a node in maintenance; the application can be restarted without losing traffic.

Three different things, often called “maintenance”

Pacemaker has three controls here and they are not interchangeable. Picking the wrong one either drops the service you were trying to protect, or leaves the cluster blind while you work.

Control	Resources on the node	Cluster still monitors?	Use it when
`pcs node standby <node>`	Stopped, started elsewhere	Yes, everywhere	You are rebooting or patching the node
`pcs node maintenance <node>`	Left running, unmanaged	No, on that node	You are restarting the application in place
`pcs property set maintenance-mode=true`	Left running, unmanaged	No, anywhere; no fencing	You are editing the cluster stack itself

Standby - drain the node

# Stop this node's resources and start them elsewhere
pcs node standby node1

# Verify resources have moved
pcs status nodes

# Return the node to service
pcs node unstandby node1

Standby is the tool for node maintenance: reboots, kernel updates, hardware. The resources move, the cluster keeps managing and monitoring them, and failover still works while you are working.

Node maintenance - service the application in place

# Leave the resources running, but stop managing them here
pcs node maintenance node1

# Now restart the application without the cluster reacting
sudo systemctl restart myapp

# Hand it back to the cluster
pcs node unmaintenance node1

This is the one people reach for standby by mistake. If you only need to bounce or reconfigure an application in place, standby is the wrong tool: it stops the resource and relocates it, which is a failover you did not want and a client disruption you did not need. pcs node maintenance leaves the resource running and simply stops the cluster reacting to it.

An unmaintained node is a monitored node again

The moment you run pcs node unmaintenance, the cluster re-probes every resource on that node. If you left the application stopped, or left it in a state its resource agent scores as failed, recovery starts immediately - possibly a fence. Bring the service back up and confirm it healthy before handing the node back:

systemctl is-active myapp
curl -sf http://localhost/health
pcs node unmaintenance node1
pcs status --full          # no Failed Resource Actions

Cluster-wide maintenance mode

For cluster-wide maintenance:

# Cluster-wide maintenance (no resources moved)
pcs property set maintenance-mode=true

# Disable maintenance
pcs property set maintenance-mode=false

“Does not move resources” understates it. Maintenance mode does far more than that, and the difference is the whole reason it is not a substitute for standby.

maintenance-mode=true switches high availability off

Cluster-wide maintenance-mode=true marks every resource unmanaged. Pacemaker stops monitoring resources, will not restart or relocate anything on failure, and will not fence. pcs status still looks healthy, so the cluster reads as “up” while being a set of unmanaged servers.

That is also why it cannot be used to test failover. With resources unmanaged, pcs node standby moves nothing, so “verifying” that the resource landed on node2 is checking for something that structurally cannot happen. A team that runs that test believes failover works when it has never been exercised.

Use maintenance mode only for editing a running cluster’s configuration. Pair it with an explicit pcs property set maintenance-mode=false, and confirm before you walk away:

pcs property config maintenance-mode   # must report false or be unset
pcs status nodes                       # no node Standby, none Maintenance

To drain a node while the cluster keeps recovering and fencing, use pcs node standby. To service an application in place, use pcs node maintenance. maintenance-mode is the tool for changing the cluster’s own configuration, and nothing else.

Drain specific resources

To drain a specific resource:

# Move a single resource. This works by injecting a location
# constraint into the CIB - it is not a hint, it is a rule.
pcs resource move web node2

# Always look at what it created
pcs constraint location config --full | grep -A2 web

# Remove the constraint. THIS is the undo.
pcs resource clear web
pcs constraint location config --full | grep web   # must be gone

The resource moves. The other resources are unaffected.

`pcs resource unmove` does not exist, and the constraint outlives your session

Two things bite here.

First, pcs resource unmove is not a command. If you have it in a runbook, the runbook has never been executed. The undo is pcs resource clear <resource>.

Second, whether the constraint cleans itself up depends on which pcs you are standing in front of:

Command	Constraint
`pcs resource move` (pcs 0.11.3+)	Removed automatically once the resource is running in its new location
`pcs resource move` (older pcs)	Persists until `pcs resource clear`
`pcs resource move-with-constraint`	Persists by design
`pcs resource ban`	Persists by design

A forgotten -INFINITY constraint is a failover that silently will not happen. Everything looks healthy - until the night the resource needs to land on the node you banned six months ago and Pacemaker refuses, because you told it to.

Audit at the end of every maintenance window, on every cluster, regardless of which pcs you think you have:

pcs constraint location config --full

Anything named cli-ban- or cli-prefer- is a leftover from a move or a ban. Clear it.

Validate the drain

# Confirm node1 is listed under Standby and hosts nothing
pcs status nodes

# Confirm the node hosts ZERO resources (no output = drained)
sudo crm_mon -1 --output-as=text | awk '/Full List of Resources/,0' | grep -w node1

# Confirm the drain left no failures behind
sudo pcs status --full | sed -n '/Failed Resource Actions/,/^$/p'

# Confirm the service still works from the remaining nodes
curl -sf http://vip/health

Read pcs status nodes, not pcs status | grep node1. A drained node reports Online (standby), so a grep for the node name matches a healthy line and tells you nothing.

If the node has no resources, no failed actions were recorded, and the service is healthy, the drain worked.

Application-level drain

For graceful drain of in-flight requests:

# Send SIGTERM to the application
sudo systemctl stop myapp    # waits for in-flight to finish

Or have the unit drain itself on stop:

# /etc/systemd/system/myapp.service
[Service]
ExecStart=/usr/local/bin/myapp
ExecStop=/usr/local/bin/myapp --drain

The application handles SIGTERM, completes in-flight requests, then exits.

Bringing the node back

# Re-enable the node
pcs node unstandby node1

# Verify: the Standby list must be empty
pcs status nodes
pcs status

The node is back in the cluster. Resources may be re-balanced.

Standby is a node attribute in the CIB. It survives a reboot and the end of your maintenance window. Nothing clears it for you, so pair every standby with an unstandby and make the empty Standby line the exit criterion.