Pacemaker resources and resource agents

Resource agents are the bridge between Pacemaker and the service. They are scripts that implement the operations Pacemaker calls. This lesson covers how they work and how to write a custom one.

The operations

A resource agent supports a set of operations:

• start: bring the resource up.
• stop: bring the resource down.
• monitor: check the resource is up. Returns 0 (OK) or non-zero (failed).
• meta-data: return information about the agent (used by pcs).
• validate-all: validate all parameters.

Some agents also support:

• promote: make this instance the primary (for master/slave).
• demote: make this instance the secondary.
• migrate: live-migrate to another node.

Standard agents

resource-agents package ships many standard agents. Agent names are case-sensitive - they are filenames on disk under /usr/lib/ocf/resource.d/heartbeat/ - so write them exactly as shown:

IPaddr2 and Filesystem really are capitalised that way. ipaddr2 and filesystem do not exist, and pcs will reject them at create time with “Agent not found”. Note also that systemd: is a different resource class, not an OCF agent - it has no provider field.

List agents:

pcs resource list
pcs resource describe apache

Use a standard agent

# Apache
pcs resource create web apache \
    configfile=/etc/apache2/apache2.conf \
    op monitor interval=30s

# IP address
pcs resource create vip ocf:heartbeat:IPaddr2 \
    ip=10.0.0.100 \
    op monitor interval=30s

# systemd service
pcs resource create nginx systemd:nginx \
    op monitor interval=30s

Write a custom agent

For a service without a standard agent, write a custom one. Three rules before a single line of code:

• Agents live in /usr/lib/ocf/resource.d/<provider>/. The provider is the directory name, so ocf:heartbeat:myapp resolves to /usr/lib/ocf/resource.d/heartbeat/myapp.
• OCF_ROOT is /usr/lib/ocf, and nothing else. The agent derives every other path from it.
• Source ocf-shellfuncs. Without it the OCF_* return-code names do not exist, and an agent that returns undefined variables is an agent that lies to the cluster.

#!/bin/bash
# /usr/lib/ocf/resource.d/heartbeat/myapp
#
# OCF resource agent for myapp.
# Install under resource.d/<provider>/, NOT under lib/heartbeat/.

# Pull in the OCF helpers. This is what defines OCF_SUCCESS,
# OCF_NOT_RUNNING, OCF_ERR_*, ocf_log and friends.
: "${OCF_FUNCTIONS_DIR=${OCF_ROOT}/lib/heartbeat}"
# shellcheck source=/dev/null
. "${OCF_FUNCTIONS_DIR}/ocf-shellfuncs"

# Pacemaker passes each configured parameter as OCF_RESKEY_<name>.
binary=${OCF_RESKEY_binary:-}

myapp_meta_data() {
    cat <<EOF
<?xml version="1.0"?>
<!DOCTYPE resource-agent SYSTEM "ra-api-1.dtd">
<resource-agent name="myapp" version="1.0">
    <version>1.0</version>
    <longdesc lang="en">MyApp resource agent</longdesc>
    <shortdesc lang="en">Manages myapp</shortdesc>
    <parameters>
        <parameter name="binary" required="1" unique="0">
            <longdesc lang="en">Path to myapp binary</longdesc>
            <shortdesc lang="en">Binary path</shortdesc>
            <content type="string" />
        </parameter>
    </parameters>
    <actions>
        <action name="start" timeout="30s" />
        <action name="stop" timeout="30s" />
        <action name="monitor" timeout="10s" interval="30s" />
        <action name="validate-all" timeout="10s" />
        <action name="meta-data" timeout="5s" />
    </actions>
</resource-agent>
EOF
}

myapp_validate() {
    if [ -z "$binary" ]; then
        ocf_exit_reason "binary parameter is not set"
        return "$OCF_ERR_CONFIGURED"
    fi
    if [ ! -x "$binary" ]; then
        ocf_exit_reason "binary %s is not executable" "$binary"
        return "$OCF_ERR_INSTALLED"
    fi
    return "$OCF_SUCCESS"
}

myapp_monitor() {
    # An unconfigured agent must NOT report healthy.
    [ -n "$binary" ] || return "$OCF_ERR_CONFIGURED"

    if pgrep -f -- "$binary" >/dev/null; then
        return "$OCF_SUCCESS"
    fi
    return "$OCF_NOT_RUNNING"
}

myapp_start() {
    myapp_validate || return $?
    myapp_monitor && return "$OCF_SUCCESS"   # already running

    "$binary" --daemon || return "$OCF_ERR_GENERIC"

    # start is not complete until monitor agrees
    while ! myapp_monitor; do
        sleep 1
    done
    return "$OCF_SUCCESS"
}

myapp_stop() {
    local waited=0 limit rc

    # Never let an empty pattern reach pkill.
    [ -n "$binary" ] || return "$OCF_ERR_CONFIGURED"

    myapp_monitor
    rc=$?
    [ "$rc" -eq "$OCF_NOT_RUNNING" ] && return "$OCF_SUCCESS"   # already stopped

    pkill -f -- "$binary"

    # Bounded wait, then escalate. Never wait past the op timeout:
    # Pacemaker treats a timed-out stop as a FAILED stop and fences.
    limit=$(( ${OCF_RESKEY_CRM_meta_timeout:-20000} / 1000 - 5 ))
    while myapp_monitor; do
        if [ "$waited" -ge "$limit" ]; then
            ocf_log warn "myapp did not exit on SIGTERM, sending SIGKILL"
            pkill -9 -f -- "$binary"
        fi
        sleep 1
        waited=$(( waited + 1 ))
    done
    return "$OCF_SUCCESS"
}

case "$1" in
    meta-data)    myapp_meta_data; exit "$OCF_SUCCESS" ;;
    start)        myapp_start ;;
    stop)         myapp_stop ;;
    monitor)      myapp_monitor ;;
    validate-all) myapp_validate ;;
    *)            exit "$OCF_ERR_UNIMPLEMENTED" ;;
esac
exit $?

Place in /usr/lib/ocf/resource.d/heartbeat/myapp, make executable, and test every action before the cluster ever calls one:

export OCF_ROOT=/usr/lib/ocf
export OCF_RESKEY_binary=/usr/local/bin/myapp
AGENT=/usr/lib/ocf/resource.d/heartbeat/myapp

# Each action, checking the exit code - that is the whole contract
$AGENT meta-data    >/dev/null; echo "meta-data=$?"    # 0
$AGENT validate-all;             echo "validate-all=$?" # 0
$AGENT monitor;                  echo "monitor=$?"      # 7 while stopped
$AGENT start;                    echo "start=$?"        # 0
$AGENT monitor;                  echo "monitor=$?"      # 0 while running
$AGENT start;                    echo "start=$?"        # 0 - must be idempotent
$AGENT stop;                     echo "stop=$?"         # 0
$AGENT stop;                     echo "stop=$?"         # 0 - already stopped is success

# Then let the official harness try to break it
ocf-tester -n myapp -o binary=/usr/local/bin/myapp "$AGENT"

Use in Pacemaker:

pcs resource create myapp ocf:heartbeat:myapp \
    binary=/usr/local/bin/myapp \
    op monitor interval=30s

Knowledge check