Hookscripts, startup order and resource pools

Three mechanisms sit around the edges of a guest’s life: something that runs when it starts and stops, something that decides in what order guests start and stop, and something that says who is allowed to touch it. Each is underused, and each shows up as a gap at the same moment — after an unplanned power event, when a cluster comes back in an order nobody chose and somebody has to work out which guest to start next.

Hookscripts

The hookscript option attaches an executable to a guest. From the documentation it is a “script that will be executed during various steps in the vms lifetime”, and the container chapter adds where to find the reference implementation:

You can add a hook script to containers with the config property hookscript, and it will be called during various phases of the guests lifetime. For an example and documentation see the example script under /usr/share/pve-docs/examples/guest-example-hookscript.pl.

That example script is the specification. Read it on your own node before you write anything, because it is the authoritative statement of which phases exist on the version you are running:

set -euo pipefail

sed -n '1,80p' /usr/share/pve-docs/examples/guest-example-hookscript.pl

# The phases the example handles, extracted:
grep -oE "'[a-z-]+'" /usr/share/pve-docs/examples/guest-example-hookscript.pl | sort -u

The script is called with the guest ID and the phase name as arguments, and it is invoked around the start and stop of the guest — before and after each. The useful property, and the reason it is worth using at all, is what happens when it fails.

set -euo pipefail

# /etc/pve is replicated across the cluster, so a snippet stored there is
# present on every node - which matters for a guest that can migrate.
pvesm set local --content backup,iso,vztmpl,snippets

install -d -m 0755 /var/lib/vz/snippets
cp /usr/share/pve-docs/examples/guest-example-hookscript.pl \
 /var/lib/vz/snippets/guest-hook.pl
chmod +x /var/lib/vz/snippets/guest-hook.pl

qm set 118 --hookscript local:snippets/guest-hook.pl

qm config 118 | grep hookscript

set -euo pipefail

cat > /var/lib/vz/snippets/require-nfs.sh <<'HOOK'
#!/usr/bin/env bash
# Called as: require-nfs.sh <vmid> <phase>
set -euo pipefail
VMID="$1"
PHASE="$2"
MOUNT=/mnt/pve/shared-data
DEADLINE=$(( $(date +%s) + 60 ))

log() { logger -t "pve-hook[$VMID]" -- "$*"; }

case "$PHASE" in
pre-start)
  while ! mountpoint -q "$MOUNT"; do
    if [ "$(date +%s)" -ge "$DEADLINE" ]; then
      log "refusing to start: $MOUNT not mounted after 60s"
      exit 1
    fi
    sleep 2
  done
  log "precondition met, allowing start"
  ;;
post-stop)
  log "guest stopped"
  ;;
*)
  log "phase $PHASE: nothing to do"
  ;;
esac
exit 0
HOOK

chmod +x /var/lib/vz/snippets/require-nfs.sh
qm set 118 --hookscript local:snippets/require-nfs.sh

Note the *) branch. A hookscript is invoked for every phase, including ones you did not plan for and ones a future version may add; a script that does not handle an unknown phase gracefully will fail on it, and with set -e that means a guest that will not start after an upgrade.

Startup and shutdown order

set -euo pipefail

qm set 101 --onboot 1 --startup order=1,up=60      # database
qm set 102 --onboot 1 --startup order=2,up=30      # application
qm set 103 --onboot 1 --startup order=3            # web front end

qm config 101 | grep -E '^(onboot|startup):'

The documented semantics:

Startup and shutdown behavior. Order is a non-negative number defining the general startup order. Shutdown in done with reverse ordering. Additionally you can set the up or down delay in seconds, which specifies a delay to wait before the next VM is started or stopped.

And, stated plainly in the same documentation set: we use the reverse startup order for shutdown, so a machine with a start order of 1 would be the last to be shut down.

Two properties that catch people:

up is a delay, not a health check. up=60 waits sixty seconds and then starts the next guest. It does not verify that anything came up. If your database takes ninety seconds after a dirty shutdown, the application starts into a database that is not listening. A guest that must not start early needs either a generous delay or a pre-start hook that actually checks.

Ordering is per node. startall on a node starts that node’s guests in that node’s order. A cluster power-up starts several nodes concurrently, each running its own ordered sequence, so a guest on node 2 can start before a guest on node 1 with a lower order number. Cross-node dependency ordering is not something the startup option provides.

set -euo pipefail

printf '%-6s %-28s %-7s %s\n' VMID NAME ONBOOT STARTUP
for vmid in $(qm list | awk 'NR>1 {print $1}'); do
cfg=$(qm config "$vmid")
name=$(printf '%s' "$cfg" | awk -F': ' '/^name:/ {print $2}')
onboot=$(printf '%s' "$cfg" | awk -F': ' '/^onboot:/ {print $2}')
startup=$(printf '%s' "$cfg" | awk -F': ' '/^startup:/ {print $2}')
[ -n "$onboot" ] || onboot=0
[ -n "$startup" ] || startup=UNORDERED
printf '%-6s %-28s %-7s %s\n' "$vmid" "$name" "$onboot" "$startup"
done

Resource pools

A pool looks like a folder in the interface. It is actually an ACL path, and that is the whole reason to use one.

set -euo pipefail

pveum pool add team-payments --comment 'Payments team workloads'

pveum pool modify team-payments --vms 210,211,212 --storage ceph-vm

# Grant on the pool path, not on each guest.
pveum acl modify /pool/team-payments \
--roles PVEVMUser --users alice@pve

pveum pool list
pveum user permissions alice@pve --path /pool/team-payments

Knowledge check