Webhooks and event-driven automation: react to cluster state changes

Polling the API for state changes is wasteful. PVE supports a notification system that calls out to webhooks when events happen, enabling truly reactive automation.

The notification matcher system

PVE’s notification system is rule-based:

• A matcher defines what events to react to (VM stopped, disk full, etc.)
• A target defines where to send the notification (webhook, email, Gotify)
• A rule combines a matcher and a target

# List existing rules
pvesh get /cluster/notifications/rules

# Add a webhook rule for VM state changes
pvesh create /cluster/notifications/matchers/vm-events \
  --comment "Match VM lifecycle events" \
  --type vm \
  --match-severity info,warning,error

pvesh create /cluster/notifications/endpoints/slack-webhook \
  --comment "Send alerts to Slack" \
  --type webhook \
  --url 'https://hooks.slack.com/services/XXX/YYY/ZZZ' \
  --method POST \
  --body '{"text": "{{message}}"}'

pvesh create /cluster/notifications/rules/vm-alerts \
  --comment "Send VM alerts to Slack" \
  --matcher vm-events \
  --target slack-webhook \
  --inhibit 0

The webhook body uses {{message}} and other placeholders that get substituted with the event details.

Webhook payload format

A PVE webhook posts JSON:

{
  "type": "vm",
  "vmid": 100,
  "node": "pve-01",
  "status": "stopped",
  "message": "VM 100 (web-01) stopped on node pve-01",
  "timestamp": "2024-01-15T12:34:56Z",
  "severity": "info",
  "fqdn": "pve-01.cluster.example.com"
}

The schema varies by event type. Document your consumer to handle multiple event types.

Building a webhook consumer

A simple Python webhook receiver:

#!/usr/bin/env python3
# webhook-receiver.py
from http.server import BaseHTTPRequestHandler, HTTPServer
import json
import subprocess
import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger('webhook-receiver')

class Handler(BaseHTTPRequestHandler):
    def do_POST(self):
        length = int(self.headers.get('Content-Length', 0))
        body = self.rfile.read(length)
        event = json.loads(body)

        logger.info(f"Event: {event}")

        # React to specific events
        if event.get('type') == 'vm' and event.get('status') == 'stopped':
            # Auto-restart a VM that went down unexpectedly
            self.auto_restart_vm(event)

        self.send_response(200)
        self.end_headers()
        self.wfile.write(b'OK')

    def auto_restart_vm(self, event):
        vmid = event['vmid']
        node = event['node']
        # Don\'t restart if there\'s an inhibit flag set
        if 'expected' in event.get('message', \').lower():
            logger.info(f"VM {vmid} stopped intentionally, not restarting")
            return
        # Restart the VM
        result = subprocess.run(
            ['pvesh', 'create', f'/nodes/{node}/qemu/{vmid}/status/start'],
            capture_output=True, text=True
        )
        logger.info(f"Restart VM {vmid}: {result.returncode}")

    def log_message(self, format, *args):
        # Quieter default logging
        pass

HTTPServer(('0.0.0.0', 8000), Handler).serve_forever()

# Run on a dedicated host behind a reverse proxy with TLS
nohup python3 webhook-receiver.py &

Event-driven patterns

Auto-remediation for HA-managed VMs

When an HA-managed VM stops unexpectedly, restart it:

if event['type'] == 'vm' and event['status'] == 'stopped':
    if 'ha-managed' in tags:
        subprocess.run(['ha-manager', 'start', f'vm:{vmid}'])

Slack alerts for critical events

import requests

def send_slack(channel, message):
    webhook_url = 'https://hooks.slack.com/services/...'
    requests.post(webhook_url, json={'channel': channel, 'text': message})

# In handler
if event['severity'] in ('error', 'critical'):
    send_slack('#proxmox-alerts', f":fire: {event['message']}")

Auto-scale from cluster metrics

# When cluster CPU >80% for 5 minutes, create a new VM from a template
if event.get('metric') == 'cpu_usage' and event['value'] > 80:
    # Create VM from web-tier template
    create_vm_from_template(template='web-tier', count=1)

Snapshot before risky operations

# When user runs qm stop on a critical VM, snapshot first
if event.get('action') == 'stop':
    vmid = event['vmid']
    subprocess.run(['pvesh', 'create', f'/nodes/{event["node"]}/qemu/{vmid}/snapshot',
                    '--snapname', 'pre-auto-stop', '--vmstate', '1'])

Avoiding the feedback loop

Event-driven automation creates a feedback loop risk. If your webhook handler restarts a VM, that restart fires another webhook, which restarts the VM again…

Solutions:

• Cooldown period: ignore repeat events for the same resource within X seconds.
• Source filter: ignore events that originated from your own automation (mark them in the message).
• State check before action: instead of reacting to “VM stopped”, check the actual state and decide whether to act.

import time

LAST_ACTION = {}

def with_cooldown(key, cooldown=300):
    """Return True if action is allowed (no recent action on this key)."""
    now = time.time()
    last = LAST_ACTION.get(key, 0)
    if now - last < cooldown:
        return False
    LAST_ACTION[key] = now
    return True

# In handler
if event['type'] == 'vm' and event['status'] == 'stopped':
    key = f"vm-stop-&#123;event['vmid']&#125;"
    if with_cooldown(key, cooldown=300):
        # Action allowed
        restart_vm(event['vmid'])

Production considerations

• HTTPS for webhooks. PVE sends webhook payloads over HTTPS; verify certificates in your consumer.
• Authentication. PVE webhooks include a signature header you can verify.
• Multiple consumers. Send the same event to multiple endpoints (Slack for visibility, your own handler for automation) for redundancy.
• Rate limits. PVE can fire many events during a cluster event. Webhook consumers should handle bursts gracefully.

Common mistakes

• No deduplication. A “VM stopped” event can fire multiple times. Always deduplicate.
• No idempotency. “Restart VM” is idempotent (a running VM is fine); “send email to user@example.com” is not.
• No retry on the receiver. Webhook delivery is at-least-once. Receivers should be idempotent and retry on internal failure.
• Auto-remediation without testing. Auto-restart loops can take a cluster down. Test the remediation in staging first.

Key takeaways

• PVE notifications drive event-driven automation.
• Build consumers with idempotency, deduplication, and cooldowns.
• Avoid feedback loops with state checks.
• Multiple consumers (Slack, automation) for the same event.

Knowledge check