Email alerting lab

This lab turns on email notifications for PVE events (HA moves, Ceph health, replication) and verifies they arrive.

Steps

1. Configure the SMTP relay on each node

On each PVE node, configure postfix as a smarthost:

apt install -y postfix
# During install, choose "Satellite system"
# Set the smarthost to your relay (e.g., smtp.gmail.com:587)

Or modify the existing postfix config:

postconf -e 'relayhost = [smtp.example.com]:587'
postconf -e 'smtp_sasl_auth_enable = yes'
postconf -e 'smtp_sasl_security_options = noanonymous'
postconf -e 'smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd'
postconf -e 'smtp_tls_security_level = encrypt'

cat > /etc/postfix/sasl_passwd << 'EOF'
[smtp.example.com]:587 alerts@example.com:app-password
EOF
postmap /etc/postfix/sasl_passwd
systemctl restart postfix

2. Configure PVE to send email

In the GUI: Datacenter → Options → Email from address.

Email from address: proxmox@example.com

Or via CLI:

pvesh set /cluster/options --email_from proxmox@example.com

3. Send a test email

In the GUI: Datacenter → Options → “Send test mail”.

Or via CLI:

pvesh create /cluster/notifications/endpoints/sendmailto \
  --name test \
  --mailto ops-team@example.com

4. Configure matchers and notification targets

# Create a notification target
pvesh create /cluster/notifications/endpoints/sendmailto \
  --name ops \
  --mailto ops-team@example.com

# Create a matcher for HA events
pvesh create /cluster/notifications/matchers/ha \
  --mode all --comment "HA events" \
  --target ops

# Create a matcher for Ceph warnings
pvesh create /cluster/notifications/matchers/ceph-warn \
  --mode all --comment "Ceph warnings" \
  --target ops

5. Verify a Ceph warning email

# Trigger a warning by stopping one OSD
ssh root@pve-01 systemctl stop ceph-osd@0

# Wait 5 minutes
# Check ops-team@example.com for an email with subject like "[PVE] Ceph: 1 OSD down"

Restore the OSD:

ssh root@pve-01 systemctl start ceph-osd@0

6. Verify an HA failover email

# Find the VM with HA enabled
ha-manager status
# Fence the node hosting the VM
ssh root@pve-01 'echo c > /proc/sysrq-trigger'
# Watch for the email

7. Customise severity thresholds

In the GUI: Datacenter → Notifications → Matchers → “Edit ha”.

HA events: include
Fencing: include
Resources: include

Verification

• Test email arrives within 1 minute
• Ceph warning email arrives within 5 minutes of OSD down
• HA email arrives within 5 minutes of fence
• Spam folder is empty (whitelist the sender)

Cleanup

# Stop postfix if it was only for this lab
systemctl disable --now postfix
# Remove the matchers
pvesh delete /cluster/notifications/matchers/ha
pvesh delete /cluster/notifications/matchers/ceph-warn
pvesh delete /cluster/notifications/endpoints/ops

Notes

• Use a dedicated sender domain with SPF/DKIM/DMARC set up to avoid the spam folder.
• For high-volume environments, route alerts to a dedicated notifications channel (Slack, PagerDuty) instead of email.
• Alert fatigue is real; tune matchers to send only on real incidents, not on routine status changes.