Backup Metrics (PBS)

The ransomware encrypted the cluster’s primary storage at 03:00 on a Tuesday. The on-call engineer pulled the disaster-recovery runbook and confirmed that backups existed - the dashboard said so, every job said “OK” for the past 60 days. They started the restore. Twenty minutes later the team discovered that the backups were encrypted blobs taken from a datastore that had filled up eight months earlier; the verification step had been turned off because it was “slow”; the last successful verified restore had been a test, three years prior. The “OK” was a lie. The customer-visible incident was recovery, not encryption.

This lesson is about backup metrics. Proxmox Backup Server (PBS) publishes a Prometheus-shaped surface at /api2/json/admin/metrics. The alerts that catch a recovery incident before it becomes a customer incident live there.

What it is

Proxmox Backup Server is a deduplicated, encrypted backup service that Proxmox VE talks to over its own REST API. PBS publishes:

• A Prometheus-format metrics endpoint at /api2/json/admin/ metrics on the PBS host (port 8007 by default).
• A second surface at /api2/json/admin/datastore/\{store\}/status that returns JSON with capacity, used, and chunk count.
• A third surface at /api2/json/admin/datastore/{store}/prune that reports job results; the prune schedule is the retention mechanism.

On the PVE side, pve_exporter exposes per-VM last-backup metadata through the cluster API call /cluster/backup. PVE and PBS speak the same protocol; the verification result of a restore on PBS is visible from the PVE side as the last-backup entry on the VM.

The two surfaces (PBS metrics and PVE last-backup) together answer the operational questions:

• Did the backup run?
• Did it succeed?
• Did it verify?
• When was it last successful?
• Is there room left on the datastore?
• Has the prune policy aged out backups faster than retention expects?

Why a sysadmin cares

Backups are the only reprieve from a primary storage incident. An alert that does not detect a backup failure for a day can leave the recovery window open on the day a customer actually needs it. The classic failure shape is:

• Backups report “OK” because the call ended without an error.
• The datastore filled up three weeks ago; every backup is rejected silently.
• The retention policy is purging backups before their target age because the prune logic has nothing else to do.
• The team discovers on the day they need the backup that the backup does not exist.

The metrics that catch this are the ones PBS does publish, but that almost no team wires by default. The lesson is to wire them and to alert on age, not on success alone.

How it works

   PVE node (host with qemu VMs)
       |
       +-- pvescheduler: backup job runs vzdump
       |
   PBS pull or push
       |
   Proxmox Backup Server
       |
   +-- proxmox-backup-proxy -> proxmox-backup (rust server)
       |
   +-- chunks on disk (deduplicated, encrypted)
       |
   +-- /api2/json/admin/metrics (Prometheus format)
       |
   Prometheus scrape
       |
   Alertmanager rule on age, capacity, failure count

The metrics endpoint is the truth. The web UI’s “OK” badge reconciles against the same datastore; the exporter’s surface is exactly what /admin/metrics returns, with no intermediary to lie.

Under the hood

PVE’s per-VM last-backup surface is the most useful signal for “how stale is the protection?”. Proxmox adds the metric pve_backup_last_age per-VM through the API call /cluster/backup. The age is the seconds since the last completed (not failed) backup, regardless of whether it was verified.

How to configure it

The PBS side:

# SEVERITY: CONFIGURATION
# Create a monitoring-only user on PBS with the DatastoreAudit
# role and an API token.
proxmox-backup-manager usercreate monitoring@pbs --comment 'Prometheus exporter'
proxmox-backup-manager userupdate monitoring@pbs --role DatastoreAudit
proxmox-backup-manager tokenadd monitoring@pbs prometheus \
  --comment 'pbs-exporter token'

The Prometheus scrape:

# /etc/prometheus/prometheus.yml (job entry)
# SEVERITY: CONFIGURATION
scrape_configs:
  - job_name: pbs
    metrics_path: /api2/json/admin/metrics
    scheme: https
    static_configs:
      - targets: ['pbs-01.example.lan']
        labels: {cluster: 'dr'}
    # PBS can take a few seconds to compute capacity and chunk
    # counts. 30s is comfortable.
    scrape_interval: 30s
    scrape_timeout: 25s
    # Token in the Authorization header. The PBS side accepts
    # the same PVEAPIToken format because the proxy expects it.
    authorization:
      type: PVEAPIToken
      credentials_file: /etc/prometheus/secrets/pbs.env

The PVE side from pve-exporter:

# /etc/pve-exporter/pve.yml
# SEVERITY: CONFIGURATION
default:
  api_url: https://pve-01.example.lan:8006/api2/json
  api_token: "monitoring-pve@pam!prometheus"
  api_token_value: "${PVE_EXPORTER_TOKEN}"
  modules:
    cluster: true
    node: true
    storage: true
    guests: true
    backup: true

The Prometheus rules:

# /etc/prometheus/rules/backup.rules.yml
# SEVERITY: CONFIGURATION (rule reload only)
groups:
- name: pbs-backup
  interval: 60s
  rules:
  # No successful backup in the last 25 hours for any VM. Pages
  # on the cause - a stale backup - not on a symptom.
  - alert: PVEBackupStale
    expr: pve_backup_last_age > 25 * 3600
    for: 30m
    labels:
      severity: page
      team: virtualization
      component: pve
    annotations:
      summary: 'No successful backup for VM {{ $labels.vmid }} in 25 hours'
      description: |
        Last successful backup of VM {{ $labels.vmid }} on node
        {{ $labels.node }} was {{ $value | humanizeDuration }} ago.
        Confirm the backup job status in PVE and the datastore
        capacity in PBS.
      runbook_url: 'https://runbooks.example.com/pbs/backup-stale'

  # PBS datastore above 85% used. Capacity hard-stop at 100% is
  # too late; 85% gives the prune schedule a chance.
  - alert: PBSDatastoreFillHigh
    expr: |
      proxmoxbackup_datastore_used_bytes
      / on(datastore) group_left()
      proxmoxbackup_datastore_total_bytes > 0.85
    for: 1h
    labels:
      severity: ticket
      team: virtualization
      component: pbs
    annotations:
      summary: 'PBS datastore {{ $labels.datastore }} above 85%'

  # PBS verify job has not run successfully in 7 days.
  - alert: PBSVerifyStale
    expr: time() - max_over_time(
      proxmoxbackup_verify_job_last_successful_run_timestamp_seconds[7d]
    ) > 7 * 24 * 3600
    for: 1h
    labels:
      severity: ticket
      team: virtualization
      component: pbs
    annotations:
      summary: 'PBS verify job stale for datastore {{ $labels.datastore }}'

  # Backup task failures climbing. Often the first signal of a
  # permission change, a credential expiry, or a network path.
  - alert: PBSBackupFailureRateHigh
    expr: |
      increase(proxmoxbackup_task_status_count{status="failed"}[1h]) > 5
    for: 30m
    labels:
      severity: page
      team: virtualization
      component: pbs
    annotations:
      summary: 'PBS task failures rising on datastore {{ $labels.datastore }}'

How to validate it

Top-level: scrape works:

# SEVERITY: READ-ONLY
curl -sk -H "Authorization: PVEAPIToken=..." \
  https://pbs-01.example.lan:8007/api2/json/admin/metrics \
  | grep -E '^proxmoxbackup_(datastore|chunk|verify|task)_' \
  | head

PVE side: per-VM last-backup age:

# SEVERITY: READ-ONLY
curl -sG http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=pve_backup_last_age' \
  | jq

Cross-check against the PBS task list:

# SEVERITY: READ-ONLY
# The CLI is the source of truth for "did this job run?".
proxmox-backup-manager task list
proxmox-backup-manager task log $(proxmox-backup-manager task list | grep backup | head -1 | awk '{print $NF}') | head

End-to-end drill: pull a backup, restore a tiny VM, validate. The restore drill is the only signal that proves “OK” actually means “recoverable”.

How it can fail

Six failure modes recur in production backup monitoring:

1. Datastore full without notice. The capacity check is on the PBS side; the per-VM “did backup” stays true because PBS reports the chunk as “stored” but the actual write was rejected. Symptom: proxmoxbackup_datastore_used_bytes / total_bytes near 1; proxmoxbackup_task_status_count \{status="failed"\} climbing; per-VM last-backup dates stop updating.
2. Verification skipped. The verify-job schedule was set to weekly and the cluster was down for maintenance last week. The next successful verify was a month ago. Symptom: PBSVerifyStale fires; nobody acts; restore drill discovers a corrupt chunk.
3. Retention policy purges faster than expected. The prune schedule was edited during an emergency but never re-tightened. Symptom: proxmoxbackup_gc_* shows high chunk deletion; pve_backup_last_age is still green because one backup is left, but the customer needs 30 days.
4. Network timeout masquerading as success. A long timeout on the PVE side reports the job as failed but as “completed with error”; the alert logic does not distinguish. Symptom: proxmoxbackup_task_status_count{status="error"} climbing; the task log shows timeout.
5. Credential expiry. The PBS user token was rotated during a key exercise; the PVE backup job was not updated. Symptom: every backup fails with 401; the per-VM last_backup_age jumps overnight.
6. Cross-datastore replication lag. PBS replicates to a second datastore; the replication runs but lags because of bandwidth contention. Symptom: proxmoxbackup_task_* shows OK but the secondary datastore’s datastore_used_bytes is far lower than the primary’s.

How to troubleshoot it

The order is: did the job run, did it succeed, did it verify, was the prune correct.

1. Did the job run? pvesh get /cluster/backup --output-format json | jq on the PVE side; check status and start time.
2. Did it succeed? proxmox-backup-manager task list on the PBS side; look for status == OK and recent time.
3. Did it verify? proxmox-backup-manager verify-job list and the most recent verify status.
4. Was the prune correct? proxmox-backup-manager prune-job list; inspect the most recent prune output for “kept X / removed Y”.
5. Run a restore drill. Restore the smallest VM to a disconnected network and confirm boot. This is the only verification that proves the metrics are not lying.

Security implications

• PBS exposes datastore names, chunk counts, and used bytes. That is reconnaissance-grade information for an attacker planning exfiltration or destruction.
• The token used by the exporter must be DatastoreAudit. Never DatastoreAdmin; an attacker who can prune or delete backups can destroy the recovery path.
• PBS chunk encryption lives inside the server; the encryption passphrase is required to restore. Treat it as a credential. The exporter does not see the passphrase and should not be able to.
• Replication credentials live on the PBS server; rotate them on the same cadence as the PVE tokens.

Performance implications

• PBS scrape load is small. The /admin/metrics endpoint walks the in-memory datastore summary; latency is a few milliseconds for hundreds of thousands of chunks. Scrape at 30 seconds.
• Backup windows place load on the underlying storage. A daily backup of 200 VMs against an NFS datastore will compete with production IO. Schedule the backup window outside the production maintenance window; observe the underlying storage latency during the backup, not just at idle.
• The exporter itself is a small HTTP server bundled with PBS; it does not need a separate process.

Production guidance

• 3-2-1: three copies, two media, one offsite. PBS to local; PBS to offsite PBS via sync; one of those to a cloud bucket.
• Alert on age first. pve_backup_last_age > 25h is the rule; success alone is not enough.
• Alert on verification. PBSVerifyStale catches the silent verification-stops-running failure.
• Alert on capacity. 85% used is the warning, 95% is the page.
• Schedule quarterly restore drills. The metrics catch what the drills find; the drills find what the metrics would only see too late.

Verification

You should now be able to answer:

• What is the difference between backup success, backup verification, and backup restore-drill status?
• Why is pve_backup_last_age the alert to wire before backup_success_total?
• What is the storage capacity threshold at which a PBS datastore begins to refuse writes silently, and why does the alert belong at 85% rather than 100%?
• Why does an alert on proxmoxbackup_task_status_count \{status="failed"\} miss a ransomware-grade deletion?

Quiz