Skip to main content
RunBook Academy

← All break/fix scenarios in Proxmox VE

advancedPBS / backup~30 min

PBS backup job completes but restores produce corrupted files

Reported symptoms

  • ●Backup job completes with status "OK" in the PVE task log
  • ●Restore produces a VM that boots but has random I/O errors
  • ●Some restored files have wrong checksums
  • ●PBS datastore shows chunk count consistent with backup size

Evidence

  • · proxmox-backup-manager verify <snapshot> returns errors
  • · zfs status on PBS shows degraded pool (one disk failed silently)
  • · scrub finds checksum errors in the PBS datastore
  • · Last successful GC was 3 weeks ago
Diagnosis and resolutionclick to reveal

Root cause

The PBS datastore lived on a degraded ZFS pool. ZFS served the corrupted chunks because it had redundancy, but the corruption had spread across both copies due to a known ZFS issue when a vdev is missing for an extended period. The backup job itself wrote the already-corrupted blocks; the "OK" status is misleading because vzdump only checks that the write call returned 0, not that the data is durable.

Remediation

1. Stop all PBS operations and put the datastore in maintenance mode. 2. Replace the failed disk. 3. Resilver the pool: `zpool replace tank <failed-disk> <new-disk>`. 4. Run a full scrub: `zpool scrub tank`. 5. Verify each backup snapshot independently: `proxmox-backup-manager verify <vmid>/<snap> --repository main` 6. Mark any damaged snapshots for deletion via prune rules. 7. Resume backups only after scrub completes and verify returns OK. In a production environment, treat any backup that was written during the degraded period as untrusted. Re-run those backups from PVE.

Verification

- `zpool status` reports ONLINE with no errors - `zpool scrub` completes with 0 errors - `proxmox-backup-manager verify` returns OK on every snapshot - A test restore produces a file with matching checksum

Prevention

- Configure ZFS scrub schedule on the PBS host (weekly) - Add a daily check for `zpool status` to the PBS checklist - Configure PBS verify jobs and check their output - Set up ZFS email alerts (or Prometheus exporter) for vdev faults

PBS restores corrupt: silent ZFS degradation

Why this matters

The dangerous lesson here is that a backup that succeeds in writing is not necessarily a backup that succeeds in restoring. Always verify your backup chain end-to-end on a cadence.