PBS troubleshooting: datastore issues, chunk corruption, and restore failures

PBS is the last line of defence for data loss. When PBS itself has problems, you discover them at the worst possible time — during a restore. This lesson covers how to diagnose and fix the common PBS failure modes.

PBS diagnostic commands

Every PBS issue starts with these commands:

# Status overview
proxmox-backup-manager status

# Datastore details
proxmox-backup-manager datastore list
proxmox-backup-manager datastore show main

# Recent tasks (the task log is the equivalent of PVE\'s)
proxmox-backup-manager task list --datastore main --limit 20
proxmox-backup-manager task show &lt;task-upid&gt;

# Verify job status
proxmox-backup-manager verify-job list

# Connected clients / namespaces
proxmox-backup-manager user list

For deep diagnosis, examine the PBS service logs:

journalctl -u proxmox-backup-proxy -u proxmox-backup --since '1 hour ago'
# Look for: errors, warnings, connection failures

Datastore full

The most common PBS failure. PBS stops accepting writes when datastore reaches 95% (configurable). Backups fail.

# Confirm
df -h /var/lib/vz/backups/main  # Or wherever the datastore is

# Find what\'s taking space
du -sh /var/lib/vz/backups/main/* | sort -h | tail -20

# Two paths forward:
# 1. Prune old backups (delete garbage)
proxmox-backup-manager prune datastore main \
  --keep-daily 7 --keep-weekly 4 --keep-monthly 6

# 2. Add storage (extend the underlying filesystem)
# For ZFS: zpool add tank /dev/sdX
# For ext4: lvextend + resize2fs
# For directory: add a new disk and mount

The right answer is to monitor and alert before the datastore fills. A PBS at 70% triggers a warning; at 85%, a page.

Garbage collection stalls

PBS garbage collection (GC) reclaims space from deleted chunks. Stalled GC means deleted backups don’t actually free space.

# Check GC status
proxmox-backup-manager task list --datastore main --type garbage_collection --limit 5

# Force a GC run
proxmox-backup-manager garbage-collection start main

Common GC issues:

Datastore is read-only. GC can’t write the index. Check mount | grep datastore.
Insufficient disk space during GC. GC needs space for temporary operations. Free some space first.
Long-running GC. GC on a multi-TB datastore can take hours. This is normal; schedule for off-peak.

# GC schedule — Sunday at 04:00
proxmox-backup-manager datastore update main \
  --gc-schedule 'Sun 04:00'

Chunk corruption

PBS verify jobs detect corruption by comparing chunk checksums against stored values. A “verify error” is a real signal.

# See verify errors
proxmox-backup-manager verify-job show weekly-verify

# Find which chunks are corrupt
proxmox-backup-manager verify-job run weekly-verify
proxmox-backup-manager task show &lt;latest-verify-task&gt;

# Show details of a specific corrupt snapshot
proxmox-backup-manager snapshot verify &lt;snapshot&gt; --datastore main

Causes and responses:

Cause	Response
Disk bit rot (ZFS without redundancy)	Move PBS to redundant storage
Disk controller failure	Replace controller, restore from sync
Network corruption during backup	Re-run the backup
Tape or backup target issue	Re-run, switch targets
Hardware failure of the underlying storage	Restore from sync job

If corruption is persistent, the underlying storage is unreliable. PBS’s value is that you can detect corruption; without redundancy in the underlying storage, detection is all you get.

Restore failures

A restore that fails is the worst possible moment to discover a PBS problem. Two common scenarios:

“No such snapshot”

# TASK ERROR: snapshot 'host/vm/2024-01-15T03:00:00Z' does not exist

The snapshot is gone. Either:

Pruning removed it (check prune schedule)
The sync job to this PBS hasn’t completed (no remote snapshot)
Manual cleanup deleted it

Fix: list available snapshots and use one that exists:

proxmox-backup-manager snapshot list &lt;vmid&gt; --repo &lt;pbs&gt;

“Missing chunks”

# TASK ERROR: download failed: chunk XYZ not found

A chunk is missing from the datastore. Either:

Garbage collection ran too aggressively (the chunk was deleted but the snapshot still references it)
The chunk store lost data (disk failure)
A backup target wasn’t fully synced

Fix: restore from a different snapshot, or restore from a different PBS that’s synced from the primary.

Performance problems

PBS performance degrades over time. Common causes:

Slow prune

Pruning iterates every chunk in the datastore. For a 10 TB datastore with millions of chunks, prune takes hours.

# Schedule prune for off-peak
proxmox-backup-manager datastore update main \
  --prune-schedule 'daily 02:00' \
  --keep-daily 7 --keep-weekly 4 --keep-monthly 6

# Monitor prune duration
proxmox-backup-manager task list --datastore main --type prune --limit 5

Slow GC

Same — GC iterates every chunk. A 10 TB datastore can take 12+ hours to GC. Schedule for weekends.

Slow backup throughput

The bottleneck is usually:

PBS storage — check iostat -dx 1 on the PBS host. If await is high, the storage is the bottleneck.
Network — check between PVE and PBS. The backup stream requires sustained throughput.
PBS CPU — PBS chunks and verifies on the CPU. Compression (zstd) is CPU-intensive. Watch top on the PBS host.

High chunk-store fragmentation

PBS stores chunks in a content-addressable store. Over time, the store becomes fragmented. Restores become slow because random I/O is slower than sequential.

The fix is to migrate to a new datastore:

# Create a new datastore on fresh storage
proxmox-backup-manager datastore add new-main /backup/new-main --chunkstore zstd

# Sync from old to new (using PBS sync jobs)
proxmox-backup-manager sync-job create old-to-new \
  --remote old-pbs --store-name main --store-name-new new-main

# Once synced, redirect new backups to new-main
# And decommission the old datastore

This is a maintenance window operation. Plan for it.

Remote PBS sync issues

PBS can sync to/from other PBS instances. Sync failures are common.

# List sync jobs
proxmox-backup-manager sync-job list

# Run a sync manually
proxmox-backup-manager sync-job run site-a-to-b

# Show sync status
proxmox-backup-manager sync-job show site-a-to-b

Common issues:

Network between PBS instances is down. The remote PBS URL isn’t reachable. Test with curl.
Authentication failed. The user/password or fingerprint is wrong. Re-create the remote.
Disk full on remote. The remote PBS doesn’t have space. Free space or add storage.

Production considerations

PBS performance scales with chunk-store size. A 1 TB datastore is fast; a 100 TB datastore is slow. Plan PBS hardware based on target datastore size.
Multiple datastores. Use multiple PBS datastores for different retention tiers: a fast SSD datastore for daily backups, a slow HDD datastore for weekly/monthly archives.
PBS backups itself. PBS needs its own backup strategy: PBS sync to a second PBS, periodic disk snapshots, or cold archive export.

Common mistakes

No PBS monitoring. A full PBS that nobody noticed is the same as no PBS.
Sync from one PBS to itself. The sync job must be between different PBS instances, or it does nothing useful.
Verifying without reading the output. Verify errors that go unread are the same as not running verify.
Long prune runs during the day. A 12-hour prune running during peak hours impacts performance.

Key takeaways

Read PBS errors and respond — every verify error is a signal.
Monitor datastore space and GC progress.
Have a second PBS for sync, not just backups.
Run prune and GC during off-peak.