Prometheus Backup

The retention policy was set to 30 days. A typo turned it into 30 minutes. The Prometheus process compacted the WAL, the compactor merged the head, and the operator noticed 90 minutes later when the dashboards went blank. The on-call engineer pulled the runbook and reached the step “restore from backup”. The backup host had a directory full of prometheus-snapshot-2026-08-09.tgz files. The most recent was three weeks old. The snapshot job had been failing silently for three weeks because the IAM role on the backup host had expired.

This lesson is the Prometheus backup. The right shape is the admin-API snapshot, shipped to a versioned object store, on a schedule, with a quarterly restore drill that proves the copy can boot a working Prometheus.

What it is

A Prometheus backup is a copy of the TSDB that the admin API produced as a consistent snapshot of in-memory state, the WAL, and the on-disk chunks. The configuration is separate; it lives in Git. The telemetry is the snapshot. The two together let you boot a working Prometheus on a different host.

The components:

• The TSDB. Default path /prometheus. Contains chunks_head/, wal/, chunks/, and meta.json. The head is in memory; the WAL is the durability queue; the chunks are the merged blocks.
• The configuration. prometheus.yml, rule files, scrape configs. Lives in Git. The Git history is the backup.
• The admin API. POST /api/v1/admin/tsdb/snapshot produces a consistent snapshot in a sibling directory and returns its name. The snapshot is a directory of files; it can be tarred.

The discipline is to take the snapshot through the API, tar it, and ship the tar to a versioned object store. Anything else is a copy of a half-written block.

Why a sysadmin cares

Prometheus is the single most important observability component for an alert-driven team. Losing the TSDB means:

• Alert state for every firing alert is gone; the alerts re-fire from the rule evaluation, but the deduplication and grouping history is reset.
• Dashboard panels return empty for the duration of the gap.
• Recording rules are re-evaluated from retained source data; if the source was remote-written elsewhere, the rules produce results; if it was local-only, the rules produce nothing until the source catches up.
• Post-incident timelines lose the metric evidence for the affected window.

A working backup turns a host failure into an hour of restoration work. A broken backup turns the same failure into a customer-facing blackout.

How it works

   Prometheus process (running)
        |
        |  POST /api/v1/admin/tsdb/snapshot
        v
   snapshots/2026-08-14T120000Z-<id>/
        |
        |  contains: meta.json, wal/, chunks/, chunks_head/
        |  all consistent at the snapshot time
        v
   tar czf -> local staging file
        |
        v
   aws s3 cp --storage-class STANDARD_IA
        |
        v
   s3://prom-backup-<account>-<region>/
        versions/
            2026-08-14/prometheus.tar.gz  (current)
            2026-08-13/prometheus.tar.gz  (non-current)
            ...
        |
        |  lifecycle:
        |    noncurrent: expire after 30 days
        |    current: keep
        v
   Cross-region replication to
   s3://prom-backup-dr-<region>/
        |
        v
   Restore drill quarterly: download tar, untar, run prometheus,
   curl a known query.

The snapshot API takes a write lock on the head block and waits for any in-flight write to settle. The directory it returns is read-only and consistent. The tar operation that follows is on a stable directory; corruption from concurrent writes is impossible.

How to configure it

The Prometheus side — leave the snapshot path at the default and document it:

# /etc/default/prometheus
# SEVERITY: CONFIGURATION (restart)
# The data directory and the retention are flags, not
# prometheus.yml keys. Retention is a separate concern from
# backup. A shorter retention reduces the snapshot size; a
# longer retention improves the recovery window. Pick the
# retention first; the backup policy then protects that window.
ARGS="--storage.tsdb.path=/prometheus \
      --storage.tsdb.retention.time=30d \
      --storage.tsdb.retention.size=200GB \
      --web.enable-admin-api"

# /etc/prometheus/prometheus.yml
# SEVERITY: CONFIGURATION (reload)
global:
  scrape_interval: 30s
  evaluation_interval: 30s

The backup job — the right primitive is the admin API, not a file copy:

#!/usr/bin/env bash
# SEVERITY: SERVICE-IMPACT (Prometheus admin call is non-disruptive;
# the head write is paused for the snapshot creation only).
set -euo pipefail

PROM_URL="http://prometheus.internal:9090"
SNAP_ROOT="/var/lib/prometheus/snapshots"
BACKUP_BUCKET="s3://prom-backup-primary-${AWS_REGION}"
STAMP="$(date -u +%Y-%m-%dT%H%M%SZ)"

# 1. Take the snapshot. Prometheus returns the snapshot name.
SNAP_NAME=$(curl -fsS -X POST "${PROM_URL}/api/v1/admin/tsdb/snapshot" \
  | sed -E 's/.*"name":"([^"]+)".*/\1/')
echo "Snapshot: ${SNAP_NAME}"

# 2. Tar the snapshot. The snapshot directory is read-only.
SNAP_PATH="${SNAP_ROOT}/${SNAP_NAME}"
tar -C "${SNAP_ROOT}" -czf "/tmp/${SNAP_NAME}.tar.gz" "${SNAP_NAME}"

# 3. Verify the tar before shipping.
promtool tsdb analyze "/tmp/${SNAP_NAME}.tar.gz#$(tar tzf "/tmp/${SNAP_NAME}.tar.gz" | head -1)" \
  || { echo "Snapshot failed analysis"; exit 1; }

# 4. Ship to S3 with server-side encryption and STANDARD_IA storage
# class. Use the backup-account role.
AWS_PROFILE=prom-backup aws s3 cp \
  --storage-class STANDARD_IA \
  --sse aws:kms \
  --sse-kms-key-id "${BACKUP_KMS_KEY}" \
  "/tmp/${SNAP_NAME}.tar.gz" \
  "${BACKUP_BUCKET}/${STAMP}/prometheus.tar.gz"

# 5. Clean the local snapshot. The tar is in S3; the snapshot
# directory can be released.
rm -f "/tmp/${SNAP_NAME}.tar.gz"

# 6. Ask Prometheus to delete the snapshot entry.
curl -fsS -X DELETE \
  "${PROM_URL}/api/v1/admin/tsdb/snapshot?name=${SNAP_NAME}" \
  >/dev/null

The S3 lifecycle:

# CloudFormation / Terraform excerpt
# SEVERITY: CONFIGURATION
Resources:
  PromBackupBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Sub 'prom-backup-primary-${AWS::Region}'
      VersioningConfiguration:
        Status: Enabled
      LifecycleConfiguration:
        Rules:
          - Id: ExpireNonCurrent
            NoncurrentVersionExpiration:
              NoncurrentDays: 30
            Status: Enabled
          - Id: TransitionToGlacier
            Transitions:
              - StorageClass: GLACIER
                TransitionInDays: 90
            Status: Enabled
      PublicAccessBlockConfiguration:
        BlockPublicAcls: true
        BlockPublicPolicy: true
        IgnorePublicAcls: true
        RestrictPublicBuckets: true
      ReplicationConfiguration:
        Role: !GetAtt BackupReplicationRole.Arn
        Rules:
          - Id: ReplicateToDR
            Status: Enabled
            Prefix: ''
            Destination:
              Bucket: !Sub 'arn:aws:s3:::prom-backup-dr-${DRRegion}'
              StorageClass: STANDARD_IA

How to validate it

Top-level: the snapshot exists and the latest one is fresh.

# SEVERITY: READ-ONLY
# List the most recent three backups.
AWS_PROFILE=prom-backup aws s3 ls \
  s3://prom-backup-primary-${AWS_REGION}/ \
  --recursive | sort | tail -3
# Expected:
# 2026-08-14 12:00:01   52428800 2026-08-14T120001Z/prometheus.tar.gz
# 2026-08-13 12:00:02   52398848 2026-08-13T120002Z/prometheus.tar.gz
# 2026-08-12 12:00:01   52416512 2026-08-12T120001Z/prometheus.tar.gz

Mid-level: the snapshot tar is internally consistent.

# SEVERITY: READ-ONLY
TMP=$(mktemp -d)
AWS_PROFILE=prom-backup aws s3 cp \
  "s3://prom-backup-primary-${AWS_REGION}/$(\
    aws s3 ls s3://prom-backup-primary-${AWS_REGION}/ --recursive \
    | sort | tail -1 | awk '{print $4}'\
  )" - | tar -C "${TMP}" -xzf -
# Inspect the meta.json for the snapshot time and the block count.
cat "${TMP}/meta.json" | jq '{minTime, maxTime, ulid}'
# Expected (illustrative):
# {
#   "minTime": 1755096000000,
#   "maxTime": 1755178800004,
#   "ulid": "01H9X..."
# }

End-level: the restore drill produced a working Prometheus.

# SEVERITY: READ-ONLY
# The drill marker is written by the drill runbook only after the
# smoke test passes.
cat /var/backups/prometheus/drill/last-drill.txt
# Last successful restore drill of prometheus: 2026-05-14, served 30d of metrics via curl :9090/api/v1/query.

The drill runbook in skeleton form:

# SEVERITY: SERVICE-IMPACT (boots a Prometheus on a staging host)
STAGE=/opt/prom-drill
mkdir -p "${STAGE}/data"
LATEST=$(AWS_PROFILE=prom-backup aws s3 ls \
  s3://prom-backup-primary-${AWS_REGION}/ --recursive \
  | sort | tail -1 | awk '{print $4}')
AWS_PROFILE=prom-backup aws s3 cp \
  "s3://prom-backup-primary-${AWS_REGION}/${LATEST}" - \
  | tar -C "${STAGE}/data" -xzf -
docker run -d --name prom-drill \
  -p 9091:9090 \
  -v "${STAGE}/data:/prometheus" \
  -v "${STAGE}/prometheus.yml:/etc/prometheus/prometheus.yml" \
  prom/prometheus:v2.55.0 \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/prometheus
sleep 30
# Smoke test: a known series from the last 24h must respond.
curl -sG http://localhost:9091/api/v1/query \
  --data-urlencode 'query=up{job="node"}' \
  | jq '.data.result | length'
# Expected: a positive integer; 0 means the drill failed.

How it can fail

Five failure modes recur in Prometheus backup:

1. The cron job that copies /prometheus instead of taking a snapshot. The tar contains a half-written WAL. The restore fails to replay. Symptom: promtool tsdb analyze rejects the restored TSDB; the restored Prometheus exits on startup with error opening storage.
2. The IAM role on the backup host has expired. The aws s3 cp returns AccessDenied. The job exits non-zero; if the alerting on the job is missing, the failure is silent for weeks. Symptom: the S3 bucket has no new keys for the RPO window; the alerting rule fires after 26h.
3. The KMS key was disabled. The aws s3 cp with --sse-kms-key-id returns KMS.AccessDeniedException. The job fails. Symptom: the bucket has no new keys; the alert fires.
4. The Prometheus version was upgraded mid-cycle. A snapshot taken on 2.55.x is restored on 2.54.x (the staging host runs the pinned minor version). Symptom: the restored Prometheus logs unknown TSDB version and refuses to start.
5. The restore drill was never run. The drill marker file freezes; the alert is silenced; the snapshot format has been changing in the background. Symptom: a real recovery fails on the format mismatch that the drill would have caught.

How to troubleshoot it

The order is: does the copy exist, is the copy consistent, does the recovery path work.

1. Does the copy exist? aws s3 ls s3://prom-backup-primary-*/ filtered by date prefix. An empty or stale result means the job is failing.
2. Is the copy consistent? Download the most recent tar; run promtool tsdb analyze against the extracted directory. A failure here means the snapshot was taken incorrectly.
3. Does the recovery path work? Run the drill. If the drill has not been run in the policy window, schedule it before any other change.
4. Does the rule file match? promtool check rules against the rule directory restored from Git. The snapshot is the state; the rules are the configuration; both must match the snapshot time for the recovered instance to evaluate the same alerts as the source.

Security implications

• The snapshot tar contains every metric value, every label, every recording-rule result for the retention window. That is the full picture of the production telemetry.
• The backup bucket must use KMS encryption with a customer-managed key. The key is a separate credential, rotated independently of the bucket.
• The IAM role used by the backup host is the highest blast-radius role in the observability stack. Limit it to s3:PutObject, s3:GetObject, s3:ListBucket on the backup bucket only. No s3:Delete*. The lifecycle rule deletes on a schedule; the backup host does not delete.
• The admin API endpoint for the snapshot is the same endpoint an attacker would use to take a copy. Restrict it to the backup network; do not expose it on the public listener.
• The restore drill staging host holds a copy of the production TSDB. Treat it as production data; wipe on completion.

Performance implications

• The snapshot API pauses the head write for the duration of the hard-link creation. On a busy cluster with high ingestion, this is a few hundred milliseconds. Schedule for off-peak.
• The tar compression of a 50 GB TSDB takes 3-5 minutes on a modern host. The S3 PUT of the compressed tar takes 10-30 seconds on a 1 Gbit link. The total snapshot window is dominated by the tar and the PUT, not by the API.
• The backup host sees CPU pressure during compression and network pressure during upload. A dedicated backup host sized for the largest realistic snapshot is the right shape.
• Lifecycle transitions to Glacier reduce storage cost; restore time from Glacier is hours. Keep the last 30 days on STANDARD_IA for fast drill restores; transition older to Glacier.
• A daily snapshot of a 50 GB TSDB over 365 days is roughly 18 TB compressed and versioned. The cost is real; the offsite copy doubles it. Plan before deploying.

Production guidance

• Snapshot via the admin API. Never copy /prometheus directly.
• Ship to a versioned bucket with cross-region replication.
• Encrypt with a customer-managed KMS key.
• Lifecycle: 30 days non-current on STANDARD_IA, transition to Glacier at 90 days, retain for one year.
• Pin the Prometheus minor version on the drill host. The snapshot format is not portable across major versions.
• Version the rule files with the snapshot. Record the Git commit SHA in the snapshot tar.
• Restore drill quarterly. Smoke test a known series from the last 24h; document the result.
• Alert on snapshot age (RPO breach) and on drill age (verification staleness).
• The backup IAM role cannot delete. The lifecycle does.

Verification

You should now be able to answer:

• Why is a copy of the live TSDB directory not a backup, and what is the right primitive?
• Where does the configuration live in a Prometheus backup, and why is it separate from the snapshot?
• What is the role of versioning and cross-region replication in the 3-2-1 rule for Prometheus?
• Why is the restore drill mandatory even when the bucket is full and the snapshot is fresh?
• What is the version-pinning constraint on the drill host?

Quiz