Backup Basics

The incident was a routine cluster credential rotation. The operator hit return on the rotation command at 14:02 on a Thursday. By 14:04 the dashboards were blank, the alertmanager was paging about its own absence, and the on-call engineer was reading a runbook that referenced a backup they had never restored. The “backup” was an S3 bucket with versioning enabled and a lifecycle rule that expired non-current versions after seven days. The rotation had been scheduled for ten days.

This lesson is the discipline that prevents the runbook from lying. A backup is not a file on a disk. A backup is a recovery path that has been exercised. The right shape for an observability stack is not a single tool. It is a per-tier policy that names what is backed up, where the copy lives, how often it is taken, and when it was last restored end-to-end.

What it is

A backup, in the observability context, is a durable copy of state that an operator can recover from after loss, corruption, or hostile action. Three things make a backup a backup:

1. The copy is independent of the primary. A second disk in the same host is not a backup. A second bucket in the same account is barely a backup.
2. The copy is recoverable. The data is consistent at the level the component requires (a snapshot, not a half-written block).
3. The recovery has been demonstrated. A restore drill has produced a working service from the copy, on a schedule.

The 3-2-1 rule expresses the first two properties as a count: three copies, on two media, with one offsite. The third property is the operational discipline that turns the count into a recovery.

Why a sysadmin cares

The observability stack is what tells you whether the rest of the stack is healthy. A stack that cannot recover itself is a stack that hides incidents. The incidents fall into three classes:

• Operator error. A retention change goes too far. A wipe of data/ before reading the path. A misconfigured compaction that drops data prematurely.
• Host or volume failure. Disk, filesystem, EBS detachment, container loss. The component does not start.
• Hostile action. Ransomware, an over-privileged CI token, a deleted bucket by mistake. The recovery path has to be outside the blast radius.

Each class demands a different copy. Operator error needs a recent snapshot on a different host. Host failure needs a copy on a different host or volume. Hostile action needs a copy on a different account or even a different provider. One copy cannot cover all three.

How it works

The mental model is a per-tier policy mapped to a per-component shape. The tiers are not about importance; they are about how recent the data must be on recovery.

Tier            RPO            RTO            Components
--------------- -------------- -------------- --------------------
Hot             minutes        minutes        Prometheus,
                                               Alertmanager
                                               (state),
                                               Grafana DB
                                               (UI-created
                                                dashboards,
                                                annotations)
Warm            hours          hours          Loki chunks,
                                               Tempo traces,
                                               Prometheus
                                               snapshot history
Cold            day            day            Grafana
                                               provisioning
                                               (Git),
                                               Alertmanager
                                               config (Git),
                                               rule files (Git)

Two shapes dominate in production observability stacks:

Shape A (stateful, on-disk):  Shape B (object-store native):
  Prometheus                     Loki
  Grafana DB                     Tempo
  Alertmanager nflog             (some Grafana HA)
       |                              |
   TSDB snapshot /                Object-store versioning +
   sqlite .backup                 cross-region replication +
       |                          lifecycle policy
   S3 with versioning +               |
   cross-region replication      Lifecycle expires
       |                          non-current versions
   Restore drill                  after N days
   quarterly                          |
                                  Restore drill
                                  quarterly

Shape A is for components that own local on-disk state. Shape B is for components whose source of truth is the object store. In both shapes the recovery path is the same: take a copy, store it independently, and prove the copy works.

How to configure it

The right per-component shapes are detailed in the following lessons. The platform-level policy is what this lesson owns. A representative policy expressed as a Prometheus alert:

# /etc/prometheus/rules/backup-policy.rules.yml
# SEVERITY: CONFIGURATION (rule reload only)
groups:
- name: backup-policy
  interval: 5m
  rules:
  # Alert when the last backup snapshot is older than the per-tier
  # RPO. The rule fires on the symptom (no recent backup) not the
  # cause (the job is broken, the disk is full, the credentials
  # expired).
  - alert: BackupSnapshotStale
    expr: time() - backup_snapshot_last_success_timestamp_seconds > 26 * 3600
    for: 30m
    labels:
      severity: page
      team: sre
      component: backup
    annotations:
      summary: 'Backup snapshot for {{ $labels.component }} is stale'
      description: |
        Last successful snapshot for {{ $labels.component }} on
        {{ $labels.instance }} was {{ $value | humanizeDuration }}
        ago. RPO target is 24h; age above 26h breaches the policy.
        Check the backup job, the storage path, and the credentials.

  # Alert when the last restore drill is older than the quarterly
  # target. A restore drill that has not run in a year is the same
  # as no restore drill.
  - alert: RestoreDrillOverdue
    expr: time() - backup_restore_drill_last_success_timestamp_seconds > 100 * 24 * 3600
    for: 24h
    labels:
      severity: ticket
      team: sre
      component: backup
    annotations:
      summary: 'Restore drill overdue for {{ $labels.component }}'
      description: |
        No successful restore drill for {{ $labels.component }} in
        {{ $value | humanizeDuration }}. The next quarterly drill is
        overdue. Schedule and document the outcome.

The corresponding instrumentation on the backup host:

# SEVERITY: CONFIGURATION
# /usr/local/bin/backup_metrics.sh
# Emits the snapshot age and the last restore-drill age to a
# node-exporter textfile collector. The Prometheus rule reads
# these gauges.
cat <<EOF > /var/lib/node-exporter/backup.prom
# HELP backup_snapshot_last_success_timestamp_seconds Unix time of the last successful backup snapshot.
# TYPE backup_snapshot_last_success_timestamp_seconds gauge
backup_snapshot_last_success_timestamp_seconds{component="prometheus"} $(stat -c %Y /var/backups/prometheus/last-snapshot.tar.gz 2>/dev/null || echo 0)
backup_snapshot_last_success_timestamp_seconds{component="grafana-db"} $(stat -c %Y /var/backups/grafana/last-db.sqlite.gz 2>/dev/null || echo 0)
# HELP backup_restore_drill_last_success_timestamp_seconds Unix time of the last successful restore drill.
# TYPE backup_restore_drill_last_success_timestamp_seconds gauge
backup_restore_drill_last_success_timestamp_seconds{component="prometheus"} $(stat -c %Y /var/backups/prometheus/drill/last-drill.txt 2>/dev/null || echo 0)
backup_restore_drill_last_success_timestamp_seconds{component="grafana-db"} $(stat -c %Y /var/backups/grafana/drill/last-drill.txt 2>/dev/null || echo 0)
EOF

How to validate it

Top-level: the policy is wired and the metrics are fresh.

# SEVERITY: READ-ONLY
# Confirm the textfile collector emits both gauges with a recent
# timestamp (now minus 24h in seconds = the RPO boundary).
curl -s http://node-exporter:9100/metrics | grep -E '^backup_(snapshot|restore_drill)_'

# Expected output (illustrative, recent values):
# backup_snapshot_last_success_timestamp_seconds{component="prometheus"} 1723564800
# backup_snapshot_last_success_timestamp_seconds{component="grafana-db"} 1723561200
# backup_restore_drill_last_success_timestamp_seconds{component="prometheus"} 1712000000

Mid-level: each component’s snapshot exists and is non-empty.

# SEVERITY: READ-ONLY
for f in /var/backups/prometheus/last-snapshot.tar.gz \
         /var/backups/grafana/last-db.sqlite.gz; do
  [ -s "$f" ] && echo "OK   $f ($(stat -c %s "$f") bytes)" || echo "FAIL $f"
done

End-level: the restore drill produced a running service on the staging host within the last 100 days.

# SEVERITY: READ-ONLY
# The drill marker file is updated only by the drill runbook, after
# the smoke test passes.
cat /var/backups/prometheus/drill/last-drill.txt
# Last successful restore drill of prometheus: 2026-05-14, validated against 30d of metrics.

How it can fail

Six failure modes recur in observability-stack backups:

1. The backup that copies a live data directory. A cron job tar czf backup.tgz /var/lib/prometheus runs while the Prometheus process is writing to the WAL. The tar captures a half-written block; the restore fails to replay. Symptom: the tar exists and is non-empty; promtool tsdb analyze rejects the restored copy.
2. The lifecycle rule that expires non-current versions too quickly. Versioning is enabled but the lifecycle expires non-current objects after seven days. A long-festering problem (a slow credential rotation) consumes the seven-day window and the recovery finds empty buckets. Symptom: the alert fires on age but the recovery still finds no data.
3. The backup account shares credentials with the production account. A compromised CI token with s3:* can delete both the primary bucket and the “backup” bucket. Symptom: both buckets show simultaneous, large delete operations in CloudTrail.
4. The sqlite file copied while the server writes. A naive cp grafana.db grafana.db.bak against a running grafana-server produces a corrupt copy. Symptom: sqlite3 grafana.db.bak "PRAGMA integrity_check" returns anything other than ok.
5. The restore drill that was scheduled and never run. A quarterly drill on the calendar, owner left the team, no replacement. The drill marker file freezes; the alert silences itself; the runbook rots. Symptom: RestoreDrillOverdue fires for a year and the team treats it as background noise.
6. The “we have Git” assumption that forgets UI-created state. Dashboards provisioned from Git are recoverable. Dashboards created through the UI live in the Grafana database; if the DB is not backed up, the dashboards vanish on restore. Symptom: a restored Grafana boots empty; the team discovers the dashboards that mattered were never in the provisioning tree.

How to troubleshoot it

The order is: does the copy exist, is it consistent, is the recovery path exercised, is the scope right.

1. Does the copy exist? stat /var/backups/<component>/last-* on the backup host. Empty or missing means the job is not running. The job log is the next stop.
2. Is it consistent? promtool tsdb analyze for Prometheus; sqlite3 ... "PRAGMA integrity_check" for Grafana sqlite; promtool check rules for any restored rule files. A copy that fails consistency is not a backup.
3. Is the recovery path exercised? cat /var/backups/<component>/drill/last-drill.txt. If the file is older than the policy, schedule the drill before changing anything else.
4. Is the scope right? Walk the in-scope list against the actual files in the backup directory. A short scope means the restore will be partial; a long scope means the job is doing unnecessary work.

Security implications

• The backup account holds a full copy of the production telemetry and the secrets used to reach it. That account is the highest blast-radius target in the observability stack.
• The credentials for the backup account are not the same as the credentials for the application. A leaked application credential must not be able to reach the backup.
• The restore drill produces a copy of the production telemetry on the staging host. That host is in scope for the same data classification as production. Treat the drill output as production data until the staging DB is wiped.
• Encryption at rest in the backup bucket is mandatory. The encryption key is a separate credential, rotated independently of the bucket.
• The backup job logs are themselves telemetry. The job runs on a host with the most privilege; the logs are a target.

Performance implications

• The snapshot operation on the source component holds a write lock or pauses compactions for the duration of the copy. Schedule the snapshot for off-peak; measure the impact before moving it.
• Object-store PUT cost is non-trivial for Prometheus TSDB and trace data. A daily snapshot of a busy cluster can be terabytes; the storage bill matters before the storage tier does.
• Lifecycle rules trade cost for recovery window. The default for observability is to keep daily snapshots for 30 days, weekly for 90, and monthly for a year. Cost-tuned defaults are tighter; revisit when the restore window changes.
• Cross-region replication doubles storage and adds replication bandwidth. It is the price of the offsite copy; do not skip it.
• A restore drill pulls the full snapshot to a staging host. The staging host’s disk and network are sized for the largest realistic snapshot, not the average.

Production guidance

• 3-2-1 as the floor: three copies, two media, one offsite. The offsite copy is in a separate account or provider.
• Tier per RPO/RTO. Hot tier gets hourly snapshots on local plus offsite; warm tier gets daily snapshots plus bucket versioning; cold tier is Git plus a documented restore procedure.
• Git for configuration, database for state. Anything that lives only in the database and is not provisioned is at risk.
• Restore drills on the calendar, owner named, outcome documented. Quarterly is the right cadence for most teams; monthly for hot paths.
• Alert on backup age, not on backup success alone. A job that completes every day but writes zero bytes is “successful”.
• Encrypt at rest. Rotate keys independently of buckets.
• Re-test credentials on rotation. A 90-day rotation that has not been validated against the backup bucket is a future incident.

Verification

You should now be able to answer:

• Why is “we have a backup” not the same as “we can recover”?
• What is the difference between the hot, warm, and cold tiers in an observability stack, and which components sit in each?
• Why is a copy on the same account not a 3-2-1 offsite copy?
• What is the right cadence for a restore drill, and what is the artefact that proves the drill ran?
• Why does Git hold the configuration but not the state?

Quiz