low riskinformational~30 min

Runbook: Back Up Observability Configuration

1 · Prerequisites

Confirm every item is in place before any state change.

Backup Basics
Prometheus Backup
Grafana Backup
Alertmanager Backup
Loki Backup
A written scope: which assets are in the backup, which tier each sits in, and who owns the restore
Credentials for the backup destination that the production workload cannot use, and cannot delete with

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· The scope is written down before anything is copied. For each asset: where the source of truth lives, what the correct copy primitive is, and which tier — hot, warm or cold — it belongs to. A backup job assembled at the console backs up what the person at the console remembered
· Configuration is already committed and pushed. prometheus.yml, the rule files, alertmanager.yml, the Grafana provisioning tree and the Loki and Tempo YAML all live in Git, and Git is their backup. A second copy of a file that is already in Git is redundancy, not a backup, and copying it can create the impression the state was covered too
· The configuration in Git is the configuration that is running. diff the deployed tree against a fresh checkout, and promtool check config / amtool check-config the deployed files. Backing up drifted configuration preserves the drift
· Free space on the Prometheus data volume is measured before the snapshot: df -h on the TSDB path. The snapshot lands inside that volume, and on a volume that is already tight it is the snapshot that fills it
· It is known whether --web.enable-admin-api is set on the Prometheus server, because the snapshot endpoint lives behind it — and so does delete_series. If it has to be enabled for this, that is a security decision with an owner, not a flag you add quietly
· The Grafana database engine is known — sqlite, MySQL or PostgreSQL. The correct copy primitive differs per engine, and cp of a live sqlite file is not one of them
· The destination is independent of the primary: a different account or provider, under credentials the production workload does not hold. A copy that a compromised production credential can delete is not an offsite copy
· The date of the last successful restore drill is known. If it is older than the policy window, this run is a copy, not a backup, and the drill is the thing that is actually overdue

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Record the start time and the versions in play — Prometheus, Grafana, Alertmanager, Loki, Tempo. A snapshot restored onto an older minor version can be refused outright, so the version is part of the artefact, not a footnote
2Confirm configuration is in Git and Git is current: git -C CONFIGREPO status --short is clean and git log -1 matches what is deployed. If it is not clean, stop and commit — this is the cheapest moment to fix drift and the most expensive one to skip
3Prometheus: take a consistent snapshot through the admin API rather than copying the data directory: curl -sf -XPOST "$PROM/api/v1/admin/tsdb/snapshot". The response names a directory under snapshots/ inside the TSDB path. A tar of a live data directory captures a half-written WAL and will not replay
4Prometheus: archive the named snapshot directory and verify the archive before shipping it — extract it to scratch and run promtool tsdb list against the extracted path. An archive nobody has opened is a file, not a backup
5Prometheus: ship the archive to the independent destination with encryption at rest, then remove the snapshot directory from the data volume. Prometheus does not clean it up; every run that skips this step leaves a copy behind on the volume it is protecting
6Grafana: dump the database with the engine's own primitive — sqlite3 grafana.db ".backup ..." for sqlite, mysqldump or pg_dump for the server engines. This is the half that holds users, API keys, annotations and every dashboard somebody built in the UI and never exported
7Grafana: verify the dump before shipping — PRAGMA integrity_check must return ok for sqlite, and a row count from the dashboard table must be non-zero. A dump that restores an empty Grafana passes every check that only asks whether the file exists
8Alertmanager: commit the configuration and export the live silences via GET /api/v2/silences. Silences are in-memory state created by humans during incidents; they exist nowhere in Git and are lost on an ungraceful restart
9Loki and Tempo: do not copy the object store. Verify instead that the properties that make it durable are on and correct — bucket versioning, cross-region replication, and a non-current-version lifetime longer than your realistic detection window. For these components the backup is a set of bucket properties, and the check is that they are still set
10Emit the freshness signal so the platform can alert on this run: update the backup_snapshot_last_success_timestamp_seconds gauge in the node_exporter textfile directory, per component. A backup nobody can alert on stops silently
11Record what was captured, where it went, under which credentials, and against which component versions. That record is what the person running the restore reads first, and they will be reading it under pressure
12If the restore drill is overdue, schedule it now with a named owner and a date. Steps 1 to 11 produce a copy; only the drill produces a backup

4 · Verification

Confirm the procedure actually fixed the problem.

✓The Prometheus archive extracts and promtool tsdb list reports blocks against the extracted directory. Extraction alone proves the tar is intact; the block listing proves the TSDB is
✓The extracted snapshot's meta.json covers the window you expect — a snapshot whose maxTime is hours old means the copy predates the change you were protecting against
✓The Grafana dump returns ok from PRAGMA integrity_check (sqlite) or restores cleanly into a scratch database (MySQL, PostgreSQL), and a count of the dashboard table is non-zero
✓The Alertmanager configuration passes amtool check-config, and the exported silence document parses and contains the number of active silences the running instance reports
✓The object-store buckets still have versioning enabled, replication in the Enabled state, and a non-current-version expiry longer than your detection window. Confirmed by reading the bucket configuration, not from a diagram
✓Every object landed under the backup credentials and is readable with them. A copy written with production credentials into a bucket only production can read has not left the blast radius
✓The snapshot directory has been removed from the Prometheus data volume, and df -h on that volume shows the space back
✓backup_snapshot_last_success_timestamp_seconds for each component is within the RPO window when queried from Prometheus — the gauge is what the staleness alert reads, so a backup that did not update it is invisible to the platform
✓The scope list was walked against what is actually in the destination, item by item. A short scope produces a partial restore, and the moment to discover that is now

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶A backup is additive: there is nothing to undo in the copy itself. What needs undoing is what the run left behind, and every item below has been left behind in somebody's production estate
↶Remove the snapshot directory from the Prometheus data volume if the shipping step failed part way. Prometheus never removes it, and a retry loop that snapshots on every attempt fills the volume it is protecting
↶If --web.enable-admin-api was enabled for this run, decide deliberately whether it stays. The same flag exposes delete_series, so leaving it on for convenience widens what anyone reaching the API can do
↶Delete any decrypted database dump left on a working host. That file contains Grafana API keys and user records; it is production data sitting outside production controls
↶Stop and remove any scratch Grafana or Prometheus started to verify a copy. A verification instance keeps running, keeps holding real credentials, and nobody remembers it a week later
↶If a lifecycle or replication setting was changed to make the destination work, put the change through the same review as any other infrastructure change. A non-current-version expiry shortened during a backup run is a recovery window quietly removed
↶If the freshness gauge was written by hand rather than by the job, revert it. A hand-written timestamp silences the staleness alert for a backup that did not happen, which is worse than the alert

6 · Escalation

When the runbook isn't enough, contact:

· Escalate to the platform owner if --web.enable-admin-api is not set and enabling it is required. That flag is a change to the server's exposed surface and the decision is not the backup operator's
· Escalate to whoever owns the storage account if the backup destination shares credentials with production. That is the failure mode where 3-2-1 is satisfied on paper and one compromised token deletes both copies; it is structural and cannot be fixed inside this runbook
· Escalate to the Grafana owner if the database dump restores with zero dashboards or a failed integrity check. Do not overwrite the previous good dump with it — the previous dump may be the only working copy
· Escalate to the security owner before backing up a Grafana database into any destination that is not encrypted at rest. That database holds API keys and user records in a form the backup makes portable
· Escalate to the team that owns the object store if versioning or replication has been turned off on a Loki or Tempo bucket. For those components that setting is the backup, and it has been silently absent for as long as it has been off
· Escalate — and treat it as a finding rather than a task — if the last restore drill predates the policy window. The correct response is to schedule the drill with a named owner, not to take another copy and hope

A backup is not a file on a disk. It is a recovery path somebody has walked. Everything in this runbook exists to close the distance between those two things, because the failure that costs the most is not a missing backup — it is a backup directory full of archives that nobody has ever opened, sitting next to a runbook that says “restore from backup” as though that step were free.

Two mistakes shape this procedure. The first is copying what Git already holds: the configuration files are in a repository with history, and a second copy of them is redundancy rather than protection. The second is copying live state with the wrong primitive — a tar of a running TSDB, a cp of a running sqlite database — which produces a file of the right size, on the right schedule, with the right name, that will not load.

When this runbook applies

A scheduled capture of platform state, on the cadence the policy names.
Immediately before a version upgrade, a retention change, or any edit to a storage flag.
Before a credential rotation that touches the backup path itself.
After onboarding a component whose state is not yet in scope.

When it does not

You are recovering something. This runbook takes copies. The restore path is its own procedure, and reading this one during an outage costs time you do not have.
The change is configuration only. A rule-file edit is protected by the commit that carries it. Taking a platform snapshot for a change that Git already records is ceremony.
The last drill is overdue. Another copy does not fix an unproven recovery path. Fix the drill first; the copies are already piling up.

What actually needs backing up

The single most common defect in observability backup jobs is scope, in both directions: copying configuration that Git already holds, while missing state that exists nowhere else.

Asset	Where truth lives	Correct primitive	If skipped
`prometheus.yml`, rule files	Git	Commit and push	Nothing, if Git is real
Prometheus TSDB	Local disk	Admin-API snapshot	The metric history is gone
`alertmanager.yml`, templates	Git	Commit and push	Nothing, if Git is real
Alertmanager silences	Alertmanager memory	`GET /api/v2/silences` export	Every suppressed alert fires at once on recovery
Grafana provisioning tree	Git	Commit and push	Nothing, if Git is real
Grafana database	Database engine	Engine-native dump	Users, API keys, annotations, every UI-built dashboard
Loki and Tempo data	Object store	Versioning plus replication	Log and trace history, and the compliance evidence in it
Secrets and tokens	Secret store	Rotated, not copied	Recovery stalls on credentials nobody can produce

Two rows deserve reading twice. Alertmanager silences are created by humans, during incidents, and live only in memory; they are the state most likely to be missing and least likely to be noticed until a recovered Alertmanager pages the whole company about maintenance that was silenced last week. The Grafana database is where every dashboard built in the UI lives — the ones teams actually use during an investigation, which are precisely the ones that were never exported to the provisioning repository.

Step 1 — prove Git is current before copying anything else

Read-only / Safedrift is preserved by backups, not fixed by them

# Substitute your own values before running:
CONFIG_REPO=/opt/observability-config

git -C "$CONFIG_REPO" status --short
git -C "$CONFIG_REPO" log -1 --oneline

# Does the repository match what is deployed?
diff -r "$CONFIG_REPO/prometheus" /etc/prometheus || echo "DRIFT: prometheus"
diff -r "$CONFIG_REPO/grafana/provisioning" /etc/grafana/provisioning || echo "DRIFT: grafana"

# Is what is deployed even valid?
promtool check config /etc/prometheus/prometheus.yml
amtool check-config /etc/alertmanager/alertmanager.yml

A dirty repository or a non-empty diff is a finding, and it is the cheapest one you will ever get. Backing up a drifted tree preserves the drift and hands it to whoever runs the restore, at the worst possible moment for discovering that the deployed file and the reviewed file were not the same file.

Step 2 — Prometheus: snapshot, do not copy

The TSDB is being written to continuously. A tar of the data directory captures a partially written WAL segment; the archive is valid and the database inside it is not. Prometheus provides the consistent primitive.

Service impact possiblethe snapshot lands inside the data volume

# Substitute your own values before running:
PROM=http://prom-primary.example.com:9090
TSDB_PATH=/var/lib/prometheus

# Headroom first: the snapshot is written into this volume.
df -h "$TSDB_PATH"

# Ask Prometheus for a consistent snapshot. Requires --web.enable-admin-api.
SNAP_NAME=$(curl -sf -XPOST "$PROM/api/v1/admin/tsdb/snapshot" | jq -r '.data.name')
echo "snapshot: $SNAP_NAME"

ls -l "$TSDB_PATH/snapshots/$SNAP_NAME"

Two costs this step has that a file copy does not

The snapshot is created inside the data volume it protects. It lands under snapshots/ in the TSDB directory, and Prometheus never removes it. Left there it holds block data on the volume Prometheus needs for ingestion, past the point where retention would otherwise have reclaimed it — so a job that fails after the snapshot and retries on a schedule quietly converts the retention policy into “keep everything” and fills the volume. Measure headroom first; clean up in step 4 even when the run failed.

The endpoint lives behind --web.enable-admin-api, and so does delete_series. Turning the flag on for backups also exposes the endpoint that deletes data. That is a defensible trade, but it is a decision about the server’s exposed surface, and it belongs to whoever owns that surface rather than to whoever wrote the backup job.

Step 3 — verify the archive before it leaves the host

This is the step that separates a backup from a file. Extract to scratch and ask the tool that will have to read it whether it can.

Read-only / Safeopen the archive you are about to trust

# Substitute your own values before running:
TSDB_PATH=/var/lib/prometheus
SNAP_NAME=20260818T120000Z-4f2c1a9b8d3e5f60
ARCHIVE=/var/tmp/prometheus-snapshot.tar.gz

tar -C "$TSDB_PATH/snapshots" -czf "$ARCHIVE" "$SNAP_NAME"

SCRATCH=$(mktemp -d)
tar -C "$SCRATCH" -xzf "$ARCHIVE"

# Does the TSDB tooling recognise what came out?
promtool tsdb list "$SCRATCH/$SNAP_NAME"

rm -rf "$SCRATCH"

The block listing is the check that matters. A tar that extracts proves the tar; only a tool that parses the blocks proves the database. If promtool tsdb list reports nothing, the copy is unusable and the run has failed — do not ship it, and do not let it overwrite the previous archive, which may be the last working one.

Step 4 — ship it, then clean up the volume

Configuration changeindependent destination, then release the space

# Substitute your own values before running:
TSDB_PATH=/var/lib/prometheus
SNAP_NAME=20260818T120000Z-4f2c1a9b8d3e5f60
ARCHIVE=/var/tmp/prometheus-snapshot.tar.gz
DEST=s3://obs-backup-primary/prometheus/2026-08-18/

# Ship under the BACKUP credentials, not the production ones.
AWS_PROFILE=obs-backup aws s3 cp --sse aws:kms "$ARCHIVE" "$DEST"

# Prometheus does not remove the snapshot. You must.
rm -rf "$TSDB_PATH/snapshots/$SNAP_NAME"
rm -f "$ARCHIVE"
df -h "$TSDB_PATH"

The profile matters more than it looks. If the copy is written with the credentials the production workload already holds, then anything that compromises production can also delete the backup, and the offsite copy you are counting on is inside the blast radius it was meant to escape.

Step 5 — Grafana: the database is the half people forget

The provisioning tree is in Git. The database is not, and it holds every dashboard somebody built in the UI, every annotation, every user, and every API key.

Read-only / Safesqlite has its own copy primitive for a reason

# Substitute your own values before running:
GRAFANA_DB=/var/lib/grafana/grafana.db
DUMP=/var/tmp/grafana-db.sqlite

# .backup is safe against a live writer. cp is not.
sqlite3 "$GRAFANA_DB" ".backup '$DUMP'"

sqlite3 "$DUMP" "PRAGMA integrity_check;"
sqlite3 "$DUMP" "SELECT count(*) FROM dashboard;"

PRAGMA integrity_check returning anything other than ok means the copy was taken with the wrong primitive — almost always a plain cp against a database with an open write transaction. A zero row count means the dump is intact and empty, which restores a Grafana with no dashboards in it and passes every check that only asks whether the file exists.

On MySQL or PostgreSQL the same discipline applies with the engine’s own tool — mysqldump or pg_dump — and the same verification: restore into a scratch database and count rows. In every case the dump must come from the same host that owns the provisioning tree, or you recover a configuration and a state that were never running together.

Step 6 — Alertmanager: export the silences

Read-only / Safehuman-created state that exists nowhere in Git

# Substitute your own values before running:
AM=http://alertmanager.example.com:9093
OUT=/var/tmp/silences.json

curl -sf "$AM/api/v2/silences" -o "$OUT"

# How many are actually active right now?
jq '[.[] | select(.status.state == "active")] | length' "$OUT"

Silences are the state most likely to be missing from a backup and least likely to be missed until recovery. A platform restored without them re-pages every alert somebody deliberately suppressed — during maintenance, during a known-bad deploy, during the very incident you are recovering from — and it does it all at once, to the people already handling the outage.

Step 7 — Loki and Tempo: verify properties, do not copy objects

For the object-store-native components there is no archive to take. The chunk and index stores are the source of truth, and the backup is a set of bucket properties. The check is that they are still set.

Read-only / Safethe backup is a configuration, so read the configuration

# Substitute your own values before running:
BUCKET=obs-loki-chunks-primary

aws s3api get-bucket-versioning --bucket "$BUCKET"
aws s3api get-bucket-replication --bucket "$BUCKET"
aws s3api get-bucket-lifecycle-configuration --bucket "$BUCKET"

Read the lifecycle output carefully. Versioning protects against deletion only for as long as non-current versions survive, so a non-current expiry of seven days means a problem that goes unnoticed for eight days is unrecoverable — and slow problems, like a credential rotation that quietly broke a writer, are exactly the ones that go unnoticed. The expiry has to be longer than your realistic time to detect, not longer than your ideal one.

Step 8 — make the run visible to the platform

Configuration changethe gauge the staleness alert reads

# Substitute your own values before running:
TEXTFILE_DIR=/var/lib/node_exporter/textfile_collector
NOW=$(date +%s)

cat > "$TEXTFILE_DIR/backup.prom.tmp" <<EOF
# HELP backup_snapshot_last_success_timestamp_seconds Unix time of the last successful backup snapshot.
# TYPE backup_snapshot_last_success_timestamp_seconds gauge
backup_snapshot_last_success_timestamp_seconds{component="prometheus"} $NOW
backup_snapshot_last_success_timestamp_seconds{component="grafana-db"} $NOW
EOF

# Rename into place so the collector never reads a half-written file.
mv "$TEXTFILE_DIR/backup.prom.tmp" "$TEXTFILE_DIR/backup.prom"

Write the gauge only on success, and only from the job. A timestamp updated by hand, or updated at the start of a run that later failed, silences the staleness alert for a backup that does not exist — which is strictly worse than having no alert, because now the platform is asserting something false.

Decision points where you should stop and think

Before overwriting the previous archive. If this run’s verification failed, the previous archive may be the only working copy on the platform. Keep it. Versioned destinations make this the default; a destination without versioning makes every failed run a coin flip.

Before backing up a database you cannot encrypt at rest. The Grafana database carries API keys and user records. A dump in an unencrypted bucket is a portable copy of your authentication surface.

Before extending the scope. Every asset added to the backup is an asset somebody has to restore, verify and pay for. Derived data does not belong here: recording rules re-evaluate, alert state re-fires, caches rebuild.

Holding is a legitimate outcome. If the destination credentials are wrong, or the volume has no headroom, stopping with a clear note beats taking a copy that cannot be restored. Record what was not captured, who owns the fix, and by when — an incomplete backup that is documented is recoverable knowledge; one that is silent is a trap.

Rollback

A backup adds rather than replaces, so the rollback list is about what the run leaves behind — and every item below has been left behind in somebody’s production estate.

Left behind	Why it matters	Action
Snapshot directory under `snapshots/`	Consumes the volume Prometheus ingests into	`rm -rf` it, even on a failed run
`--web.enable-admin-api` newly enabled	Also exposes `delete_series`	Decide deliberately; do not leave it on by drift
Decrypted database dump on a work host	Contains API keys and user records	Delete it; it is production data outside production controls
Scratch Grafana or Prometheus from verification	Keeps running, keeps holding real credentials	Stop and remove it before you close the ticket
Lifecycle or replication setting changed to make shipping work	Silently shortens the recovery window	Put it through normal change review
Freshness gauge written by hand	Silences the staleness alert for a backup that did not happen	Revert it and let the alert fire

Escalation

Escalate when:

The admin API is not enabled and enabling it is required. Platform owner — it changes the server’s exposed surface.
The backup destination shares credentials with production. Storage account owner, and treat it as structural: it cannot be fixed inside this runbook.
The Grafana dump fails its integrity check or restores empty. Grafana owner, and do not let the bad dump overwrite the previous one.
A Loki or Tempo bucket has lost versioning or replication. Object store owner, urgently: for those components that setting is the backup, and it has been absent for as long as it has been off.
The last restore drill predates the policy window. That is a finding with an owner and a date, not another copy.