Lab: Backup and Restore

Objective

By the end of this lab you will have destroyed the state of a working observability stack and rebuilt it from copies you took yourself, and you will have a number: the wall-clock minutes the rebuild took. That number is the only honest statement anyone can make about their RTO, and almost nobody has one.

The point is not the three backup commands. It is that a backup you have never restored is a hypothesis, and this lab is where you test it while the cost of being wrong is a docker compose command.

Architecture

Three stateful services, three named volumes, and a backup directory on the host.

  +--------------+  admin API snapshot   +----------------------+
  | Prometheus   | --------------------> |                      |
  | :9090        |                       |                      |
  +--------------+                       |  ./backup/           |
                                         |    prometheus-snap/  |
  +--------------+  sqlite .backup       |    grafana.db        |
  | Grafana      | --------------------> |    silences.json     |
  | :3000        |                       |                      |
  +--------------+                       |                      |
                                         |  ./  (the config —   |
  +--------------+  GET /api/v2/silences |      Git is the      |
  | Alertmanager | --------------------> |      backup)         |
  | :9093        |                       +----------------------+
  +--------------+

  volumes: obs-backup-prom-data
           obs-backup-grafana-data
           obs-backup-am-data

The volumes carry fixed names rather than Compose-generated ones. That is not cosmetic: the restore mounts them into helper containers by name, and a name derived from the directory you happened to be in is a name that changes when somebody clones the repo somewhere else.

Requirements

Linux or macOS with a shell, Docker Engine 28.x and Docker Compose v2.
TCP ports 9090, 3000 and 9093 free, and about 2 GB of free disk.
curl and jq on the host. Everything else runs in a container.
Internet access: the restore helper installs sqlite inside a throwaway Alpine container.
90 minutes, of which roughly 10 are spent letting Prometheus accumulate data worth restoring.
No out-of-band access requirement. Nothing touches host networking or SSH.

Scenario

A team runs the quarterly backup review. The S3 bucket has objects in it, recent ones, correctly sized. The retention rule is right. The IAM policy is right. Everybody signs off.

Four months later a volume is lost. The restore begins and stops nine minutes in, because the Prometheus snapshot was taken with cp on a running server and the blocks are half-written; because the Grafana database was copied while grafana-server held it open and now fails an integrity check; and because nobody had ever asked whether the silence that covered the maintenance window would come back, so the recovered stack pages the on-call for the outage they are currently fixing.

Nothing about the backup job was wrong. What was missing was the drill.

Tasks

Task 1: Record the starting state and bring up the stack

Read-only / Safehost— Ports free, and no volume of these names already present

$ ss -ltnp 2>/dev/null | grep -E ':(9090|3000|9093)\b' ; docker volume ls | grep obs-backup || echo 'clean start'

WORKDIR="$HOME/obs-backup-lab"
mkdir -p "$WORKDIR"/provisioning/datasources "$WORKDIR"/backup
cd "$WORKDIR"

# docker-compose.yml
services:
  prometheus:
    image: prom/prometheus:v2.55.1
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.path=/prometheus
      - --web.enable-admin-api
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prom-data:/prometheus
    ports:
      - "9090:9090"

  grafana:
    image: grafana/grafana:11.3.0
    environment:
      GF_SECURITY_ADMIN_PASSWORD: lab-not-a-secret
    volumes:
      - ./provisioning:/etc/grafana/provisioning:ro
      - grafana-data:/var/lib/grafana
    ports:
      - "3000:3000"

  alertmanager:
    image: prom/alertmanager:v0.28.1
    command:
      - --config.file=/etc/alertmanager/alertmanager.yml
      - --storage.path=/alertmanager
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
      - am-data:/alertmanager
    ports:
      - "9093:9093"

volumes:
  prom-data:
    name: obs-backup-prom-data
  grafana-data:
    name: obs-backup-grafana-data
  am-data:
    name: obs-backup-am-data

--web.enable-admin-api is what makes the snapshot endpoint exist. It is off by default, and it also enables the delete-series endpoints, which is why production usually exposes it on a loopback listener or behind a proxy that allows exactly one path.

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']

  - job_name: alertmanager
    static_configs:
      - targets: ['alertmanager:9093']

# alertmanager.yml
route:
  receiver: 'sink'
  group_by: ['alertname']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 1h

receivers:
  - name: 'sink'

# provisioning/datasources/prometheus.yml
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: false

Configuration changehost— Starts all three services and their named volumes

$ docker compose up -d && docker compose ps

Now leave it alone for five minutes. Prometheus is scraping itself and Alertmanager every 15 seconds; the drill needs data that predates the destruction, and a TSDB with ninety seconds in it does not prove anything.

Task 2: Create the state that has to survive

Three pieces of state, one per component, each created the way a human would create it rather than the way a fixture would.

Grafana: a dashboard that exists only in the database. The datasource came from the provisioning tree and will come back on its own; a dashboard created through the API lives in grafana.db and will not.

cat > dashboard.json <<'JSON'
{
  "dashboard": {
    "uid": "lab-restore-proof",
    "title": "Restore proof",
    "tags": ["drill"],
    "panels": [],
    "schemaVersion": 39
  },
  "overwrite": true
}
JSON

curl -sf -u admin:lab-not-a-secret \
  -H 'Content-Type: application/json' \
  --data-binary @dashboard.json \
  http://localhost:3000/api/dashboards/db | jq -r '.uid, .status'

Alertmanager: a silence covering a change window.

curl -sf -X POST -H 'Content-Type: application/json' \
  --data '{"matchers":[{"name":"alertname","value":"HostDown","isRegex":false,"isEqual":true},{"name":"instance","value":"db-1","isRegex":false,"isEqual":true}],"startsAt":"'"$(date -u +%Y-%m-%dT%H:%M:%S.000Z)"'","endsAt":"'"$(date -u -d '+4 hours' +%Y-%m-%dT%H:%M:%S.000Z)"'","createdBy":"you@example.com","comment":"CHG-1042 planned reboot of db-1"}' \
  http://localhost:9093/api/v2/silences | jq -r '.silenceID'

On macOS, date -u -d '+4 hours' is not available; use date -u -v+4H +%Y-%m-%dT%H:%M:%S.000Z instead.

Prometheus: a timestamp you can prove against. Record the current epoch second and the value of a series at that instant. This is the assertion the restore has to satisfy.

T0=$(date -u +%s)
echo "T0=$T0" | tee backup/drill-t0.txt

curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=up{job="prometheus"}' \
  --data-urlencode "time=$T0" \
  | jq -r '.data.result[] | "\(.metric.job) \(.value[1])"'

A restored Prometheus that answers this query with the same value has the data. An empty result means you restored an empty directory and started a healthy, useless server.

Task 3: Snapshot the Prometheus TSDB

Do not copy /prometheus. A running Prometheus is writing into the head block and the WAL, and a file-level copy of a moving target is a directory of fragments that will fail to load. The admin API exists for exactly this reason: it pauses the head write, produces a consistent directory, and tells you its name.

Service impact possiblehost— Non-disruptive to queries; briefly pauses the head write

$ SNAP=$(curl -sf -XPOST http://localhost:9090/api/v1/admin/tsdb/snapshot | jq -r '.data.name') && echo "snapshot: $SNAP"

Look at what it produced before you copy it. Do not take anybody’s word for the layout, including this lab’s:

docker compose exec -T prometheus ls -la "/prometheus/snapshots/$SNAP"

Copy it out of the container and onto the host:

docker compose cp "prometheus:/prometheus/snapshots/$SNAP" ./backup/prometheus-snapshot
du -sh backup/prometheus-snapshot

Then release the snapshot inside the container. It is made of hard links, so it costs almost nothing until the source blocks are compacted away — but “almost nothing” becomes real money on a server that snapshots hourly and never deletes.

Destructivehost— Removes the snapshot directory inside the container, not your copy

$ curl -sf -X DELETE "http://localhost:9090/api/v1/admin/tsdb/snapshot?name=$SNAP" && echo "released $SNAP"

Under the hood

The snapshot endpoint does not duplicate data. It creates a directory and hard-links each block file into it, then includes the current head so that recent samples — the ones a file copy is most likely to mangle — are part of the consistent view. Because the result is a set of links to immutable files, the directory is stable from the moment it is returned: the tar or cp that follows is reading files nothing will write to again.

What it does not do is coordinate with anything outside the process. If Prometheus is remote-writing to a long-term store, samples that flow during the snapshot window land there and not in your copy, and the restored instance will have a gap that the remote store does not. And the snapshot is a TSDB format, not an interchange format: a snapshot taken on 2.55 is not guaranteed to load on an older minor. Restore onto the version you backed up from, or rehearse the upgrade as part of the drill.

Task 4: Back up the Grafana database, and verify it before you trust it

cp grafana.db while grafana-server is running produces a file that usually looks fine and sometimes is not — which is the worst possible failure mode, because it is discovered during a restore. SQLite’s online backup API takes a consistent copy of a database that is being written to, and the sqlite3 shell exposes it as .backup.

The Grafana image is not guaranteed to ship the sqlite3 shell, so run it from a throwaway container mounted on the same volume:

docker run --rm \
  -v obs-backup-grafana-data:/var/lib/grafana \
  -v "$PWD/backup:/backup" \
  alpine:3.20 sh -c '
    apk add --no-cache sqlite >/dev/null &&
    sqlite3 /var/lib/grafana/grafana.db ".backup /backup/grafana.db" &&
    sqlite3 /backup/grafana.db "PRAGMA integrity_check;"'

The integrity check is the part people skip. It prints ok for a sound database and a list of problems for a damaged one, and it costs a second on a file this size. A backup job that ships without running it is a job that will faithfully replicate corruption to the offsite copy.

ls -l backup/grafana.db

Task 5: Export the Alertmanager silences

The configuration is already safe — it is a file in this directory, which is the stand-in for Git. Routing, receivers, inhibit rules and templates all live there and need no separate backup.

Silences do not. They are runtime state created by whoever is running the change, and they are the piece a restore most often forgets.

curl -sf http://localhost:9093/api/v2/silences \
  | jq '[.[] | select(.status.state != "expired")]' \
  > backup/silences.json

jq -r '.[] | "\(.id)  \(.comment)"' backup/silences.json

The notification log is deliberately not in the backup. It exists to de-duplicate notifications across a short window, and a de-duplication decision that is an hour old has no value; restoring it would at best suppress a notification that should be sent.

Task 6: Destroy it

Note the wall-clock time. From here to a working stack is your measured RTO, and the clock does not stop for reading.

date -u +%H:%M:%S | tee backup/drill-start.txt

The next command deletes all three volumes and everything in them

docker compose down -v removes the containers, the network, and every volume declared in the Compose file: obs-backup-prom-data, obs-backup-grafana-data and obs-backup-am-data. The metrics, the dashboard and the silence are gone from the host at that moment, and the only remaining copies are the three files in ./backup.

Confirm the backup directory is populated before you run it. If any of these is missing or zero bytes, stop and go back:

ls -l backup/grafana.db backup/silences.json backup/drill-t0.txt
du -sh backup/prometheus-snapshot

Data-loss riskhost— Deletes the three named volumes. This is the drill.

$ docker compose down -v && docker volume ls | grep obs-backup || echo 'volumes gone'

Task 7: Restore

Create the volumes first, so the restore can write into them before any service is running. Restoring into a volume a live Prometheus is already using is how you get a server holding a half-replaced data directory in memory.

docker volume create obs-backup-prom-data
docker volume create obs-backup-grafana-data
docker volume create obs-backup-am-data

Prometheus. Copy the snapshot contents into the data directory and fix ownership: the image runs as uid 65534, and a data directory it cannot write to produces a server that starts and then fails on its first compaction.

docker run --rm \
  -v obs-backup-prom-data:/prometheus \
  -v "$PWD/backup:/backup:ro" \
  alpine:3.20 sh -c '
    cp -a /backup/prometheus-snapshot/. /prometheus/ &&
    chown -R 65534:65534 /prometheus &&
    ls /prometheus'

Grafana. Same shape, different uid: the Grafana image runs as 472.

docker run --rm \
  -v obs-backup-grafana-data:/var/lib/grafana \
  -v "$PWD/backup:/backup:ro" \
  alpine:3.20 sh -c '
    cp -a /backup/grafana.db /var/lib/grafana/grafana.db &&
    chown -R 472:0 /var/lib/grafana &&
    ls -l /var/lib/grafana/grafana.db'

Start the stack:

Configuration changehost— Brings the three services back onto the restored volumes

$ docker compose up -d && sleep 20 && docker compose ps

Alertmanager. Silences are re-created through the API rather than by replacing files, because the export carries fields the API assigns — id, status, updatedAt — and posting them back is not meaningful. Rebuild each one from the fields that describe intent, and set startsAt to now: you are creating this silence at this moment, for the remainder of the window it was meant to cover.

NOW=$(date -u +%Y-%m-%dT%H:%M:%S.000Z)

jq -c --arg now "$NOW" \
  '.[] | {matchers, startsAt: $now, endsAt, createdBy, comment}' \
  backup/silences.json \
| while read -r silence; do
    curl -sf -X POST -H 'Content-Type: application/json' \
      --data "$silence" \
      http://localhost:9093/api/v2/silences | jq -r '.silenceID'
  done

Stop the clock:

date -u +%H:%M:%S | tee backup/drill-end.txt

Task 8: Prove the restore, one component at a time

A stack that starts is not a stack that recovered. Each of these asks for something that could only be there if the copy worked.

Prometheus — a sample from before the destruction:

Read-only / Safehost— The same query as Task 2, at the same instant

$ T0=$(cut -d= -f2 backup/drill-t0.txt) && curl -sG http://localhost:9090/api/v1/query --data-urlencode 'query=up{job="prometheus"}' --data-urlencode "time=$T0" | jq -r '.data.result[] | "\(.metric.job) \(.value[1])"'

prometheus 1

Illustrative output

An empty result array is a failed restore, and it is worth being precise about why: the server is healthy, the query is valid, and the data simply is not there. Nothing in the stack will alert you about this. Only the assertion does.

Grafana — the dashboard that lived only in the database:

curl -sf -u admin:lab-not-a-secret \
  http://localhost:3000/api/dashboards/uid/lab-restore-proof \
  | jq -r '.dashboard.title'

Then confirm the other half of the split: the datasource is back too, but for a different reason — it came from the provisioning tree, not from the database.

curl -sf -u admin:lab-not-a-secret \
  http://localhost:3000/api/datasources | jq -r '.[] | "\(.name) \(.type)"'

Alertmanager — the change-window silence:

curl -sf http://localhost:9093/api/v2/silences \
  | jq -r '.[] | "\(.status.state)  \(.comment)"'

Write the drill record. This is the artefact the next reviewer reads, and the one the lesson’s RestoreDrillOverdue alert exists to demand:

{
  echo "drill: observability stack full restore"
  echo "start: $(cat backup/drill-start.txt)"
  echo "end:   $(cat backup/drill-end.txt)"
  echo "restored: prometheus tsdb, grafana db, alertmanager silences"
  echo "verified: sample at T0, dashboard uid lab-restore-proof, 1 silence"
} | tee backup/drill-record.txt

Validation

backup/prometheus-snapshot is non-empty and du -sh reports a plausible size for the data the TSDB held.
The PRAGMA integrity_check in Task 4 printed ok.
backup/silences.json contains the silence with its comment.
After Task 6, docker volume ls | grep obs-backup returned nothing.
The Task 8 Prometheus query returns a sample at T0 with the same value it returned in Task 2.
api/dashboards/uid/lab-restore-proof returns Restore proof.
api/datasources lists the provisioned Prometheus datasource, and you can say which of the two came from Git and which from the database.
The restored silence is listed with its original comment and an endsAt still in the future.
backup/drill-record.txt exists and contains a start time, an end time and what was verified.

Expected Outcome

obs-backup-lab/
├── alertmanager.yml
├── backup/
│   ├── drill-end.txt
│   ├── drill-record.txt
│   ├── drill-start.txt
│   ├── drill-t0.txt
│   ├── grafana.db
│   ├── prometheus-snapshot/
│   └── silences.json
├── dashboard.json
├── docker-compose.yml
├── prometheus.yml
├── scope.md
└── provisioning/
    └── datasources/
        └── prometheus.yml

Three services running on volumes that were empty twenty minutes ago, serving data that predates their creation. A drill record with a measured recovery time. A scope list that says what is in Git, what is in the state backup, and what is not backed up on purpose.

Troubleshooting

The snapshot POST returns 404 or “admin APIs disabled”. --web.enable-admin-api is missing from the Prometheus command in the Compose file. Add it, docker compose up -d prometheus, retry.

docker compose cp says no such file. The $SNAP variable is empty because the POST failed or jq found no .data.name. Re-run the snapshot and echo $SNAP before using it.

apk add sqlite fails. The helper container has no route to the Alpine mirrors. This is the one step that needs internet; on an air-gapped host, stop grafana-server first and copy grafana.db with cp instead — a stopped server makes the copy consistent, at the cost of an outage the online backup would have avoided.

integrity_check prints anything other than ok. Do not ship it. Take it again; if it fails twice, the source database is damaged and the backup is doing its job by telling you.

Prometheus restarts in a loop after the restore. Almost always ownership. docker compose logs prometheus will name a path it cannot write. Re-run the chown -R 65534:65534 helper.

Grafana starts but the dashboard is missing and the login fails. The database was not restored, so Grafana created a fresh one on first start with the password from the environment. Bring the stack down without -v, re-run the Grafana restore helper, and start again.

The silence POST returns 400. The JSON lost its shape somewhere in the pipeline. Print one object first — jq -c '.[0] | {matchers, startsAt, endsAt, createdBy, comment}' backup/silences.json — and post that by hand to see the message.

The Task 8 query returns an empty result but Prometheus is healthy. The snapshot copy landed one directory too deep. Check with docker compose exec -T prometheus ls /prometheus: the block directories belong directly there, not inside a prometheus-snapshot/ subdirectory.

Cleanup

Data-loss riskhost— Removes the three containers, the network and the three named volumes

$ cd "$HOME/obs-backup-lab" && docker compose down -v

docker volume ls | grep obs-backup || echo 'volumes gone'
ss -ltnp 2>/dev/null | grep -E ':(9090|3000|9093)\b' || echo 'ports free'

rm -rf takes the backup directory too

The block below deletes ~/obs-backup-lab and everything under it, including backup/ — the snapshot, the database copy and the drill record. The copy in front of it keeps the drill record and the scope list. List the directory before you delete it:

ls -la "$HOME/obs-backup-lab" "$HOME/obs-backup-lab/backup"

Data-loss riskhost— Keeps the drill record, then deletes the lab directory

$ mkdir -p "$HOME/obs-lab-deliverables" && cp -a "$HOME/obs-backup-lab/backup/drill-record.txt" "$HOME/obs-lab-deliverables/backup-drill-record.txt" && rm -rf "$HOME/obs-backup-lab"

docker image rm prom/prometheus:v2.55.1 grafana/grafana:11.3.0 prom/alertmanager:v0.28.1 alpine:3.20

Production notes

Take the copy through the component’s own primitive. The snapshot API for Prometheus, the SQLite online backup or pg_dump/mysqldump for Grafana, the v2 API for silences. Every one of those exists because a file copy of live state is unreliable, and each of them is one command.
Verify before you ship. PRAGMA integrity_check on the database, and a size and listing check on the snapshot. Verifying after the object reaches the bucket is verifying the wrong thing.
Independence is the property that matters. A second copy on the same volume survives nothing. A second bucket in the same account survives a disk but not a compromised credential. Match the copy to the failure you are actually defending against, and be honest that one copy cannot cover operator error, host loss and hostile action at once.
A backup job needs its own alert. Instrument the job with the age of its last success and alert on the age, not on the exit code — a job that stopped running emits no failures at all. The same applies to the drill: an overdue restore drill is a page-worthy condition.
Restore onto the version you backed up from. A TSDB snapshot is an internal format. If your recovery plan involves a newer Prometheus, rehearse that combination rather than discovering it during the outage.
Schedule the drill, and rotate who runs it. A drill only the author can perform is a single point of failure with a pulse. The runbook should be followable by whoever is on-call, and the drill is how you find out whether it is.

What You Learned

A backup is a recovery path that has been exercised. You destroyed three volumes and rebuilt from copies, which is the only evidence that counts.
Copy through the API, not through the filesystem. The snapshot endpoint and the SQLite online backup both exist because live state does not survive cp.
Verify the copy at the moment you take it. The integrity check turns a silent corruption into a loud failure, on the day it happens rather than on the day you need the file.
Config and state are two different backups. The datasource came back from the provisioning tree; the dashboard came back from the database. Knowing which is which is what makes the restore ordered rather than hopeful.
Silences are state, and the notification log is not. One is an acknowledgement a human made and would have to make again; the other is a de-duplication decision with a short shelf life.
Ownership is half of every restore failure. Uid 65534 and uid 472 are not interchangeable, and a data directory the process cannot write to produces a service that starts and then fails later.
You now have a measured RTO, and you know exactly which parts of a real recovery it leaves out.

Objectives

Prerequisites