critical riskcluster affecting~60 min

Runbook: Restore Observability Configuration

1 · Prerequisites

Confirm every item is in place before any state change.

Backup Basics
Prometheus Loss
Grafana Loss
Prometheus Backup
Grafana Backup
Alertmanager Backup
Config as Code Basics
The output of the backup runbook: a verified snapshot, a database dump, and a silence export, each with a known age
The commit hash the platform was running before the loss, or an explicit acknowledgement that it was never recorded

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· What was lost is classified before anything is restored: configuration only (files gone or corrupt, the stores intact), state only (the TSDB or the Grafana database gone, the configuration intact), or both. The three have different procedures and the same symptom on a dashboard, which is why guessing here costs the whole restore
· The damaged state is moved aside, not deleted. Rename the data directory, copy the corrupt database somewhere else, keep the broken configuration file. This is the only rollback the state half of this procedure has, and it takes ten seconds to preserve and cannot be recreated afterwards
· The age of every artefact you are about to restore is known and written down: the snapshot timestamp, the database dump timestamp, the silence export timestamp, and the commit the configuration is being taken from. A restore assembled from four different points in time is a platform that was never running
· The commit the platform was actually running is identified from the change record, not assumed to be the branch head. Between the last good state and now, other people merged things, and checking out the head applies all of it as a side effect of a recovery
· The component versions match. A TSDB snapshot restored onto an older Prometheus minor version can be refused at startup, and a Postgres dump does not go into a SQLite Grafana at all. Confirm before the copy starts rather than after it finishes
· The target service is stopped before its state is replaced. Grafana holds the SQLite write-ahead log open and Prometheus owns its data directory; restoring underneath either one produces a file that looks correct and a process that will corrupt it on the next write
· The Alertmanager silence export is located before Alertmanager is started. A restored routing tree with no silences re-pages every alert somebody deliberately suppressed, to the people currently running the restore
· It is established whether the loss was accidental or hostile. If a backup was taken after a compromise, restoring it restores the compromise, and the credentials inside the Grafana database were valid at backup time and are being reinstated along with the dashboards

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Classify the loss — configuration, state, or both — and write the classification where the team can see it. Every branch below depends on it, and two people restoring against different classifications will produce a platform assembled from two incompatible halves
2Preserve the damaged original before touching anything. Rename the data directory rather than deleting it, copy the corrupt database aside, keep the broken configuration file. If the restore goes wrong, this is the only thing you can go back to
3Restore configuration from Git at the recorded commit, not at the branch head, and into a working tree rather than straight over the live files. Nothing is deployed yet; you are staging a candidate
4Validate the candidate configuration with the tools that will have to parse it — promtool check config, promtool check rules, amtool check-config — before it goes anywhere near a running process. A reload that rejects the file leaves the previous configuration running, so an invalid restore looks exactly like a successful one
5Deploy the validated configuration, then reload rather than restart wherever the component supports it. A reload costs nothing; a restart costs a write-ahead-log replay during which the instance serves no queries and evaluates no rules
6Restore Alertmanager next, and re-apply the exported silences before any rule evaluation resumes. Order matters here more than anywhere else in the procedure: a routing tree without its silences turns the recovery into a paging storm aimed at the recovery team
7Restore Prometheus in two halves and do the fast half first. Configuration from Git restores scraping and alerting in minutes, which is the platform's actual job; restoring TSDB history from the snapshot is slower and must not be allowed to block paging
8Restore the TSDB by stopping Prometheus, moving the damaged directory aside, copying the snapshot blocks into the data path, fixing ownership, and starting the service. Preserve external_labels exactly — a changed cluster or replica value produces a duplicate series for every scrape and silently breaks deduplication downstream
9Restore Grafana as two separate operations: the provisioning tree from Git, and the database from its dump. The first brings back data sources and provisioned dashboards; the second brings back users, annotations, and every dashboard somebody built in the UI and never exported
10Pin the provisioning tree and the database dump to the same point in time. Provisioning wins on conflict, so an older provisioning commit restored over a newer database overwrites data source UIDs, and every dashboard that references them renders as No data
11Rotate the Grafana admin credential and review the API keys immediately after the database restore. The dump carries the tokens that were valid when it was taken, which is precisely the property you do not want if the cause was hostile
12For Loki and Tempo, restore configuration only and point the components at the surviving object store. The chunks and blocks are the source of truth and are not rebuilt from a backup; confirm a canary query returns pre-incident data before any writer starts
13Verify each component against its own endpoint rather than against "the platform is back", then declare the data gap per signal in writing. A stack that answers HTTP on every port and returns nothing for the last six hours is not restored
14Record the measured restore time per component against the RTO target, and record the age of each artefact you used. Those two numbers are the only honest input to the next backup-policy decision and they are only available now

4 · Verification

Confirm the procedure actually fixed the problem.

✓The damaged originals still exist, renamed and untouched. Nothing is deleted until every check below has passed, and a restore that has already destroyed its predecessor has no verification failure it can survive
✓promtool check config and promtool check rules pass against the deployed files, not against the staging copy. The file that matters is the one on the path the process reads
✓Prometheus answers /-/ready, and count(up == 1) returns a number close to the expected target count rather than merely non-zero. A partially scraping instance is the failure that looks recovered
✓/api/v1/targets shows the expected jobs with health up, and /api/v1/rules shows the expected groups with recent evaluation timestamps. Rules restored from a different source than the scrape configuration is the classic half-restore, and it produces an instance that scrapes perfectly and alerts on nothing
✓promtool tsdb list against the restored data directory reports the blocks you expected, and a query for a time range that predates the incident returns samples. A restored TSDB that is intact and empty passes every check that only asks whether the process started
✓Alertmanager answers /-/healthy, amtool check-config passes against the deployed file, and the active silence count matches the export. Verify this before alerts start flowing rather than after
✓A deliberate test alert routes end to end to a real receiver and is acknowledged by a human. The routing tree has just been restored, and restored routing is untested routing
✓Grafana /api/health reports database: ok, the expected data sources are listed with their original UIDs, and both a provisioned dashboard and a known UI-built dashboard render. The first proves provisioning loaded; only the second proves the database restore worked
✓The Grafana admin credential has been rotated and the API keys reviewed. Until that happens, the restored instance is trusting whatever was valid on the day the dump was taken
✓The data gap is declared per signal, with times — "metrics lost 02:10 to 04:35, dashboards intact, silences restored, traces unaffected" is a verification result; "it looks fine" is not

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶The configuration half is genuinely reversible: the previous state is a commit, and reverting it plus a reload puts the platform back where it was. Take that path rather than hand-editing the deployed file
↶The state half has no undo except the copy you preserved in step two. If the damaged data directory was deleted rather than renamed, a bad restore is permanent and there is nothing in this section that helps
↶If the TSDB restore fails to start, stop and preserve the snapshot rather than retrying into the same directory. A retry loop that keeps writing into a half-populated data path makes every subsequent attempt harder to reason about, and the snapshot may be the only copy
↶Never overwrite a surviving backup with a copy taken from the partially restored platform. The artefact you are restoring from may be the last intact one, and a well-meant "let us take a fresh backup now" has destroyed more recoveries than the faults that started them
↶Keep the Alertmanager receivers muted, or pointed at a test destination, until the routing tree has been verified end to end. Cutting half-verified routing straight into production paging is the change most likely to need an emergency undo
↶If the restore was assembled from artefacts of different ages and the result is inconsistent, do not patch it forward component by component. Stop, decide which point in time is canonical, and restore the rest to match it
↶Anything hand-edited during the restore is temporary until it exists in the configuration repository. An estate accumulates these silently, and the next restore reproduces none of them

6 · Escalation

When the runbook isn't enough, contact:

· Escalate to the platform owner before restoring state onto a component that is still serving. Stopping Prometheus or Grafana to replace what is underneath it is a service-affecting change, and it should be a decision rather than a step somebody took because the runbook said so
· Escalate to the security owner before choosing a restore point if the loss is unexplained or hostile action is suspected. The Grafana database carries data source credentials and API keys, and restoring it reinstates whatever was valid at backup time
· Escalate to the backup owner if the most recent usable artefact is older than the RPO target. That is a policy failure surfacing during a recovery, and how much loss to accept is not the on-call engineer's decision alone
· Escalate if no commit hash was recorded for what the platform was running. The restore is then a best guess, it must be stated as one in the incident record, and everything downstream needs correspondingly more verification
· Escalate to the vendor or the object-store owner if the loss touches a managed tier. Managed backends have recovery options that are not exposed to you, and the case should be opened at the start rather than after the self-service path has failed
· Escalate to the data or compliance owner with the declared gap as soon as it is known. Missing log or trace history can be a reportable event, and the clock starts when you know rather than when you finish

Two different jobs share the word restore, and confusing them is the most expensive mistake in this procedure.

Restoring configuration means checking out a commit, validating it, and asking a process to reload. It is cheap, it is fast, and it is reversible — the previous state is another commit and Git still has it.

Restoring state means stopping a service, replacing what is under it, and starting it again. It is slower, it is service-affecting, and it is irreversible the moment you overwrite the thing you were trying to recover. Its only rollback is a copy of the damaged original that you either preserved or did not.

Almost everything in this runbook follows from that asymmetry. Do the cheap reversible half first, keep the damaged original until the expensive half has been verified, and never assemble the two halves from different points in time.

When this runbook applies

Configuration files, rule files, or the Grafana provisioning tree are missing, corrupt, or have been overwritten with something wrong.
A component’s local state is gone: the TSDB was wiped, the Grafana database is corrupt, an Alertmanager restart took the silences with it.
A change went out, made things worse, and the fastest way back is the previous known-good configuration.
A restore drill, which is the only reason to read this document calmly.

When it does not

The whole platform is gone. That is full DR: a rebuild in dependency order rather than a restore into surviving components. Come back here for the individual component procedures.
You are taking copies. The backup runbook is the counterpart to this one, and reading it during an outage costs time you do not have.
You are mid-incident on something else. If a production incident is running and the platform is degraded, the first job is operating blind, not restoring. Restore is what happens once somebody else owns the customer problem.

First: what did you actually lose?

Three shapes, one symptom. A dashboard full of No data looks identical in all three cases, and they have different procedures.

Shape	Evidence	Restore	Reversible
Configuration only	Stores intact, files missing or invalid, process refusing to start or scraping nothing	Git checkout, validate, reload	Yes — revert the commit
State only	Configuration valid, process healthy, queries return empty	Stop, replace the store, start	No — only the preserved original
Both	Neither the files nor the stores survived	Configuration first, then state	Partly

The middle row is the one people misread. A Prometheus whose data directory has been wiped keeps scraping happily, reports its targets as up, and answers every health check — and returns nothing for any query about yesterday. Nothing about the process is unhealthy. The instrument is simply empty.

Read-only / Safeseparate configuration loss from state loss in one pass

# Substitute your own values before running:
PROM=http://prometheus.example.com:9090
TSDB_PATH=/var/lib/prometheus/data

# Is the configuration valid and is it the configuration you think it is?
promtool check config /etc/prometheus/prometheus.yml
curl -s "$PROM/api/v1/status/config" | jq -r '.data.yaml' | head -20

# Is there anything in the store at all?
du -sh "$TSDB_PATH"
promtool tsdb list "$TSDB_PATH"

# Is it scraping now, and does it remember anything from before?
curl -s -G "$PROM/api/v1/query" --data-urlencode 'query=count(up == 1)' \
| jq -r '.data.result[0].value[1] // "0"'

Healthy configuration plus an empty block list is state loss. A refusal to parse is configuration loss. Both is both, and the order below still holds.

Step 1 — preserve the damaged original

Step 2 — configuration, from the right commit

The cheap half. Restore into a working tree first, validate it there, and only then put it on the path the process reads.

Configuration changestage the candidate, do not deploy it yet

# Substitute your own values before running:
CONFIG_REPO=/opt/observability-config
GOOD_COMMIT=4f2a9c1
STAGE=/var/tmp/restore-candidate

git -C "$CONFIG_REPO" fetch --all
git -C "$CONFIG_REPO" log -1 --format='%H %ci %s' "$GOOD_COMMIT"

rm -rf "$STAGE" && mkdir -p "$STAGE"
git -C "$CONFIG_REPO" archive "$GOOD_COMMIT" | tar -x -C "$STAGE"

# Validate with the tools that will have to parse it.
promtool check config "$STAGE/prometheus/prometheus.yml"
promtool check rules "$STAGE/prometheus/rules/"*.yml
amtool check-config "$STAGE/alertmanager/alertmanager.yml"

Then deploy and reload. Prefer reload over restart everywhere it is available: a reload is instantaneous, while a restart makes Prometheus replay its write-ahead log before it serves a single query or evaluates a single rule.

Service impact possibledeploy, reload, and confirm the reload actually took

# Substitute your own values before running:
STAGE=/var/tmp/restore-candidate
PROM=http://prometheus.example.com:9090

install -m 0644 -o prometheus -g prometheus \
"$STAGE/prometheus/prometheus.yml" /etc/prometheus/prometheus.yml
cp -a "$STAGE/prometheus/rules/." /etc/prometheus/rules/

# Validate the deployed path, not the staging copy.
promtool check config /etc/prometheus/prometheus.yml

# Reload requires --web.enable-lifecycle; SIGHUP works either way.
curl -sf -XPOST "$PROM/-/reload" && echo reloaded

# Confirm the running config is the one you just deployed.
curl -s "$PROM/api/v1/status/config" | jq -r '.data.yaml' | head -20

Step 3 — Alertmanager, and silences before rules

Alertmanager comes back before rule evaluation resumes, and its silences come back before it is allowed to notify anything.

Configuration changerouting tree from Git, then the exported silences

# Substitute your own values before running:
AM=http://alertmanager.example.com:9093
SILENCES=/var/tmp/silences.json

amtool check-config /etc/alertmanager/alertmanager.yml
curl -sf -XPOST "$AM/-/reload" && echo reloaded

# Re-apply each silence that was still active when the export was taken.
jq -c '.[] | select(.status.state == "active")' "$SILENCES" \
| while read -r s; do
    curl -sf -XPOST "$AM/api/v2/silences" \
      -H 'Content-Type: application/json' -d "$s" > /dev/null
  done

# The count must match the export before anything is allowed to fire.
curl -s "$AM/api/v2/silences" \
| jq -r '[.[] | select(.status.state == "active")] | length'

A silence is a human decision, made during an incident, held only in memory. Restore the routing tree without them and every alert somebody deliberately suppressed — the maintenance window, the known-bad deploy, the failover already being handled — fires at once, at the people currently doing the restore.

Keep the receivers muted or pointed at a test destination until the routing tree has been verified end to end.

Step 4 — Prometheus state, with the service stopped

The expensive half. Nothing here is reversible once the copy starts, which is why step one exists.

Data-loss riskstop, preserve, replace, fix ownership, start

# Substitute your own values before running:
TSDB_PATH=/var/lib/prometheus/data
SNAP=/var/tmp/restore/20260818T120000Z-4f2c1a9b8d3e5f60
PROM=http://prometheus.example.com:9090

# Confirm the snapshot is readable BEFORE stopping anything.
promtool tsdb list "$SNAP"

systemctl stop prometheus

# Preserve, do not delete. This is the rollback.
mv "$TSDB_PATH" "$TSDB_PATH.damaged-$(date +%Y%m%dT%H%M%S)"
mkdir -p "$TSDB_PATH"
cp -a "$SNAP/." "$TSDB_PATH/"
chown -R prometheus:prometheus "$TSDB_PATH"

systemctl start prometheus
curl -sf "$PROM/-/ready" && echo ready

Three things go wrong here often enough to name.

Ownership. A data directory copied as root starts a Prometheus that cannot write to it. The process comes up, logs a permission error, and the symptom on a dashboard is indistinguishable from a scrape failure.

Version. A snapshot from a newer minor version restored onto an older binary can be refused at startup, and you find that out at the end of a long copy rather than the beginning. Check first.

external_labels. If the restored instance comes up with a different cluster or replica value than the one that produced the data, every scrape adds a second series alongside the original, deduplication downstream stops working, and dashboards double-count. The instance looks completely healthy.

Step 5 — Grafana, as two operations pinned to one moment

The provisioning tree is configuration and comes from Git. The database is state and comes from a dump. They restore separately and they must agree with each other.

Data-loss riskgrafana must be stopped before its database is replaced

# Substitute your own values before running:
GRAFANA_DB=/var/lib/grafana/grafana.db
DUMP=/var/tmp/restore/grafana-db.sqlite
GRAFANA=http://grafana.example.com:3000

# Check the dump before trusting it: intact, and not intact-and-empty.
sqlite3 "$DUMP" "PRAGMA integrity_check;"
sqlite3 "$DUMP" "SELECT count(*) FROM dashboard;"

systemctl stop grafana-server

mv "$GRAFANA_DB" "$GRAFANA_DB.damaged-$(date +%Y%m%dT%H%M%S)"
install -m 0640 -o grafana -g grafana "$DUMP" "$GRAFANA_DB"

systemctl start grafana-server
curl -sf "$GRAFANA/api/health" | jq '.database'

Grafana holds the SQLite write-ahead log open while it runs. Replacing the database underneath a live process produces a file of the right size that breaks on the next write, so the failure surfaces minutes later and looks like corruption rather than like a restore mistake.

An integrity_check that returns anything but ok means the dump was taken with the wrong primitive. A dashboard count of zero means the dump is perfectly intact and empty, which restores a Grafana with no dashboards and passes every check that only asks whether the file exists.

Then rotate. The dump contains the admin credential and the API keys that were valid when it was taken, along with the data source credentials. Anyone holding a copy of that file can log in to the restored instance. Rotation is part of the restore, not a follow-up ticket.

Step 6 — Loki and Tempo: configuration only

Neither of these is restored from a backup. The object store is the source of truth for chunks and blocks; the components in front of it are stateless readers configured from Git.

So the restore is a configuration restore, and the verification is a canary query for a window that predates the incident — which proves the store answers, rather than proving that new writes are landing.

Read-only / Safeprove the surviving store answers before trusting the restore

# Substitute your own values before running:
LOKI=http://loki.example.com:3100
TEMPO=http://tempo.example.com:3200

curl -sf "$LOKI/ready" && echo ready

curl -s -G "$LOKI/loki/api/v1/query_range" \
--data-urlencode 'query={job="observability"}' \
--data-urlencode 'limit=5' \
--data-urlencode 'start=2026-08-17T00:00:00Z' \
--data-urlencode 'end=2026-08-17T01:00:00Z' \
| jq '.data.result | length'

curl -sf "$TEMPO/ready" && echo ready

Do not enable retention or compaction as part of a restore. The compactor is the only component that deletes, and during a recovery the bucket it would act on is frequently the last surviving copy.

Verify per component, then declare the gap

Component	Restored means	Not merely
Prometheus config	`promtool check config` passes on the deployed path and `/api/v1/status/config` matches	The process is running
Prometheus state	`promtool tsdb list` shows the expected blocks and a pre-incident query returns samples	`/-/ready` returns 200
Rules	`/api/v1/rules` lists the expected groups with recent evaluation timestamps	The rule files exist on disk
Alertmanager	`amtool check-config` passes, the silence count matches the export, a test alert reaches a human	The port answers
Grafana	`/api/health` reports `database: ok`, UIDs match, and a UI-built dashboard renders	The login page loads
Loki / Tempo	A canary query returns pre-incident data	`/ready` returns 200

Then write down the gap, per signal, with times. “Metrics lost 02:10 to 04:35; dashboards intact; silences restored from the 02:05 export; traces unaffected” is a verification result somebody can act on next week. “Restored successfully” sends the next investigator chasing a hole in the data as though it were a fault in production.

Record the artefact ages too — how old the snapshot was, how old the dump was, how old the silence export was. Those numbers are the honest input to the next backup-policy conversation, and this is the only moment they are available without guessing.

Common patterns

Symptom	Likely cause	Resolution
Prometheus healthy, all queries empty	State loss, not configuration loss	`promtool tsdb list`; restore the snapshot with the service stopped
Restore deployed but behaviour unchanged	The reload was rejected and the old configuration is still running	Read back `/api/v1/status/config`; validate before deploying
Prometheus starts, then logs permission errors	Data directory copied as root	`chown` the restored path to the service user
Every Grafana panel shows No data after a clean restore	Provisioning restored from a different commit than the database; data source UIDs changed	Pin both to one point in time; confirm UIDs
Grafana corrupts minutes after the restore	The database was replaced while the service was running	Stop, restore from the preserved copy, restore again properly
Alerts flood the on-call immediately after recovery	Silences were not re-applied before notification resumed	Re-apply the export; keep receivers muted until verified
Duplicate series and double-counted dashboards	`external_labels` changed on the rebuilt instance	Restore the original `cluster` and `replica` values
Nothing to fall back to when the restore failed	The damaged original was deleted rather than renamed	Nothing. This is why step one exists

Escalation

Escalate when:

Stopping a serving component to replace its state. That is a service-affecting change and it needs an owner who agreed to it.
The loss is unexplained or hostile. The Grafana database reinstates credentials that were valid at backup time.
The newest usable artefact is older than the RPO target. How much loss to accept is not the on-call engineer’s call alone.
No commit hash was recorded. The restore is a best guess and must be recorded as one.
A managed tier is involved. Open the vendor case at the start.
The declared gap touches regulated telemetry. The compliance clock starts when you know.

Why this matters

The reason this runbook spends so long on classification and ordering is that the commands themselves are short. Restores fail in four ways, and none of them is a hard technical problem.

The damaged original was deleted before the replacement was verified, so a failed restore had nowhere to go. Configuration and state were assembled from different points in time, producing a platform that never existed and behaves accordingly. A reload was rejected and nobody read the running configuration back, so the restore was declared complete while the old file was still in use. And the silences were left out, which turns the last five minutes of a successful recovery into a paging storm aimed at the people who just performed it.

All four are decided in the first ten minutes, by whether you classified the loss, preserved the original, and pinned everything to one moment. The copying is the easy part, and it is not the part that goes wrong.