critical riskdata loss risk~240 min

Runbook: Full Observability DR

1 · Prerequisites

Confirm every item is in place before any state change.

RPO and RTO
Prometheus Loss
Grafana Loss
Loki Loss
Object Storage Loss
Observability During the Incident
Written per-signal RPO and RTO targets, and the date of the last drill that measured them
Credentials for the backup and replica estate that are reachable without the platform being rebuilt

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· The incident is declared and an incident commander is named before the first rebuild command. This is a multi-hour, multi-component recovery and the failure mode of running it as a solo repair is two people rebuilding the same component against different backups
· The shape of the loss is identified before anything is provisioned: compute loss (hosts or cluster gone, data intact), data loss (object store or volumes gone), credential loss (everything intact, nothing can reach it), or hostile action. Only one of these is unrecoverable from configuration, and the others are made worse by treating them as if they were
· The fallback observation path is in use. You are about to rebuild the system you would normally use to tell whether the rebuild is working — direct Prometheus API access, a replica, an exported dashboard, the printed symptom map
· The damaged state is frozen, not deleted. Volumes detached rather than reformatted, buckets left in place, snapshots taken of whatever survives. This is the only genuine rollback the procedure has
· The component versions the backups were taken from are known. A TSDB snapshot restored onto an older minor version can be refused outright, and discovering that mid-rebuild costs the whole restore step
· The secret material is reachable: scrape credentials, data source credentials, object-store keys, receiver keys for the paging integrations. Recovery stalls on credentials nobody can produce more often than it stalls on data
· The last Alertmanager silence export is located before Alertmanager is started. A recovered Alertmanager with no silences re-pages every alert somebody deliberately suppressed, to the people already handling this outage
· If hostile action is suspected, the security owner has approved the restore point. Restoring from a copy taken after the compromise restores the compromise, and the platform is where the evidence lives

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Declare the incident, name the commander, and announce that the platform is blind. Everyone else's alerting depends on this stack; teams need to know that silence is not health right now
2Freeze the wreckage before provisioning anything. Detach volumes rather than reformatting, leave damaged buckets in place, and snapshot whatever survives. If the rebuild goes wrong, the damaged original is the only thing you can go back to
3Classify the loss — compute, data, credentials, or hostile — and write the classification where the team can see it. Every step below branches on it, and a team that has not agreed on the classification will run two different recoveries in parallel
4Restore credentials and secret material first. Object-store keys, scrape credentials, data source credentials and the paging receiver keys are prerequisites for every subsequent step, and a rebuild that reaches Grafana before it reaches the secret store simply stops there
5Provision the hosts or cluster from the same configuration source that built the originals. Anything hand-built during recovery is drift you have chosen to introduce at the worst moment for reviewing it
6Bring Alertmanager back first, from the configuration in Git, and re-apply the exported silences before any rule evaluation resumes. Order matters: a routing tree without its silences turns recovery into a paging storm aimed at the recovery team
7Bring Prometheus back in two halves, and do the fast half first. An empty instance with the configuration from Git restores scraping and alerting in minutes — that is the platform's job. Restoring TSDB history from the snapshot is a separate, slower step that does not block paging and must not be allowed to
8Restore the TSDB snapshot onto the same minor version the snapshot came from, and preserve external_labels — particularly cluster and replica. A rebuilt instance with a different replica label produces duplicate series for every scrape and breaks deduplication downstream
9Bring Grafana back as provisioning first, database second. The Git provisioning tree restores data sources, provisioned dashboards and file-provisioned alert rules; the database restore adds the users, annotations and UI-built dashboards that exist nowhere else
10Rotate the Grafana admin credential and review API keys immediately after the database restore. The restored database carries the tokens that were valid at backup time, which is exactly the property you do not want if the cause was hostile
11Point Loki and Tempo at the surviving object-store copy in read mode, and confirm a canary query returns results before any writer is started. The object store is canonical for both; nothing about them is rebuilt from configuration
12Take an independent copy of the surviving object data before enabling compaction or retention. The compactor is the only component that deletes, and pointing a freshly configured one at your last remaining copy is the way a recovery becomes the second data-loss event
13Resume ingestion, then verify each signal separately against its own RPO and RTO target rather than against "the platform is back". A stack that answers HTTP on every port and returns no data for the last six hours is not recovered
14Declare the data-loss window per signal, in writing — metrics, logs, traces, dashboards, silences. Somebody will investigate this outage next week using this platform, and they need to know which gap is real loss and which is a rendering artefact
15Record the measured wall-clock time per tier against the target. That number is the only honest input to the next capacity or replication decision, and it is only available now

4 · Verification

Confirm the procedure actually fixed the problem.

✓Prometheus answers /-/ready, and count(up == 1) is close to the target count you expect from promtool check service-discovery — not merely non-zero
✓Rule evaluation is producing results: recording rules have fresh samples and /api/v1/rules shows the expected groups with recent evaluation timestamps. A Prometheus that scrapes but evaluates nothing pages nobody
✓Alertmanager is reachable, amtool check-config passes against the restored file, and the active silence count matches the export. Verify this before alerts start flowing, not after
✓A deliberate test alert routes end to end to a real receiver and is acknowledged by a human. The routing tree is configuration that has just been restored, and untested routing is the failure that hides until the next genuine page
✓Grafana serves the provisioned dashboards and a known UI-built dashboard. The first proves the provisioning tree loaded; only the second proves the database restore worked
✓A canary LogQL query and a canary trace lookup both return results from the surviving object store, for a time window that predates the incident
✓Each signal's recovered window is compared against its RPO target and the gap is written down. "Metrics lost 00:40 to 03:10; logs lost 00:40 to 01:05; traces in that window unrecoverable" is a verification result. "It looks fine" is not
✓Measured RTO per tier is recorded against the target, and any tier more than 50 percent over target is raised as a finding rather than absorbed
✓The damaged originals are still present and still frozen. Nothing is deleted until every check above has passed

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶There is no rollback for a restore — the only real one is the frozen wreckage from pre-check four, which is why that step exists and why nothing is reformatted until verification passes
↶Build the recovered platform on new endpoints and cut traffic over by DNS or load-balancer change rather than rebuilding in place. A cutover is reversible; an in-place rebuild is not
↶Keep the Alertmanager receivers muted, or pointed at a test receiver, until the routing tree has been verified. Cutting a half-verified routing tree straight into production paging is the change most likely to require an emergency undo
↶Never overwrite a surviving backup with a copy taken from the partially recovered platform. The backup you are restoring from may be the only intact copy, and a well-meant "let's take a fresh backup now" has destroyed more recoveries than the original faults
↶If the TSDB restore fails to start, stop and preserve the snapshot rather than retrying against the same directory. A restore loop that keeps writing into a half-populated data directory makes the next attempt harder to reason about
↶If retention or compaction was enabled against the surviving object copy before an independent copy existed, there is nothing to roll back — the deletion is done. Treat it as a second incident and say so
↶Any component brought up by hand during recovery is temporary until it exists in the configuration repository. An estate accumulates these silently and the next rebuild reproduces none of them

6 · Escalation

When the runbook isn't enough, contact:

· Escalate to the incident commander immediately and keep the platform's blindness visible to every other team. Their alerting runs through this stack; while it is down, quiet does not mean healthy and they must be told so explicitly
· Escalate to the security owner before choosing a restore point if hostile action is even suspected. Restoring from a copy taken after the compromise restores the compromise, and this platform is also where the evidence lives
· Escalate to the object-store or cloud owner the moment the loss looks like a bucket or region event. That is the one branch that cannot be rebuilt from configuration, and the recovery path is theirs, not yours
· Escalate to the vendor if any tier is managed. A managed backend has recovery options that are not exposed to you, and the support case should be opened at the start of the incident rather than after the self-service path has failed
· Escalate to the backup owner if the most recent copy predates the incident by more than the RPO target. That is a policy failure surfacing during a recovery, and the decision about what to accept is not the on-call engineer's to make alone
· Escalate to the data or compliance owner with the declared loss window as soon as it is known. Log and trace loss can be a regulatory event, and the clock on notifying starts when you know, not when you finish

Full observability DR is not “restore, but bigger”. It is the one recovery where the thing you have lost is the thing you would normally use to tell whether any of your actions are working. Every other runbook in this course assumes you can see the platform. This one is what you read when you cannot.

That changes the shape of the work in two ways. Sequence stops being a matter of taste and becomes the difference between a recovery and a second incident — a routing tree restored without its silences pages the whole company at the people already handling the outage. And speed stops being the objective. The frozen original you did not delete is worth more than twenty minutes saved, because it is the only rollback this procedure has.

When this runbook applies

The observability platform is gone or unusable as a whole: the cluster, the hosts, or the storage behind them.
The object store backing Loki, Tempo or long-term metrics is unreachable, deleted, or has lost objects.
A region or account holding the platform is unavailable.
A compromise has made the running platform untrustworthy.

When it does not

One component is down. A single Prometheus, a single Grafana, one Loki ingester: those have their own runbooks and their own much shorter recoveries. Escalating a component failure to a platform rebuild multiplies the outage by hand.
The platform is degraded but readable. If you can still query it, you have an instrument. Use it. This runbook trades data and time for a rebuild, and that trade is only correct when there is nothing left to read.
The data is intact and only access is broken. Credential loss looks identical to data loss from the outside and is fixed by restoring access, not by rebuilding. Section “the four shapes” below separates them, and it is the highest-value ten minutes in this document.

The four shapes of loss

Everything downstream branches here, so establish it before provisioning anything.

Shape	Evidence	Recovery	Data lost
Compute loss	Storage present and readable, hosts or cluster gone	Rebuild compute, reattach or restore state	Only the unflushed window
Data loss	Volumes or buckets deleted, corrupted, or objects missing	Restore from an independent copy	Everything the copy does not cover
Credential loss	Everything present, nothing can authenticate	Restore access. Do not rebuild	None
Hostile action	Unexplained deletions, unexpected access, altered state	Security chooses the restore point first	Depends on the chosen point

The expensive misclassification is credential loss treated as data loss. The cluster cannot read the bucket, somebody concludes the bucket is gone, and a rebuild begins against a replica while the original is sitting there intact. Now there are two divergent stores and no clean way back. The cheapest check is whether the bucket answers to a different identity.

Read-only / Safeis the store gone, or just unreachable from here

# Substitute your own values before running:
BUCKET=obs-loki-chunks-primary

# From the cluster's own identity
aws s3 ls "s3://$BUCKET/" --summarize | tail -3

# Then from an independent break-glass identity
AWS_PROFILE=obs-breakglass aws s3 ls "s3://$BUCKET/" --summarize | tail -3

# And confirm the replica is real, and how far behind it is
AWS_PROFILE=obs-breakglass aws s3api get-bucket-replication --bucket "$BUCKET"

If the break-glass identity can read the bucket, this is not a data-loss event and most of what follows does not apply.

The first action is not technical. Every other team’s alerting runs through this stack, and while it is down their silence means nothing. Say so explicitly, in the channel they read, before you start rebuilding.

Then switch yourself onto the fallback observation path: direct API access to whatever Prometheus survives, the exported incident dashboard, the replica, the printed symptom map. You are about to spend hours rebuilding the instrument, and you need something to measure with in the meantime.

Step 2 — freeze the wreckage

Step 3 — credentials before components

Recovery stalls on credentials more often than on data. Object-store keys, scrape credentials, data source credentials and the paging receiver keys are prerequisites for everything below, and a rebuild that gets as far as Grafana before discovering the secret store is also unreachable has spent its first hour on work it cannot finish.

Restore access first, verify each credential against the thing it is supposed to reach, and only then start provisioning.

Step 4 — Alertmanager first, and silences before rules

This is the ordering that surprises people. Alertmanager comes back before Prometheus, and its silences come back before any rule evaluation resumes.

Configuration changeconfig from Git, then the silences, then real receivers

# Substitute your own values before running:
AM=http://alertmanager-new.example.com:9093
SILENCES=/var/tmp/silences.json

# The routing tree comes from Git and must parse before it is loaded.
amtool check-config /etc/alertmanager/alertmanager.yml

# Re-apply each exported silence that was still active.
# Post only the writable fields; id, status and updatedAt are server-owned.
jq -c '.[] | select(.status.state == "active")
     | {matchers, startsAt, endsAt, createdBy, comment}' "$SILENCES" \
| while read -r s; do
    curl -sf -XPOST "$AM/api/v2/silences" \
      -H 'Content-Type: application/json' -d "$s" > /dev/null
  done

# Confirm the count matches the export before anything can fire.
curl -sf "$AM/api/v2/silences" \
| jq '[.[] | select(.status.state == "active")] | length'

A silence is a human decision, made during an incident, that lives only in memory. Restore the routing tree without them and every alert somebody deliberately suppressed — the maintenance window, the known-bad deploy, the noisy service already being worked on — fires simultaneously, at the team currently rebuilding the platform.

Keep the receivers muted or pointed at a test destination until you have verified the routing tree end to end. Restored routing is untested routing.

Step 5 — Prometheus in two halves, fast half first

Prometheus does two jobs and only one of them is urgent. An empty instance with the configuration from Git resumes scraping and alerting in minutes; that is the platform’s actual function. Restoring history from the snapshot is slower, and it must not be allowed to block the first half.

Service impact possiblefunction first

# Substitute your own values before running:
PROM=http://prom-new.example.com:9090

promtool check config /etc/prometheus/prometheus.yml
promtool check rules /etc/prometheus/rules/*.yml

systemctl start prometheus
curl -sf "$PROM/-/ready"

# Function restored means targets and rules, not a process that is up.
curl -s -G "$PROM/api/v1/query" --data-urlencode 'query=count(up == 1)' \
| jq -r '.data.result[0].value[1] // "0"'
curl -s "$PROM/api/v1/rules" \
| jq -r '.data.groups[] | [.name, .lastEvaluation] | @tsv'

Only then restore history, onto the same minor version the snapshot came from, and with external_labels unchanged.

Step 6 — Grafana: provisioning, then database, then rotate

Provisioning comes from Git and restores data sources, provisioned dashboards, and file-provisioned alert rules. The database restores users, annotations, and every dashboard somebody built in the UI and never exported — which, during an investigation, are usually the ones people actually open.

Verify both halves separately. A provisioned dashboard rendering proves Git loaded; only a known UI-built dashboard proves the database restore worked.

Then rotate. The restored database contains the API keys and the admin credential that were valid when the backup was taken. If the cause of this incident was hostile, restoring those tokens restores the attacker’s access along with your dashboards.

Step 7 — Loki and Tempo: read first, compactor last

Nothing about these two is rebuilt from configuration. The object store is canonical; the components are stateless readers in front of it. Point them at the surviving copy, confirm a canary query returns data for a window that predates the incident, and only then think about writers.

Read-only / Safeprove the surviving copy answers before trusting it

# Substitute your own values before running:
LOKI=http://loki-new.example.com:3100
TEMPO=http://tempo-new.example.com:3200

curl -sf "$LOKI/ready"

# A window that predates the incident, so this proves the store and not the new writes.
curl -s -G "$LOKI/loki/api/v1/query_range" \
--data-urlencode 'query={job="observability"}' \
--data-urlencode 'limit=5' \
--data-urlencode 'start=2026-08-17T00:00:00Z' \
--data-urlencode 'end=2026-08-17T01:00:00Z' \
| jq '.data.result | length'

curl -sf "$TEMPO/ready"

Holding is a legitimate outcome

If the loss is a region or a credential rather than a deletion, waiting can beat rebuilding. A rebuild against a replica while the primary still exists produces two divergent stores and no clean reconciliation, and the cost of that lands weeks later on whoever tries to query across the seam.

Holding is only legitimate with three things attached: a named owner, an explicit end time at which the decision is revisited, and an announcement that the platform is blind for that duration. “We are waiting for the region, decision revisited at 06:00 by the platform on-call, alerting is down until then” is a decision. “Let’s give it a bit” is not.

Verification, per signal and against a target

A stack that answers HTTP on every port and returns nothing for the last six hours is not recovered. Check each signal separately, against the target it has:

Signal	Recovered means	Against
Metrics	`up == 1` count close to expected, rules evaluating recently	Prometheus RPO and RTO
Alerting	A test alert routed to a real receiver and acknowledged	Paging RTO
Dashboards	Provisioned and UI-built dashboards render	Grafana RPO
Logs	A canary LogQL query returns pre-incident data	Loki RPO
Traces	A known trace ID resolves from the block store	Tempo RPO

Then write down the gap. “Metrics lost 00:40 to 03:10; logs lost 00:40 to 01:05; traces in that window unrecoverable” is a verification result that the person investigating this outage next week can use. “Looks fine now” sends them chasing a hole in the data as though it were a fault.

Record the measured wall-clock time per tier against the target too. A tier more than 50 percent over its target is a finding, and this is the only moment the real number is available.

Escalation

Escalate when:

Always, immediately, to the incident commander — and keep the platform’s blindness visible to every other team for the duration.
Hostile action is suspected. Security chooses the restore point before you choose it for them.
The loss looks like a bucket or region event. That is the branch that cannot be rebuilt from configuration.
Any tier is vendor-managed. Open the support case at the start, not after the self-service path has failed.
The most recent copy predates the incident by more than the RPO target. That is a policy failure surfacing mid-recovery, and accepting it is not the on-call engineer’s decision alone.
The declared loss window touches regulated data. The compliance clock starts when you know, not when you finish.