critical riskcluster affecting~45 min

Runbook: Observability Platform Outage During Incident

1 · Prerequisites

Confirm every item is in place before any state change.

Major Incident Observability Overview
Observability During the Incident
Meta-Monitoring Basics
Separate Monitoring Platform
Meta Alerts
Avoiding Circular Assumptions
Evidence Preservation
A fallback pack that already exists: replica endpoint, read-only API token, exported incident dashboard JSON, printed symptom-to-metric map
A production incident already in progress, with a named incident commander

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· The production incident has a commander and the platform outage has a different owner. One person cannot hold both, and the characteristic failure of this runbook is a response team that quietly stops working the customer incident because the platform problem is more tractable and feels like progress
· Which of the three failures this is has been established before anything is restarted: the platform is down, the platform is up and lying (rendering stale, empty or partial data as though it were current), or the platform is healthy and production really is that bad. The third case is the expensive misdiagnosis, and the cheapest test that separates them is a direct scrape of one target's own metrics endpoint
· The fallback pack is located, not assumed. The replica endpoint and its read-only token, the exported dashboard JSON on the on-call laptop, and the printed symptom map are open in front of you before the ladder is descended. A fallback nobody can find during the incident is a document, not a fallback
· The replica has been confirmed to sit on a network path the primary failure does not share. If the same partition that isolated the primary also isolated the replica, that rung of the ladder is gone and the minutes spent retrying it are minutes not spent on the customer
· Every other team has been told, in the channel they actually read, that the platform is blind. Their alerting runs through this stack; while it is degraded, silence is not health, and they will otherwise read the absence of pages as an all-clear
· Whether the meta path is also affected has been established. It decides whether you will be paged about the next platform component to fail or will have to notice it yourself, and it is the difference between one outage and a sequence of surprises
· The incident channel and the decision log are being written somewhere that does not depend on this platform. The evidence a post-incident review needs is being generated right now, and the place it is usually kept is one of the things that is down
· No configuration change, upgrade, retention change or cardinality intervention is made to the platform while the production incident is live, unless it is the fix for the platform outage itself and its blast radius has been said out loud to the incident commander

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Declare the platform outage as its own incident with its own owner, and keep the production incident's commander on the production incident. Two incidents, two owners, one shared channel
2Announce the blindness explicitly to every team whose alerting traverses this stack, and state how long you expect it to last. "Alerting is down, quiet does not mean healthy" is the message; nothing softer survives being read at 03:00
3Classify the failure before touching anything: platform down, platform lying, or production genuinely that bad. Prove it from below by scraping one known-good target's own metrics endpoint directly — that is the one reading that does not pass through any component you are currently doubting
4Descend the fallback ladder one rung at a time, highest fidelity first: replica data source, then direct Prometheus HTTP API, then the exported dashboard JSON plus direct target scrapes, then the printed symptom map and a verbal read. Jumping straight to paper throws away instruments that are still working
5If Grafana is alive and the primary data source is not, re-point the response team at the replica. It is a single data source change and it restores the whole dashboard layer at once, which is why it is the first rung
6If Grafana is the failed component, query the replica's HTTP API directly with the read-only token and the query templates from the on-call playbook. The queries are the same ones the dashboard would have run; only the rendering is gone
7If no query path survives, open the exported dashboard JSON for the panel layout and the expressions it contained, then read raw counters from the targets themselves. You lose history and get current state, which is usually the half you need during mitigation
8Run the platform repair as a second workstream. Treat "restart the primary" as a change with a stated cost, not a reflex: a restart replays the write-ahead log before the instance serves queries or evaluates rules, and on a large head that is a further blind window you have chosen to take
9If you have to choose what to bring back first, bring back the alert path before the query path. A platform that cannot draw graphs but can still page is degraded; a platform that draws graphs but pages nobody is silently absent
10Preserve evidence continuously while blind: the queries you ran, the values you read, the times you read them, and the decisions taken from them. Screenshots without URLs and timestamps are not evidence, and the platform that would normally have recorded this is the thing that is down
11When the platform returns, re-read the incident window before trusting it. A gap in the data is not a return to zero, a stale panel is not a recovery, and the graph you are about to put in the post-mortem may be a rendering artefact of the outage rather than a fact about production
12Reconcile: compare what the recovered platform says about the window against what you decided while blind, and write down every place they disagree. Those disagreements are the highest-value output of the whole incident
13Record the two timelines separately — platform blind from when to when, production impacted from when to when — and the fallback rung actually used. That last number is the only honest measure of whether the fallback pack works

4 · Verification

Confirm the procedure actually fixed the problem.

✓A reading of production's current state has been obtained from a path that does not traverse the failed component, and the response team agrees on what it says. Until that exists, every statement about the production incident is an inference from a broken instrument
✓The three-way classification is written down and agreed. "The platform is lying" and "production is that bad" produce opposite responses, and a team that has not agreed which one it is will run both
✓The fallback rung in use is named in the channel, so that everyone reading a number knows which instrument produced it and how stale it is
✓Prometheus answers /-/ready and count(up == 1) returns a number close to the expected target count rather than merely non-zero. A partially scraping instance is the case that looks recovered and is not
✓Rule evaluation is producing fresh results: /api/v1/rules shows the expected groups with recent evaluation timestamps. A Prometheus that scrapes but does not evaluate pages nobody
✓Alertmanager is reachable, answers /api/v2/status, and a deliberate test alert reaches a real receiver and is acknowledged by a human. Restored routing is untested routing
✓Grafana answers /api/health and both a provisioned dashboard and a known UI-built dashboard render. The first proves provisioning loaded; only the second proves the database is intact
✓The data gap is declared per signal, with times: which minutes of metrics, logs and traces are genuinely missing rather than merely slow to render. The person writing the post-mortem next week needs to tell a hole in the data from a fault in production
✓The production incident is still owned, still being worked, and its timeline is intact. If the platform outage consumed the response, that is the finding, and it belongs in both post-mortems

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶There is nothing to roll back in the reading path — the fallback ladder only observes. The rollback that matters is on the repair workstream, and every action there needs its undo named before it is taken
↶Re-pointing dashboards at the replica is reversible by re-pointing them back, but record that you did it. A response team reading replica data and a repair team reading primary data will disagree about the state of the world and neither will know why
↶Do not restart the primary to "see if it helps" while the production incident is live. WAL replay is a blind window whose length you cannot shorten once it has started, and a restart also discards whatever in-memory state you might have been able to inspect
↶Do not change the platform's configuration mid-incident to make a query cheaper or a dashboard render. That change is untested, unreviewed, and will be forgotten in the recovery; it becomes drift discovered weeks later by whoever cannot reproduce the old behaviour
↶Do not delete or reformat anything that might explain the platform failure. This is the one incident where the wreckage and the evidence are the same object
↶If a fallback rung was found to be broken during the incident, do not fix it during the incident. Write it down as a finding; a half-repaired fallback is worse than a known-absent one because somebody will trust it

6 · Escalation

When the runbook isn't enough, contact:

· Escalate immediately to the incident commander, and keep the platform's blindness visible to every other team for the whole duration. Their pages route through this stack and they are entitled to know that quiet is currently meaningless
· Escalate to the platform owner as a separate page, not as a note in the production incident channel. If the platform outage is only ever mentioned inside the production incident, it gets the attention left over, which is none
· Escalate if the fallback ladder has no surviving rung. A response with no instrument at all is a decision to mitigate on hypothesis, and that is a call for the commander to make explicitly rather than for the on-call engineer to slide into
· Escalate to the security owner if the platform outage is unexplained and coincides with the production incident. Two simultaneous failures are usually one cause, and one of the causes worth ruling out early is somebody removing your ability to watch
· Escalate to the vendor at the start if any tier is managed. A support case opened in the first ten minutes and not needed costs nothing; one opened after two hours of self-service has already cost the two hours
· Escalate to the compliance or data owner if the blind window covers regulated telemetry. A gap in audit-relevant logs is a reportable fact, and the clock starts when you know about it, not when the platform comes back

The checkout service is failing. The on-call engineer opens the dashboard and every panel is red — not red as in “the error rate is high”, red as in “no data”. Prometheus returns 503. Loki times out. Within ninety seconds the engineer has stopped investigating checkout and started investigating Grafana.

That drift is the actual hazard in this runbook, and it is not a competence problem. The platform outage is legible, bounded and yours to fix; the customer incident is ambiguous, unbounded and somebody else’s system. The response team moves toward the tractable problem without deciding to. Meanwhile the thing that is costing money carries on uninvestigated, and nobody notices because the usual signal that it is still costing money is the platform.

So this runbook is mostly about two things: keeping the two incidents separate, and descending the fallback ladder in order instead of jumping to the bottom of it.

When this runbook applies

A production incident is in progress and the observability platform is degraded, unreachable, or returning data you do not trust.
The platform is the only thing that appears broken, but you are mid-incident and cannot tell whether it is hiding production damage.
A page arrived, the dashboard is empty, and the response team is about to conclude that the page was a false positive.

When it does not

The platform is down and nothing else is. That is a component recovery, and it has its own runbook and its own unhurried pace. This one exists specifically for the case where the clock is running on something else.
The platform is gone as a whole and must be rebuilt. That is full DR. Come back here for the part about operating blind, but the rebuild sequence lives in the DR runbook.
The platform is fine and production is genuinely that bad. This is the misdiagnosis worth naming up front, because it looks identical from the dashboard. The section below is how you tell.

The first decision: down, lying, or telling the truth

Everything downstream branches on this, and it takes about two minutes.

Shape	What you see	What it means	First move
Platform down	Timeouts, 503s, connection refused	The instrument is gone	Descend the ladder
Platform lying	Panels render, data is stale, empty, or partial	The instrument is worse than gone	Descend the ladder and distrust everything read in the last N minutes
Platform truthful	Panels render, values are current and terrible	Production is that bad	Stay on the production incident

The second row is the dangerous one. A platform that is down announces itself. A platform that renders a dashboard full of zeroes looks exactly like a service that has stopped serving traffic — and a response team will mitigate against that reading, at speed, because a graph that has fallen to the floor is the most convincing artefact in the building.

The test that separates them does not go through any component you are currently doubting. Read the counter from the target itself.

Read-only / Safeground truth from below the platform

# Substitute your own values before running:
TARGET=192.0.2.41:9100
APP=192.0.2.55:8080

# Does the target still expose metrics at all?
curl -sf "http://$TARGET/metrics" | head -5

# Is the application still counting requests? Read it twice, 15s apart,
# and compare. A counter that moves means traffic is being served.
curl -s "http://$APP/metrics" | grep '^http_requests_total' | sort > /tmp/t1
sleep 15
curl -s "http://$APP/metrics" | grep '^http_requests_total' | sort > /tmp/t2
diff /tmp/t1 /tmp/t2

If the counters are moving and the dashboard says zero, the dashboard is lying and the platform is your problem. If the counters are flat, the platform may be perfectly healthy and reporting a genuine outage. That one diff is the cheapest thing in this document and it decides which incident you are actually in.

The fallback ladder

Four rungs, in descending fidelity. The discipline is the order: engage the highest rung that still works, not the one you remember best.

Rung	Path	Gives you	Costs you
1	Replica data source in Grafana	The whole dashboard layer, one config change	Replica retention is shorter; history may be truncated
2	Direct HTTP API against the replica	Any query the dashboard would have run	No rendering; you read numbers, not shapes
3	Exported dashboard JSON plus direct target scrapes	Panel layout, the expressions, current raw counters	No history at all — current state only
4	Printed symptom map plus a verbal read	Thresholds, the escalation path, phone numbers	Institutional memory, no data

Rung 1 is a single Grafana data source change and it restores everything at once, which is why it is worth thirty seconds of checking before you give up on it. Rung 3 is the one people skip: current raw counters are usually the half you need during mitigation, because mitigation is about what is happening now rather than what happened at 02:14.

Read-only / Safeis the replica alive and is it current

# Substitute your own values before running:
REPLICA=http://prometheus-replica.example.com:9090

curl -sf "$REPLICA/-/ready" && echo ready

# How many targets is it actually scraping right now?
curl -s -G "$REPLICA/api/v1/query" --data-urlencode 'query=count(up == 1)' \
| jq -r '.data.result[0].value[1] // "0"'

# And how far behind is it? Compare the newest sample's timestamp
# against the wall clock before trusting any number it returns.
curl -s -G "$REPLICA/api/v1/query" \
--data-urlencode 'query=time() - timestamp(up)' \
| jq -r '.data.result[0].value[1] // "no data"'

A replica that answers but is forty minutes behind is still useful, and it is dangerous if nobody says so. Announce the rung and its staleness together: “we are on the replica, data is current to 02:31” is a usable statement. “I’m looking at the graphs” is not.

This is the step with the worst effort-to-value ratio in the entire runbook — it costs one message and it is the one most often skipped.

Every other team’s alerting traverses this stack. While it is degraded, their silence carries no information, and human beings read silence as health. A team three floors away is currently deciding not to escalate something because they have not been paged, and the reason they have not been paged is you.

Say it plainly, in the channel they read, with an expected duration and a name attached. Then say it again when the duration turns out to be wrong.

Two incidents, two owners

Repairing the platform while an incident runs

The repair is constrained in ways it would not be at any other time, because the usual freedom to experiment costs blind minutes now.

A restart is a change with a price. On startup, Prometheus replays the write-ahead log into a new head before it serves queries or evaluates rules. On a large head that is a further stretch of blindness you have chosen to take, on top of the one you are already in. Sometimes it is the right call. It is never a reflex, and the incident commander should hear the estimate before it happens.

Configuration changes made now will not be reviewed. A retention value dropped to make a query complete, a scrape interval widened to reduce load, a rule group disabled to stop an evaluation stall: each is plausible in the moment and each becomes undocumented drift, discovered weeks later by someone who cannot work out why the platform behaves differently from its repository.

If you must make one, write it in the decision log as it happens, with the intent to revert. The log entry takes ten seconds and it is the only thing standing between an emergency fix and permanent drift.

Alert path before query path. If the repair forces a choice, restore paging first. A platform that cannot draw graphs is degraded and everyone knows it. A platform that draws graphs and pages nobody is silently absent, and the second failure is discovered by a customer.

Read-only / Safeis the alert path alive, end to end

# Substitute your own values before running:
PROM=http://prometheus.example.com:9090
AM=http://alertmanager.example.com:9093

# Are rules being evaluated, and how recently?
curl -s "$PROM/api/v1/rules" \
| jq -r '.data.groups[] | [.name, .lastEvaluation] | @tsv'

# Is anything actually firing right now?
curl -s -G "$PROM/api/v1/query" --data-urlencode 'query=ALERTS' \
| jq -r '.data.result | length'

# Is Alertmanager up, and is its cluster healthy?
curl -sf "$AM/-/healthy" && echo healthy
curl -s "$AM/api/v2/status" | jq '.cluster.status'

# What has it silenced? A silence set during an earlier maintenance
# window is a common reason a real page never arrived.
curl -s "$AM/api/v2/silences" \
| jq -r '[.[] | select(.status.state == "active")] | length'

That last check earns its place. A stack that is scraping, evaluating and routing correctly can still page nobody because an active silence is swallowing the alert, and during a dual incident that looks exactly like an Alertmanager failure.

Holding is a legitimate outcome

If the platform failure has an owner elsewhere — a cloud provider, a network team, a managed backend — waiting can beat intervening. A half-repaired platform that renders partial data is worse than one that is plainly down, because partial data gets believed.

Holding is only a decision when it has three things attached: a named owner, an explicit time at which it is revisited, and an announcement that the platform is blind for that duration. “The network team owns the VLAN, revisited at 04:00 by the platform on-call, alerting is down until then” is a decision. “Let’s give it a few minutes” is a drift.

Evidence, while you cannot record it automatically

The post-incident review will need the timeline, and the system that normally keeps it is the system that is down. So it is being written by hand right now whether anyone intends it or not.

Four things to capture as they happen, in a place that does not depend on the platform:

The query and its result together. A number without the expression that produced it cannot be re-derived later.
The time you read it, and the time it referred to. During a degraded window those are different, and the gap between them is frequently the most important fact in the review.
Which fallback rung produced it. A value from a replica forty minutes behind is a different kind of evidence from a live scrape.
The decisions taken, and what they were taken from. Especially the mitigations chosen on incomplete data, which is most of them.

Screenshots without a URL and a timestamp do not qualify. Neither does a graph pasted into a channel that will be trimmed by a retention policy before the review happens.

When the platform comes back, do not trust the window

A gap in the data is not a drop to zero, and a stale panel is not a recovery

The first thing everyone does when the platform returns is open the incident window and read it. That graph is not what it appears to be.

Ingestion stopped and restarted, so counters have a hole. Rate functions evaluated across a gap produce values that are arithmetically correct and operationally meaningless. Panels may render cached results from before the outage. A recovered Loki answers queries while its most recent chunks are still being flushed, so the last minutes of logs arrive late or not at all.

Read the window, then reconcile it against what you observed directly while blind. Every disagreement is either a data artefact or something you got wrong in the dark, and both are worth knowing. Putting the recovered graph straight into the post-mortem without that step is how a platform outage becomes a permanent false belief about production.

Common patterns

Symptom	Likely cause	Resolution
Dashboard renders, every panel is empty	Data source unreachable from Grafana, or Prometheus returning errors	Read a target’s metrics endpoint directly; if counters move, the platform is lying
Page fired, dashboard says nothing is wrong	The production Prometheus is the broken thing and the meta caught it	Treat the meta alert as true until disproved from below
Replica also unreachable	Shared network path, DNS, or cluster control plane	Drop a rung; record the shared dependency as a finding
Alerts stopped arriving, everything else is healthy	Active silence, or the notification path rather than the platform	Check active silences and Alertmanager cluster status before restarting anything
Recovered graph shows a clean drop to zero across the outage	Ingestion gap, not a production outage	Reconcile against direct readings taken while blind
Nobody worked the customer incident for forty minutes	The response converged on the tractable problem	Two owners, one channel — and name it in both post-mortems

Escalation

Escalate when:

Always, immediately, to the incident commander — and keep the platform’s blindness visible to every other team for the duration.
The platform outage needs its own owner and there is nobody free. That is a staffing decision, not an engineering one.
No rung of the fallback ladder survives. Mitigating with no instrument at all is a call the commander makes explicitly.
The platform failure is unexplained and simultaneous with the production incident. Two failures at once are usually one cause.
Any tier is vendor-managed. Open the case in the first ten minutes.
The blind window covers regulated telemetry. The compliance clock starts when you know.

Why this matters

Every other runbook in this course assumes you can see the platform. This is the one you read when you cannot, and its content is almost entirely about sequence and attention rather than commands — because the commands are easy and the failure is human.

Three things go wrong, reliably. The response team migrates to the platform problem because it is the one where effort produces visible progress, and the customer incident runs unattended. Nobody tells the other teams that silence has stopped meaning health, so they read the absence of pages as an all-clear for however long the outage lasts. And when the platform returns, everybody trusts the recovered graph of the window they were blind for, which is the one graph in the estate that is guaranteed to be wrong.

None of those three is prevented by a better platform. They are prevented by deciding the order in advance, keeping two owners on two incidents, and rehearsing the fallback pack often enough that the ladder is muscle memory rather than a document somebody has to find.