Observability During the Incident

The checkout service is failing. The on-call engineer opens Grafana. The dashboard renders. Every panel is red. The error rate is unknown. The latency is unknown. The dependency metrics are unknown. The only thing the engineer knows is that Grafana is red. The engineer refreshes. The dashboard is still red. The engineer opens Prometheus directly. The Prometheus UI returns 503. The Prometheus host is healthy; the API is failing. The engineer opens the Loki URL. The Loki URL is timing out. The engineer is now investigating the observability platform instead of the checkout service.

This is the meta-incident: the incident is about the platform that is supposed to be investigating the incident. The discipline is observability during the incident: the fallback paths that let the response team read the system’s state when the observability platform is the thing that is broken.

What it is

Observability during the incident is the discipline of operating the response when the observability platform is itself degraded or unavailable. The discipline is four pre-staged fallbacks:

• Pre-exported dashboards. The pinned incident dashboard is exported as a static JSON file on every on-call engineer’s laptop. The engineer opens the file in a lightweight viewer even when Grafana is unavailable.
• Replicated data sources. A read-only replica of Prometheus is available on a separate host. The replica is on a separate network path. The replica is not on the same shared storage as the primary.
• Paper-form runbooks. A printed copy of the symptom-to-metric map is in the on-call bag. The engineer reads the printed threshold values when the dashboard is unavailable.
• Direct API access. The on-call engineer has curl and the read-only API token. The engineer queries the Prometheus HTTP API directly, bypassing Grafana.

The fallback is not the primary path. The primary path is the Grafana dashboard. The fallback is the path that the response team uses when the primary path is unavailable. The discipline is rehearsing the fallback so that the response team knows it exists.

Why a sysadmin cares

The meta-incident is rare but expensive. The platform is the thing that the response team is using to investigate the incident; when the platform is the thing that is broken, the response team loses the most important tool. The team that has rehearsed the fallback path reaches the same answer in five minutes; the team that has not, in fifty.

The Google SRE Book’s chapter on monitoring distributed systems is explicit: the monitoring system must be simpler than the system it monitors. The corollary is that the monitoring system must be redundant. The redundancy is the fallback.

How it works

The fallback is four parallel paths, each capable of answering the same questions as the primary path.

   Primary Path and Fallbacks
   ==========================

   Primary:
   +-----------+    +-----------+    +-----------+
   | Source    | -> | Prometheus| -> | Grafana   |
   | (app)     |    | (primary) |    | dashboard |
   +-----------+    +-----------+    +-----------+

   Fallback 1: replicated data source
   +-----------+    +-----------+    +-----------+
   | Source    | -> | Prometheus| -> | Grafana   |
   | (app)     |    | (replica) |    | (replica) |
   +-----------+    +-----------+    +-----------+

   Fallback 2: direct API access
   +-----------+    +-----------+
   | Source    | -> | Prometheus|
   | (app)     |    | (replica) |
   +-----------+    +-----------+

   Fallback 3: pre-exported dashboard
   +-----------+    +-----------+    +-----------+
   | Source    | -> | Prometheus| -> | Static    |
   | (app)     |    | (replica) |    | JSON      |
   +-----------+    +-----------+    +-----------+

   Fallback 4: paper and phone
   +-----------+
   | printed   |
   | runbook   |
   +-----------+

The four paths are exercised in order. The primary path is the default. Fallback 1 is engaged when the primary path is unavailable. Fallback 2 is engaged when Grafana is unavailable but the data source is. Fallback 3 is engaged when the data source is unavailable but a recent snapshot is on the engineer’s laptop. Fallback 4 is engaged when the data source is unavailable and the engineer’s laptop is unavailable.

The discipline is the order. The fallback is not “switch directly to paper”. The fallback is “first try the replica, then try the API, then try the snapshot, then try the paper”. The order preserves the highest-fidelity tool that is available.

How to configure it

The configuration is the replica Prometheus, the off-host snapshot, the printed runbook, and the on-call engineer’s local copy of the pinned dashboard.

The replica Prometheus is a second Prometheus host with the same scrape configuration but a different external_labels and no Alertmanager wiring.

# prometheus replica - prometheus.yml
global:
  scrape_interval: 30s
  evaluation_interval: 30s
  external_labels:
    replica: 'true'
    replica_region: 'eu-west-1'

rule_files: []

alerting:
  alertmanagers: []

scrape_configs:
  - job_name: 'checkout'
    static_configs:
      - targets: ['checkout-1:9100', 'checkout-2:9100']
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance
      - source_labels: [__address__]
        regex: '([^:]+):.*'
        target_label: instance
        replacement: '${1}'

The replica scrapes the same targets as the primary but does not write to the same TSDB. The replica is on a separate host with a separate network path. The discipline is that the replica is not a hot standby; the replica is a cold mirror that can answer read queries when the primary is unavailable.

The off-host snapshot is a daily script that copies the pinned dashboard JSON to the on-call engineer’s laptop.

# SEVERITY: READ-ONLY
DASHBOARD_UID="${DASHBOARD_UID:-sev1-main}"
OFFSITE_DIR="${HOME}/.cache/incident-dashboards"

mkdir -p "${OFFSITE_DIR}"
curl -s -u "${GRAFANA_USER}:${GRAFANA_API_KEY}" \
  "https://grafana.example.com/api/dashboards/uid/${DASHBOARD_UID}" \
  | jq '.dashboard' \
  > "${OFFSITE_DIR}/${DASHBOARD_UID}.json"

git -C "${HOME}/.cache/incident-dashboards" \
  add "${OFFSITE_DIR}/${DASHBOARD_UID}.json"
git -C "${HOME}/.cache/incident-dashboards" \
  commit -m "snapshot: ${DASHBOARD_UID} $(date -Iseconds)"

The Git repo is on the engineer’s laptop, not on the shared file server. The discipline is that the snapshot is on the device that the engineer is holding; the snapshot is available even when the shared file server is the thing that is broken.

The printed runbook is a quarterly print job. The runbook carries the symptom-to-metric map, the on-call escalation path, and the contact information for the disaster-recovery vendor.

# Paper runbook (excerpt)

Symptom: checkout 5xx rate above 5%
Metric:  sum by (service) (rate(http_requests_total{status=~"5.."}[5m]))
        /
        sum by (service) (rate(http_requests_total[5m]))

Threshold: 0.05

First action: open Sev1 dashboard, check dependency
             panel for payment-svc latency.

Fallback:   curl -s http://prometheus-replica:9090/api/v1/query \
              --data-urlencode "query=<expr>"

Contact:    alice (IC), bob (deputy), charlie (vendor)

The contact information is the cell phone numbers of the IC and the deputy. The contact information is the human-fallback when the system fallback is unavailable.

The direct API access is the on-call engineer’s curl with the read-only API token. The token is a read-only service account, not the admin token.

# SEVERITY: READ-ONLY
curl -s -H "Authorization: Bearer ${PROM_TOKEN}" \
  "http://prometheus-replica:9090/api/v1/query" \
  --data-urlencode "query=sum by (service) (rate(http_requests_total[5m]))" \
  | jq '.data.result | length'

The query is a copy of the template from the on-call playbook. The template is the same query the analyst would have run in Grafana; the engineer runs the query against the replica because the primary is unavailable.

How to validate it

Five checks before the meta-incident is real.

1. The replica Prometheus is scraping.

# SEVERITY: READ-ONLY
curl -s "http://prometheus-replica:9090/api/v1/query?query=up" \
  | jq '.data.result | map({job: .metric.job, instance: .metric.instance, value: .value[1]})'

Expected output:

[
  {"job": "checkout", "instance": "checkout-1", "value": "1"},
  {"job": "checkout", "instance": "checkout-2", "value": "1"}
]

A non-empty value: 1 array means the replica is scraping. An empty array means the replica is not scraping; the network path is broken.

2. The off-host snapshot is recent.

# SEVERITY: READ-ONLY
stat -c '%y' "${HOME}/.cache/incident-dashboards/sev1-main.json"

A timestamp within the last 24 hours is the discipline. A timestamp older than seven days means the snapshot is stale; the engineer cannot trust the panel layout.

3. The paper runbook is current.

The print date is on the inside cover. The current date is the print date within the last 90 days. A runbook older than one quarter is a stale runbook; the contact information may have changed.

4. The read-only API token is valid.

# SEVERITY: READ-ONLY
curl -s -H "Authorization: Bearer ${PROM_TOKEN}" \
  "http://prometheus-replica:9090/api/v1/status/runtimeinfo" \
  | jq '.status'

Expected output:

{"status": "success"}

A 403 or 401 means the token is revoked or expired. The on-call engineer cannot query the replica.

5. The game day rehearsed the fallback.

The game day is a quarterly drill. The drill simulates the primary Prometheus being unreachable; the response team switches to the replica within the agreed time budget. The drill is the validation.

How it can fail

Five failure modes recur in production observability fallbacks.

1. Replica is on the same network path. The replica Prometheus is on the same network path as the primary. Symptom: when the primary is unreachable, the replica is also unreachable. The fallback is a paper fallback.
2. Snapshot is on the same file server. The dashboard JSON is on the same shared file server as the Grafana state. Symptom: when the file server is unreachable, the snapshot is also unreachable.
3. API token is admin-scoped. The on-call API token can write to the data source. Symptom: the engineer accidentally wipes a recording rule while trying to read a query.
4. Runbook is stale. The paper runbook was printed six months ago. The contact numbers are out of date. Symptom: the engineer calls an empty number; the deputy is unreachable.
5. Fallback is unrehearsed. The team has never switched to the replica. Symptom: the response team spends fifteen minutes deciding which fallback to use; the meta-incident is the dominant cost.

How to troubleshoot it

When the primary path is unavailable, the order is:

1. Confirm the primary is down. Run curl against the primary Prometheus API. A 503 or timeout is the signal.
2. Switch to the replica. The IC announces the switch; the response team re-points their dashboards to the replica. The replay is a single Grafana data source change.
3. Fall back to direct API. If Grafana is unavailable, the analyst queries the replica directly via curl. The query template is the on-call playbook.
4. Fall back to the snapshot. If the data source is unreachable, the engineer reads the snapshot JSON on the laptop. The snapshot is the panel layout reference; the engineer reads the layout, then calls the on-call deputy for a verbal data read.
5. Fall back to paper. The engineer reads the printed runbook. The runbook is the institutional memory.

Security implications

The fallback is a security boundary. The replica Prometheus is a read-only mirror; the replica should not be writable by the on-call engineer. The replica’s API token should be a read-only service account, not the admin token.

The snapshot on the engineer’s laptop is a sensitive artefact. The snapshot contains the dashboard JSON, which includes the panel labels and the templating variables. The snapshot does not contain the panel data, but the labels are sufficient to reconstruct the schema. The laptop must be encrypted; the snapshot must be in the encrypted home directory.

The paper runbook is also a sensitive artefact. The runbook contains the contact numbers of the IC and the deputy. The runbook is in the on-call bag; the bag is not left unattended. The runbook is shredded when superseded.

The direct API access is the highest-risk fallback. The API token is a credential. The token is held in the on-call engineer’s password manager; the password manager is unlocked only when the engineer needs to read the token. The token is rotated on the same cadence as the on-call roster.

Performance implications

The replica Prometheus is a second host that scrapes the same targets as the primary. The replica is a second network and storage cost. The discipline is to size the replica for read-only queries, not full retention. A replica with seven days of retention is sufficient for the meta-incident; the replica is not a long-term archive.

The off-host snapshot is a one-off operation. The snapshot is a few hundred kilobytes. The snapshot is committed to the on-call engineer’s laptop repo; the repo is a small Git repository.

The paper runbook is a printed document. The document is small (a few pages). The document is reprinted quarterly.

The direct API access is bounded by the replica’s query concurrency. The replica is configured with the same max-concurrency and timeout as the primary. The discipline is that the analyst respects the same budgets on the replica as on the primary.

Production guidance

• The fallback is rehearsed quarterly. The game day simulates the primary path being unavailable; the response team switches to the replica within the agreed time budget.
• The replica is on a separate network path. The replica is not a hot standby on the same network; the replica is a deliberate redundancy.
• The snapshot is on the engineer’s laptop. The snapshot is not on the shared file server; the snapshot is on the device that the engineer is holding.
• The runbook is printed quarterly. The print date is on the inside cover; the contact information is reviewed at every print.
• The API token is read-only. The token is a service-account credential scoped to the read role; the token is rotated on the same cadence as the on-call roster.

Verification

You should now be able to answer:

• What is the meta-incident, and how does it differ from the underlying application incident?
• What are the four fallback paths, and what is the order in which they are engaged?
• Why must the replica Prometheus be on a separate network path, not a hot standby on the same network?
• What five failure modes recur in observability fallbacks, and how do you diagnose each one?
• What is the operational discipline that turns the fallback pack from a document into a rehearsed response?

Quiz