Major Incident Observability Overview

A major incident arrives at 03:00. Checkout is failing for roughly one in three attempts. The on-call engineer is awake, a Slack channel has formed, and the noise is rising. The first Grafana load takes nine seconds. The on-call engineer opens the wrong dashboard. Three minutes of investigation produce nothing but a wall of green panels from an unrelated service. The customer-impact number on the status page is unowned.

This is the situation major incident observability exists to prevent. The point is not to gather more metrics. The point is to have a shared, current, navigable view of the system that the whole response team can read at the same time, and a discipline that keeps that view current.

What it is

A major incident is a user-visible, time-critical service degradation that exceeds the response capacity of a single on-call engineer and requires a coordinated response. A “Sev1” sounds similar, but the operational definition is sharper: the response team is now multi-person, the cost of confusion is no longer paid by the engineer alone, and the output of the response is judged against a customer-impact metric, not a service-health metric.

Major incident observability is the discipline of operating the observability platform during that response. It is distinct from the day-to-day work of building dashboards, recording rules, and alerts. The system that works for steady-state monitoring is not the same system that works at minute twelve of a Sev1. The discipline is the second one.

Why a sysadmin cares

The cost of poor observability during a major incident is not the incident itself. It is the multiplier between time of detection and time of mitigation. A team that has a pinned dashboard, a recognised commander, and a standing evidence ledger will reach mitigation in minutes. A team that does not has to negotiate coordination before it can negotiate the problem. The Atlassian Incident Handbook estimates that the gap between these two shapes is measured in hours, not minutes, and that the dominant contributor is almost always the coordination gap, not the technical gap.

The same Grafana instance serves both teams. The difference is what the team does with it before the incident and during it.

How it works

The operating model is small and named. The names are the most important part: every role has a name because un-named work collapses to whoever shouts loudest.

   Major Incident Observability Model
   ===================================

          +------------------------+
          |   Incident Commander   |  owns the decision
          |   (IC)                 |  cadence and comms
          +-----------+------------+
                      |
        +-------------+-------------+
        |             |             |
        v             v             v
   +---------+  +----------+  +-----------+
   |  Scribe |  | Comms    |  | Ops Lead  |
   |         |  | Lead     |  |           |
   +----+----+  +----+-----+  +-----+-----+
        |            |              |
        v            v              v
   Timeline      Channel &      Mitigation
   writes        status page    actions
        \            |              /
         +-----------+-------------+
                     |
                     v
             Pinned Dashboard
             (Grafana, single page)
                     |
                     v
             Evidence Ledger
             (Slack, Git, screenshots)

• Incident Commander (IC). Single decision-maker. Not the most senior engineer by default; the one with the calmest read of the situation. Owns the cadence (typically every 5-15 minutes for a Sev1), the bridge, and the “we are rolling back” call.
• Scribe. Owns the timeline. The scribe’s job is to write down what was decided, by whom, and at what time. The scribe is the single source of truth that the post-incident review will read.
• Comms Lead. Owns the customer-facing surface: status page, executive updates, and external communications. The comms lead does not run root-cause investigation.
• Ops Lead. Owns the mitigation path. The ops lead is usually the on-call engineer who has the most context, or the engineer who owns the service.
• Pinned Dashboard. A single Grafana dashboard that the IC and the team share. The dashboard is pinned to the incident channel, refreshed on a known cadence, and is the only dashboard the team reads together.
• Evidence Ledger. Slack channel messages, decision notes, saved dashboard panels, exported queries. The ledger is append-only; edits are visible.

The composition is more important than the technology. A team of four with this model and a stale dashboard will outperform a team of twelve with a fresh dashboard and no model.

How to configure it

The configuration is the on-call schedule, the escalation policy, the pinned dashboard, and the evidence ledger. The technology is the easy part; the discipline is the rehearsal.

The on-call schedule in PagerDuty (or an equivalent such as OpsGenie or Grafana OnCall) defines the IC rotation. The escalation policy defines who picks up when the IC does not acknowledge within the SLA. Both must be reviewed before the incident, not during it.

# pagerduty escalation policy (illustrative, not a complete file)
name: observability-oncall
escalation_rules:
  - escalation_delay_in_minutes: 5
    targets:
      - type: user
        id: PABC123   # primary IC
      - type: schedule
        id: PSCH456   # observability-oncall schedule
  - escalation_delay_in_minutes: 10
    targets:
      - type: user
        id: PDEF789   # secondary IC
      - type: team
        id: PTEAM001  # observability team

The pinned dashboard is provisioned ahead of time and stored in version control. The dashboard uses Grafana’s templating variables for the service and time range; the IC narrows the template to the affected service at minute zero.

# grafana provisioning - dashboards.yml
apiVersion: 1
providers:
  - name: incident-dashboards
    orgId: 1
    folder: MajorIncidents
    type: file
    disableDeletion: true
    updateIntervalSeconds: 30
    options:
      path: /var/lib/grafana/dashboards/incidents
      foldersFromFilesStructure: false

The dashboard JSON lives at /var/lib/grafana/dashboards/incidents/sev1.json and is the single artefact the team reads. The folder MajorIncidents is not editable by the on-call rotation; changes go through PR review. The discipline is to keep the dashboard boring and reviewable, not bespoke-per-incident.

The evidence ledger is a Slack channel per incident, archived to Git. The archive step is a CI job that runs hourly during an incident:

# SEVERITY: READ-ONLY
slack-cli --token "$SLACK_BOT_TOKEN" \
  conversations.history --channel "$INCIDENT_CHANNEL" \
  --oldest "$INCIDENT_START_EPOCH" \
  --limit 1000 \
  | jq '.messages' > incident-${INCIDENT_ID}.json
git -C /var/lib/incident-archive/ add incident-${INCIDENT_ID}.json
git -C /var/lib/incident-archive/ commit -m "incident ${INCIDENT_ID}: snapshot"

How to validate it

Four checks before the incident is real. Run them in a rehearsal; do not run them in anger.

1. The on-call schedule resolves.

# SEVERITY: READ-ONLY
pd-schedules list --rotation observability-oncall \
  | jq '.[] | select(.name=="observability-oncall") | .on_call_now'

Expected output (illustrative):

{
  "user": "alice",
  "started_at": "2026-08-13T00:00:00Z",
  "ends_at": "2026-08-20T00:00:00Z"
}

2. The pinned dashboard renders.

# SEVERITY: READ-ONLY
curl -s -u "${GRAFANA_USER}:${GRAFANA_API_KEY}" \
  "https://grafana.example.com/api/dashboards/uid/sev1-main" \
  | jq '.meta.slug, .dashboard.title, (.dashboard.panels | length)'

Expected output:

"sev1-main"
"Sev1 - Checkout Service"
24

A board that renders but contains zero panels is the failure mode of a provisioning misconfiguration that reads the template but does not instantiate the rows.

3. The status page can be updated.

# SEVERITY: READ-ONLY
curl -s -X POST -H "Authorization: Bearer ${STATUSPAGE_TOKEN}" \
  "https://api.statuspage.io/v1/pages/${PAGE_ID}/components" \
  -d '{"component":{"status":"investigating"}}'

4. The evidence ledger exports.

Run the export job manually and confirm the JSON file is written and committed to the archive repository.

How it can fail

Six failure modes recur across teams that have not rehearsed the discipline. Each has an observable symptom.

1. No IC. The same engineer runs the investigation, the comms, and the mitigation. Symptom: status page updates arrive fifteen minutes late; the wrong mitigation is applied because the engineer was typing a customer update when the rollback window closed.
2. Pinned dashboard not narrowed. The pinned dashboard shows “all services” because the IC forgot to set the service template variable. Symptom: every panel shows aggregate traffic; the engineer cannot answer the “which service is affected” question.
3. Dashboard pinned, but for the wrong service. The IC pasted yesterday’s URL, which was for a different incident. Symptom: the team agrees on a mitigation for a service that is not on fire.
4. No comms cadence. The first status update is given at minute zero; the next is given at minute forty, when the customer-success team pages the IC. Symptom: customer support escalates before the engineering team does; external comms fall behind.
5. Evidence lost. The Slack channel is created but never archived. The channel is then deleted at incident closure. Symptom: the post-incident review has no decision log; the root cause is reconstructed from memory.
6. Change chaos. Unrelated deploys land during the incident. Symptom: the time series is unreadable because five unrelated events have stamped the dashboard at the same minute.

How to troubleshoot it

When the major incident response starts to feel chaotic, the order is:

1. Name the IC. If nobody is named, the most senior engineer is the IC by default. The IC is the only person who says “we are rolling back.”
2. Pin the dashboard. Generate the URL, paste it into the channel, and ask everyone to read from that URL only.
3. Set the cadence. Write the next-up time in the channel description. The cadence is real-time pressure against status-page drift.
4. Start the evidence ledger. The scribe writes the first line: “Incident opened at HH:MM. Initial symptom: X. Initial hypothesis: Y.”
5. Freeze changes. Any deploy that is not the mitigation is paused. The freeze is announced in the channel.
6. Deduplicate. If five people are investigating the same thing, the IC reassigns four of them to adjacent questions.

Security implications

The incident channel is a sensitive artefact. It contains internal service names, customer-impact language, and often references to unpublished security vulnerabilities. The channel access list must be the on-call rotation plus the incident participants, not the whole engineering organisation. The archive repository must be private. The status page token must be rotated on the same cadence as other production credentials; an incident response is a high-value target for credential theft because the defenders are distracted.

The dashboard may contain labels that interpolate user PII (customer ID, email). The provisioning template must use __user filtering or redaction at the data source level, not the dashboard level. The incident channel is not a place to paste a panel that includes user PII; the redaction has to be upstream.

Performance implications

The observability platform is itself a load-bearing piece of the incident response. A Sev1 that starts a query storm on Prometheus can take the alerting platform down. The preparation is:

• Recording rules. The dashboards the IC reads during a Sev1 should be backed by recording rules, not live queries. A recording rule reading sum by (service)(rate(...)) is cheap; the same expression as a live query is a per-panel scan.
• Dashboard panels. Each panel evaluates against the data source. A 24-panel dashboard with 24 live queries is a 24x load at every refresh. Pin the dashboard to the recording rules layer.
• Query limits. During an incident, the IC is not the only person querying Prometheus. Set a query concurrency limit in prometheus.yml (--query.max-concurrency=20) and a per-query timeout (--query.timeout=2m) so that one runaway query does not starve the analyst queries.

Production guidance

• Rehearse the model in a quarterly game day. The game day uses a synthesised Sev1 in a staging environment; the rehearsed outcome is the docs the IC will read at minute three of the real incident.
• The pinned dashboard is not a service dashboard. It is a single page that shows the user-journey metrics at the top, the dependencies in the middle, and the host metrics at the bottom. The lesson on incident dashboards expands this.
• Comms cadence is a number, not a feeling. The IC writes the next update time in the channel description. The post-incident review computes the cadence that was actually delivered and compares it to the cadence that was promised.
• The evidence ledger is append-only. Edits are visible. Deleted messages are evidence of their own deletion.

Verification

You should now be able to answer:

• What is the operational definition of a major incident, and how does it differ from a routine alert?
• What are the four named roles in the major incident model, and which one must never be the same person as the operator?
• What is the pinned dashboard, where is the URL pasted, and why is the URL itself the source of truth?
• What is the evidence ledger, where is it stored, and why is it append-only?
• What six failure modes recur in teams that have not rehearsed the discipline?
• How does the recording-rules layer protect the observability platform from a query storm during a Sev1?

Quiz