Incident Dashboards

The Incident Commander opens Grafana at minute one of a Sev1. The page renders. Forty panels are visible. Only four of them are relevant to the failing service. The engineer spends the next three minutes selecting the right panels, narrowing the time range, and arguing with a dropdown that does not show the affected region. By minute four the rollback window has closed.

This is the failure mode the incident dashboard exists to prevent. An incident dashboard is a single Grafana page, pinned to the incident channel, pre-encoded for the response team. It is not a service dashboard. It is not an SLI dashboard. It is a read-only battle map for the incident itself.

What it is

An incident dashboard is a single Grafana page that the Incident Commander and the response team read together during a major incident. It is provisioned in version control, has a stable UID, and shows the user-journey metric at the top, the dependencies in the middle, and the host metrics at the bottom. Every panel has a log pivot and a trace pivot. The dashboard is not bespoke; it is the same dashboard for every Sev1, narrowed at minute zero by template variables.

The contrast matters. A service dashboard is the engineer’s day-to-day view. It has every panel for the service, including the ones nobody reads. An SLI dashboard shows the service-level objective rate over time. A per-deploy dashboard shows pre/post comparison of a single release. The incident dashboard is none of these. It is the response team’s view.

Why a sysadmin cares

The Grafana wall-of-green problem is the operational symptom of a team that has not separated the steady-state view from the incident view. A service dashboard has forty panels; the incident has four that matter. The four are the ones the response team will read together for the duration of the incident. The other thirty-six are noise that competes for attention during the response window.

The pinned incident dashboard is the artefact that makes the response team read from the same mental model. Three engineers looking at the same URL is the cheapest possible coordination mechanism. Three engineers looking at three different dashboards is the most expensive coordination failure in incident response.

How it works

The layout is opinionated. The opinion is built from the investigation tree: symptoms show up at the top, dependencies in the middle, hosts at the bottom. The team reads top-down.

   Incident Dashboard Layout
   =========================

   +------------------------------------------------------+
   |  Row 1: USER JOURNEY (the symptom)                   |
   |  - request rate by service                           |
   |  - error rate by service                             |
   |  - latency p50/p95/p99 by service                    |
   |  - SLI burn rate                                     |
   +------------------------------------------------------+
   |  Row 2: DEPENDENCIES (the next pivot)                |
   |  - downstream service latency                        |
   |  - downstream service error rate                     |
   |  - database query latency                            |
   |  - cache hit rate                                    |
   +------------------------------------------------------+
   |  Row 3: HOST & PLATFORM (the floor)                  |
   |  - CPU, memory, disk, network per host               |
   |  - container restarts                                |
   |  - node_exporter / cAdvisor metrics                  |
   +------------------------------------------------------+
   |  Row 4: TELEMETRY PIVOTS (the escape hatches)        |
   |  - last 100 error logs (Loki)                        |
   |  - top error traces (Tempo)                          |
   |  - recent deploy / config annotations                |
   +------------------------------------------------------+

Each row has a job. The user-journey row answers the question “is the user affected?”. The dependencies row answers “which component is responsible?”. The host row answers “is the infrastructure the constraint?”. The pivots row answers “where do I go for evidence?”.

Every panel has a data link. A data link is a label-driven URL that opens the corresponding log query or trace query in Grafana Explore. The link is the escape hatch from the dashboard to a non-dashboard query. The discipline is that every panel has one; the engineer should never have to copy a metric name into a new search bar.

How to configure it

The dashboard is provisioned as a JSON file. The file lives in version control and is read by Grafana’s file-based provisioning on a refresh interval. The UID is a stable string; renaming it breaks the URL and the pinned-dashboard discipline.

# grafana provisioning - dashboards.yml
apiVersion: 1
providers:
  - name: incident-dashboards
    orgId: 1
    folder: MajorIncidents
    type: file
    disableDeletion: true
    updateIntervalSeconds: 30
    options:
      path: /var/lib/grafana/dashboards/incidents
      foldersFromFilesStructure: false

The dashboard JSON itself is too long to embed in the lesson in full; the shape is the important part. The trimmed skeleton below shows the template variables, the row structure, and a single panel with its data link.

{
  "uid": "sev1-main",
  "title": "Sev1 - All Services",
  "tags": ["incident", "sev1"],
  "templating": {
    "list": [
      {
        "name": "service",
        "type": "query",
        "datasource": "Prometheus",
        "query": "label_values(up{job=~\".+\"}, job)",
        "refresh": 2,
        "includeAll": true
      },
      {
        "name": "region",
        "type": "query",
        "datasource": "Prometheus",
        "query": "label_values(up{job=\"$service\"}, region)",
        "refresh": 2,
        "includeAll": true
      }
    ]
  },
  "panels": [
    {
      "id": 1,
      "type": "timeseries",
      "title": "Request rate by service",
      "gridPos": {"x": 0, "y": 0, "w": 12, "h": 8},
      "targets": [
        {
          "expr": "sum by (service) (rate(http_requests_total{service=~\"$service\",region=~\"$region\"}[1m]))",
          "datasource": "Prometheus"
        }
      ],
      "fieldConfig": {"defaults": {"unit": "reqps"}},
      "links": [
        {
          "title": "Logs for this service",
          "url": "/explore?panes=%7B%22loki%22%3A%7B%22query%22%3A%22%7Bservice%3D%5C%22${__field.labels.service}%5C%22%7D%22%7D%7D",
          "targetBlank": true
        }
      ]
    }
  ]
}

Several things to notice:

• The uid is sev1-main and is stable. The pinned URL is https://grafana.example.com/d/sev1-main. Renaming the UID breaks the channel’s pinned link.
• The tags include incident and sev1. The team can search by tag during the incident to find the dashboard by name.
• The service variable is a query type, populated by a Prometheus label query. The dropdown refreshes on every dashboard load (refresh: 2); an old dropdown is the first-mover failure mode.
• The data link is a URL-encoded Loki query. The escape sequence %7B is {, %22 is ", %5C%22 is the escaped quote around the label value. The link is the escape hatch from the dashboard to Loki.
• The expression rate(http_requests_total[1m]) is bounded to a one-minute range so the panel renders at refresh. The discipline is that no live panel on the incident dashboard uses a wider range than five minutes.

The companion recording rules feed the dashboard. The dashboard panels refer to the recording rule output, not the raw counter. The trade-off is operational: a recording rule is one up-front cost and many cheap dashboard reads.

# prometheus rules - sev1.rules.yml
groups:
- name: sev1.recording
  interval: 30s
  rules:
  - record: sev1:request_rate:5m
    expr: sum by (service, region) (rate(http_requests_total[5m]))
  - record: sev1:error_rate:5m
    expr: |
      sum by (service, region) (rate(http_requests_total{status=~"5.."}[5m]))
      /
      sum by (service, region) (rate(http_requests_total[5m]))
  - record: sev1:latency_p95:5m
    expr: |
      histogram_quantile(0.95,
        sum by (service, region, le) (rate(http_request_duration_seconds_bucket[5m]))
      )

The dashboard panels then read sev1:request_rate:5m instead of the raw expression. The cost is moved from the dashboard refresh to the rule evaluator, which is single-threaded but bounded.

How to validate it

Five checks before the dashboard is pinned to a real incident.

1. UID is stable.

# SEVERITY: READ-ONLY
curl -s -u "${GRAFANA_USER}:${GRAFANA_API_KEY}" \
  "https://grafana.example.com/api/dashboards/uid/sev1-main" \
  | jq '.meta.slug, .dashboard.uid, .dashboard.title'

Expected output:

"sev1-main"
"sev1-main"
"Sev1 - All Services"

2. Template variables resolve.

# SEVERITY: READ-ONLY
curl -s -u "${GRAFANA_USER}:${GRAFANA_API_KEY}" \
  "https://grafana.example.com/api/datasources/proxy/1/api/v1/label/job/values" \
  | jq '.data | length'

A non-zero count means the variable query has returned services. Zero means the variable is bound to a recording rule or metric that does not exist.

3. Promtool validates the recording rules.

# SEVERITY: READ-ONLY
promtool check rules /etc/prometheus/rules/sev1.rules.yml

Expected output:

SUCCESS: /etc/prometheus/rules/sev1.rules.yml
        3 rules found
  sev1:request_rate:5m  expr ok
  sev1:error_rate:5m    expr ok
  sev1:latency_p95:5m   expr ok

4. Panel refresh is within budget.

Run the dashboard with a fresh load and inspect the backend_response_time in the Grafana server log.

# SEVERITY: READ-ONLY
journalctl -u grafana-server -n 200 \
  | grep -E '“sev1-main”' \
  | head -10

A panel that takes longer than 2 seconds to render is a candidate for a recording rule. The discipline is no live expensive query on the incident dashboard.

5. The data link redirects to a populated query.

Click the data link in the panel and confirm the Loki Explore view shows results within two seconds. A data link that opens to an empty panel is a link the team will not trust.

How it can fail

Five failure modes recur in production incident dashboards.

1. UID churn. The dashboard UID is renamed during a re-import. Symptom: every pinned Slack link returns 404; the on-call engineer has to re-find the dashboard by name.
2. Template variable unbound. The $service template was left as All by the IC. Symptom: every panel shows the aggregate; the engineer cannot answer “which service is affected”.
3. Live expensive query on a panel. A panel evaluates histogram_quantile(0.99, sum by (le) (rate(...[1h]))) on every refresh. Symptom: panel load time exceeds five seconds; the dashboard times out at minute three of the incident when the data source is hot.
4. Datasource down. The Prometheus data source returns 502. Symptom: the entire dashboard is red; the engineer cannot distinguish “service is broken” from “Prometheus is broken”. The lesson on observability-during-incident covers the fallback.
5. Dashboard sprawl. Each engineer has their own pinned dashboard. Symptom: the team reads from three different URLs; coordination cost multiplies.

How to troubleshoot it

When the dashboard misbehaves during a Sev1, the order is:

1. Strip the URL. Open the raw d/sev1-main URL without any query parameters. If the stripped URL renders, the template variables are the problem. If the stripped URL does not render, the dashboard JSON is the problem.
2. Inspect the variables. Settings -> Variables. Confirm $service and $region resolve. If a query-type variable returns empty, the upstream label is missing.
3. Inspect the rule. If a panel reads from a recording rule, run promtool check rules and inspect the rule state in the Prometheus UI.
4. Inspect the data source. Configuration -> Data sources -> Prometheus -> Save & Test. The Save & Test dialog reports the round-trip time and any HTTP error.
5. Pin the new URL. If the dashboard was changed during the incident, paste the new URL into the channel. The scribe re-archives the new URL.

Security implications

The incident dashboard often shows more labels than the service dashboard. A session that includes the affected user’s ID, the trace ID, or the request URL is sensitive. The data link to Loki or Tempo can carry the session label into the URL.

The mitigation is at the data source layer. Loki’s label allow-list and Tempo’s derived-fields configuration must exclude user identifiers. The dashboard is then automatically sanitised; the engineer cannot accidentally copy a PII-laden panel into a public incident channel.

Grafana’s session cookie and the API key used to programmatically pin the dashboard are credentials. Both must be rotated on the same cadence as the on-call roster. The incident dashboard is a high-value read target during a Sev1; an attacker who pwns the dashboard gains real-time visibility into the most sensitive window of the platform.

Performance implications

The incident dashboard is the most-refreshed dashboard in the platform during a Sev1. Twelve viewers refreshing every five seconds is twelve times the per-second load of steady-state operations. Three mitigations:

• Recording rules. The dashboard reads from pre-aggregated series. The per-panel refresh is a cheap scan, not a cross-region aggregation.
• Bound the time range. A panel that defaults to now-1h is three hundred times the cost of a panel that defaults to now-15m. The incident dashboard defaults to the last fifteen minutes; the IC widens the range only if the symptom predates the default.
• Disable auto-refresh. The dashboard refreshes on load, not on timer, during an incident. The team reads the same static view; they refresh on the cadence of a human decision, not the cadence of a screen re-render.

Production guidance

• The incident dashboard is one dashboard, not one per service. The per-service narrowing is a template variable. The team that pins three dashboards has lost the coordination benefit of the pinned URL.
• The dashboard is provisioned, not edited live. The on-call rotation does not have edit permissions. A change to the dashboard goes through PR review and a re-import.
• The dashboard is rehearsed. The quarterly game day opens the dashboard, narrows the variables, and the team reads from the same URL for thirty minutes. The rehearsal is the reason the dashboard is discoverable at minute three of a real Sev1.
• The dashboard is one of several. The business dashboard shows revenue impact; the user-journey dashboard shows the user-visible symptom. The incident dashboard is the one pinned to the channel. The other dashboards are discoverable through tags; they are not pinned.

Verification

You should now be able to answer:

• What is the operational difference between a service dashboard, an SLI dashboard, and an incident dashboard?
• Why is the pinned URL the source of truth, and what query parameters must the URL include to be reproducible?
• What is the row layout of an incident dashboard, and what question does each row answer?
• Why should every panel read from a recording rule, not a raw expression, during a Sev1?
• What five failure modes recur in production incident dashboards, and how do you diagnose each one?

Quiz