Overview Dashboards · Observability

A senior SRE opens the Grafana home page and is briefed in five seconds. Top row: green / yellow / red tiles for three wildcard SLOs. Second row: top 10 services by request rate and by error rate. Third row: open alerts by severity, latest deployment per service. Twelve panels, total. They close the tab and write an incident-review follow-up they had on their mind.

That five-second read is the overview dashboard’s only job. If the overview cannot deliver it in five seconds, the overview has a panel that does not earn its keep.

What it is

A platform overview is the dashboard an operator opens first. One per cluster, one per region, or one per business domain — not one per service. It shows fleet state, not service state.

The shape is determined by the WIG discipline (Wildly Important Goal), borrowed from execution strategy: two or three metrics that must be green, plus the supporting context an operator needs to interpret a non-green tile. The overview answers four questions and nothing more:

Is the fleet healthy?
If not, which service is it?
What just changed (deploy, alert, incident)?
Where do I click to find out more?

A panel that does not answer one of those four is a candidate for removal.

Why a sysadmin cares

Three operational costs come from a misjudged overview:

Wallpaper effect — sixty panels of green. The operator learns to skip the dashboard because reading it is not faster than not reading it. After six months nobody opens it.
Hot-spot blindness — the overview shows a service tile, but the operator cannot tell whether the underlying failure is saturation, an error rate, or a dependency. They pivot to the service dashboard regardless. The overview becomes a directory, not a status page.
Slow load cost — sixty panels times aggregate queries across the fleet is several seconds of CPU per refresh. The overview becomes the dashboard nobody wants to be the first to load in the morning.

The “right number of panels” is empirical. Start with ten, watch operator behaviour for a quarter, drop panels that do not drive action. The discipline is subtraction.

How it works

A clean overview has four rows. The four rows map to the four questions the dashboard exists to answer.

  +--------------------------------------------------+
  | ROW 1: WILDCARD STATUS (2-3 panels)              |
  | SLO overall (gauge with thresholds)              |
  | Open P1 / P2 alerts (stat with colour)           |
  | Last successful canary deploy (stat)             |
  +--------------------------------------------------+
  | ROW 2: TOP-N SERVICES (4 panels)                |
  | Top 10 by request rate (bar gauge)              |
  | Top 10 by error rate (bar gauge)                |
  | Top 10 by p99 latency (bar gauge)               |
  | Top 10 by saturation (bar gauge)                |
  +--------------------------------------------------+
  | ROW 3: WHAT CHANGED (3 panels)                  |
  | Last deploy per service (table)                 |
  | Open incidents (stat / table)                   |
  | Active burn-rate alerts (stat)                  |
  +--------------------------------------------------+
  | ROW 4: KEY DEPENDENCIES (3 panels)              |
  | Platform OK (stat with thresholds)              |
  | Critical dependencies OK (stat)                 |
  | Last 1h error budget burn (timeseries)          |
  +--------------------------------------------------+

Twelve to fifteen panels. Every panel answers one of the four questions. Every panel pivots via a templating link to the service dashboard that owns the metric.

Panel ordering follows eye-scan priority. Top-left is the most-read region. The wildcard status tiles sit there because they are the only panels that should change during an incident. The “last successful canary” tile sits next to it because that is the second thing an executive asks.

Burn rate panels show error budget consumption over time. A 1-hour burn-rate panel rising 14× faster than the budget rate is the visual cue to open a PagerDuty ticket.

How to configure it

The dashboard JSON below is a working skeleton. The trick is to rest every panel on a recording rule, not a raw query:

# prometheus_alert_rules.yaml
groups:
  - name: fleet-recording
    interval: 30s
    rules:
      - record: fleet:request_rate:5m_by_service
        expr: |
          sum by (service) (
            rate(http_server_requests_seconds_count[5m])
          )
      - record: fleet:error_rate:5m_by_service
        expr: |
          sum by (service) (
            rate(http_server_requests_seconds_count{status=~"5.."}[5m])
          ) / ignoring(status) group_left
          sum by (service) (
            rate(http_server_requests_seconds_count[5m])
          )

{
  "uid": "platform-eu",
  "title": "Platform / eu-west",
  "tags": ["tier:platform", "region:eu-west"],
  "schemaVersion": 39,
  "time": { "from": "now-1h", "to": "now" },
  "panels": [
    {
      "id": 1, "type": "stat", "title": "Overall SLO (30d)",
      "gridPos": { "x": 0, "y": 0, "w": 4, "h": 4 },
      "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
      "targets": [{
        "refId": "A",
        "expr": "1 - (sum(increase(fleet:error_rate:5m_by_service[30d])) / 0.001)"
      }],
      "fieldConfig": {
        "defaults": {
          "unit": "percentunit",
          "thresholds": {
            "mode": "absolute",
            "steps": [
              { "color": "red",    "value": null },
              { "color": "yellow", "value": 0.99 },
              { "color": "green",  "value": 0.999 }
            ]
          },
          "color": { "mode": "background" }
        }
      }
    },
    {
      "id": 2, "type": "bargauge", "title": "Top 10 by error rate",
      "gridPos": { "x": 4, "y": 0, "w": 20, "h": 6 },
      "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
      "targets": [{
        "refId": "A",
        "expr": "topk(10, fleet:error_rate:5m_by_service)"
      }],
      "fieldConfig": {
        "defaults": {
          "unit": "percentunit",
          "thresholds": {
            "mode": "absolute",
            "steps": [
              { "color": "green",  "value": null },
              { "color": "yellow", "value": 0.001 },
              { "color": "red",    "value": 0.01 }
            ]
          }
        }
      }
    },
    {
      "id": 3, "type": "stat", "title": "Open P1",
      "gridPos": { "x": 0, "y": 4, "w": 4, "h": 3 },
      "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
      "targets": [{
        "refId": "A",
        "expr": "sum(ALERTS{severity=\"page\", alertstate=\"firing\"})"
      }],
      "fieldConfig": {
        "defaults": {
          "color": { "mode": "background" },
          "thresholds": {
            "mode": "absolute",
            "steps": [
              { "color": "green", "value": null },
              { "color": "red",   "value": 1 }
            ]
          }
        }
      }
    }
  ],
  "links": [
    {
      "title": "Service drilldown",
      "url": "/d/svc-${__series.labels.service}?var-service=${__series.labels.service}",
      "type": "link",
      "includeVars": true,
      "keepTime": true,
      "asDropdown": false
    }
  ]
}

Three notes on what is not in this skeleton:

No log panels. Logs go to the per-service dashboard.
No trace panels. Traces go to the per-service or per-instance dashboard.
No per-host metrics. Those go to the per-instance dashboard.

The overview is intentionally aggregate-only. The four question budget excludes per-host metrics by construction.

How to validate it

# READ-ONLY: confirm the recording rule is producing series
curl -s -u admin:admin \
  http://grafana:3000/api/ds/query \
  -H 'content-type: application/json' \
  -d '{
    "queries": [{
      "refId": "A",
      "datasource": { "type": "prometheus", "uid": "prom-prod" },
      "expr": "topk(10, fleet:error_rate:5m_by_service)",
      "instant": true
    }],
    "from": 1700000000,
    "to":   1700000600
  }' | jq '.results.A.frames[0].data.values[1][]'

Expected output: ten numeric values, one per top service. If the array has fewer than ten entries, fewer than ten services have service labels in the last 5 minutes.

# READ-ONLY: confirm the SLO panel is reading a sane value
curl -s -u admin:admin \
  http://grafana:3000/api/ds/query \
  -H 'content-type: application/json' \
  -d '{
    "queries": [{
      "refId": "A",
      "datasource": { "type": "prometheus", "uid": "prom-prod" },
      "expr": "1 - (sum(increase(fleet:error_rate:5m_by_service[30d])) / 0.001)",
      "instant": true
    }],
    "from": 1700000000,
    "to":   1700000600
  }' | jq '.results.A.frames[0].data.values[1][0]'

Expected output: a value close to 1.0 (the SLO is “met”) or slightly below (the SLO is “at risk”). A value above 1.0 means the error budget is positive (impossible for a 99.9% SLO).

# READ-ONLY: list the panels on the overview and confirm count
curl -s -u admin:admin \
  http://grafana:3000/api/dashboards/uid/platform-eu \
  | jq '.dashboard.panels | length'

A successful overview returns a number between 8 and 20. Above 20, the overview is overloaded.

How it can fail

The high-frequency failure modes:

Recording rule falls behind — fleet:error_rate:5m_by_service disappears after a Prometheus restart because the rule file has a syntax error. Symptom: top-N panels render “No data”. Fix: promtool check rules prometheus_alert_rules.yaml.
Cross-service aggregation hides the bad service — sum by (service) on a metric without the service label collapses every series into one and the top-10 panel becomes “Top 1”. Symptom: panel shows one bar with the sum. Audit the underlying instrumentation.
Flood of new panels added without review — a team adds a panel for their own service each month. After a year the overview has 80 panels. Symptom: dashboard takes 30 seconds to load. Run a quarterly audit; remove any panel no engineer can justify.
SLO panel renders red but no page fires — the SLO is on the overview but the burn-rate alert watches a different recording rule. Symptom: operator sees the dashboard turn red and learns not to trust it. Reconcile the alert rule with the panel query.
Pivot link goes to a 404 — a service was renamed in the source of truth but the service-tier dashboard was not renamed. Symptom: clicking the top-N bar goes to an empty dashboard. Bind the dashboard UID to a service catalog.
Colour-blind unreadable — only colour is used to encode the wildcard status, ignoring “value” along the y-axis. Symptom: a colour-blind operator cannot read the dashboard. Use both colour and the numeric value, or use icons.

How to troubleshoot it

Confirm the recording rule exists. curl /api/v1/rules | jq '.data.groups[].rules[].name' should return fleet:error_rate:5m_by_service and friends.
Confirm the panel query returns rows. Open Explore, paste the panel expression, run for the same time range. If empty, the data source is the problem.
Confirm the threshold is honest. Hover over the panel tile and read the number. The colour should map to the value.
Confirm the pivot link. Click the bar. If you land on /d/svc- with no service, the link template does not bind.
Run a panel-by-panel latency check. Promote the panel queries to a recording rule if any takes more than 1 second.

Security implications

Overview panels expose fleet totals. A public overview (a status page board, for instance) reveals request volume per service. Strip that detail before publishing.
Recording rules may sample sensitive request paths. The metric http_server_requests_seconds_count records the URL template, not the path, but a path-template that includes a user identifier still leaks. Audit the recording-rule expressions for path-label leakage.
Service catalog link is a privileged object — rotate the service catalog URL on personnel changes.

Performance implications

Overview load is the sum of its panel query times plus parallelisation cap. Each panel that hits a raw query adds 0.5–3 seconds. Each panel that hits a recording rule adds 50–100 ms. Aim for total render under 2 seconds.
Recurring refresh every 30 seconds means 30 panel refreshes per minute. An overview that runs 12 raw queries sustained is a continuous load on Prometheus. Move every query into a recording rule.
Top-N cardinality — topk(10, ...) keeps the top 10 in memory but the underlying query still walks every series. Bound the source series (drop instance and pod labels upstream).

Production guidance

8 to 20 panels. Anything above 20 fails the read-in-five-seconds test.
Every panel query hits a recording rule. Raw queries are forbidden on the platform overview.
SLO tiles drive colour, not number. The number is for context, the colour is the signal.
One folder per ownership domain. The overview sits at the folder root.
One tag: tier:platform on every overview dashboard. A search by tag returns the inventory.
Quarterly panel audit. The audit removes panels that did not drive an action in the last 90 days.

Verification

You should now be able to answer:

Which four questions does a platform overview exist to answer?
Why does a recording rule belong in front of every overview panel query?
What is the WIG discipline, and what does it mean for the panel count?
Why must every overview panel pivot to a per-service dashboard with a templating link?