Dashboard Hierarchy

A production incident arrives. The on-call engineer opens Grafana, sees the platform overview turn yellow, follows a row on the “top services” panel, lands on the service dashboard for checkout, then clicks a single hot pod in the saturation row to reach the instance dashboard for pod-7. Three pivots, one investigation. Each dashboard is short, named, owned. Two minutes later they have the answer.

That discipline is the dashboard hierarchy. Without it, every incident starts with a search bar, a Slack question, and a guess about who to call.

What it is

A dashboard hierarchy is the relationship between the dashboards inside one Grafana org. Three logical tiers appear in any non-trivial production:

• Platform tier — fleet-wide. One dashboard per cluster or per region. Shows the fleet as one entity: top N services, error budget, open alerts, recent deploys.
• Service tier — one service. A dashboard per named service (checkout, payment-svc, cart). Shows the SLO, RED, USE, dependencies, recent incidents.
• Instance tier — one process or one host. A dashboard per instance pattern (pod-7, db-1, lb-edge). Shows the per-host resource panel, GC, hot loops, process logs filtered to that instance.

Folders, dashboard UIDs, and data source UIDs are the plumbing that makes the hierarchy work. Variables are the input layer that takes the operator from one tier to the next.

A 200-service fleet usually has 1 platform dashboard, 200 service dashboards (one per service), and templates (not 1 per instance) for the instance tier. Templates bring the count down to “a few” which is the number an operator can remember.

Why a sysadmin cares

The hierarchy is the difference between two operational outcomes:

The cost of getting this wrong is paid during incidents. The cost of getting it right is paid once, at provisioning.

How it works

The mental model is three nodes with edges between them. The edges are Grafana templating links that carry the relevant variable across the pivot.

                  Platform Overview (cluster-eu-west)
                    Dashboard UID: d/platform-eu
                          |
                          |  templating link: type=link,
                          |  url = /d/service-$service/
                          |  includeVars = true
                          v
                    Service Dashboard (checkout)
                       Dashboard UID: d/svc-checkout
                          |
                          |  templating link: url =
                          |  /d/instance-$instance/
                          v
                    Instance Dashboard (pod-7)
                       Dashboard UID: d/instance-pod-7

Variables live on the dashboard that consumes them. When you click “checkout” on the platform overview, Grafana sets var-service=checkout, navigates to /d/svc-checkout, and every panel that references $service re-queries against that value.

The data source UID alias is the second key. Every dashboard ships with uid: ${DS_PROMETHEUS} in its templating list. Grafana substitutes the actual UID at load time based on the org’s provisioned data sources. The dashboard JSON is portable across environments: the same file runs against staging Prometheus and production Prometheus without editing the JSON.

How to configure it

The shape is two halves: data sources provisioned as YAML, and a folder hierarchy for dashboards.

# /etc/grafana/provisioning/datasources/datasources.yaml
apiVersion: 1

datasources:
  - name: Prometheus
    uid: prom-prod
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: false
    jsonData:
      httpMethod: POST
      timeInterval: 30s
      manageAlerts: false
  - name: Loki
    uid: loki-prod
    type: loki
    access: proxy
    url: http://loki:3100
    editable: false
    jsonData:
      maxLines: 1000
  - name: Tempo
    uid: tempo-prod
    type: tempo
    access: proxy
    url: http://tempo:3200
    editable: false

# /etc/grafana/provisioning/dashboards/dashboards.yaml
apiVersion: 1

providers:
  - name: platform-tier
    orgId: 1
    folder: Platform
    folderUid: platform
    type: file
    disableDeletion: true
    updateIntervalSeconds: 30
    allowUiUpdates: false
    options:
      path: /var/lib/grafana/dashboards/platform
      foldersFromFilesStructure: true
  - name: service-tier
    orgId: 1
    folder: Services
    folderUid: services
    type: file
    disableDeletion: false
    updateIntervalSeconds: 30
    allowUiUpdates: false
    options:
      path: /var/lib/grafana/dashboards/services
      foldersFromFilesStructure: true

The relevant pieces of a service-tier dashboard JSON:

{
  "uid": "svc-checkout",
  "title": "Service: checkout",
  "tags": ["tier:service", "service:checkout"],
  "templating": {
    "list": [
      {
        "name": "service",
        "type": "custom",
        "query": "checkout",
        "current": { "text": "checkout", "value": "checkout" },
        "hide": 0
      },
      {
        "name": "instance",
        "type": "query",
        "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
        "query": "label_values(http_server_requests_seconds_count{service=\"$service\"}, instance)",
        "refresh": 2,
        "includeAll": true,
        "multi": true
      }
    ]
  },
  "links": [
    {
      "title": "Platform overview",
      "url": "/d/platform-eu?var-service=${service}",
      "type": "link",
      "icon": "dashboard",
      "includeVars": true,
      "keepTime": true,
      "asDropdown": false
    },
    {
      "title": "Instance drilldown",
      "url": "/d/instance-pod?var-service=${service}&var-instance=${instance:raw}",
      "type": "link",
      "icon": "external link",
      "includeVars": true,
      "keepTime": true,
      "asDropdown": true
    }
  ]
}

The ${DS_PROMETHEUS} placeholder in the variable datasource is the alias. It resolves against the provisioned data source named Prometheus. Staging uses the same dashboard JSON but the provisioning YAML points the alias at a different UID.

How to validate it

# READ-ONLY: confirm the provisioning files are well-formed
grafana-cli admin provisioning lint \
  /etc/grafana/provisioning/datasources

# READ-ONLY: list folders Grafana has loaded
curl -s -u admin:admin \
  http://grafana:3000/api/folders | jq '.[] | {uid, title}'

Expected output:

{"uid":"platform","title":"Platform"}
{"uid":"services","title":"Services"}

# READ-ONLY: confirm a dashboard resolves all aliases
curl -s -u admin:admin \
  http://grafana:3000/api/dashboards/uid/svc-checkout \
  | jq '.dashboard.templating.list[]
        | select(.name=="service") | .datasource.uid'

The expected value is prom-prod. If it returns the literal string ${DS_PROMETHEUS}, the alias has not been resolved — usually a missing data source provisioning file.

Open the dashboard in the browser. Type ?var-service=checkout into the URL. Every panel with $service in its query must update within the next refresh interval. If a panel still shows the default value, the query is hard-coded — fix the JSON, do not edit the live dashboard.

How it can fail

The failure shapes that turn hierarchy into chaos:

1. Missing data source UID — ${DS_PROMETHEUS} is unrendered after reload. Every panel is stuck on a red “data source not found” badge. Symptom: dashboard renders but every panel shows a red broken icon. Check /api/datasources for the expected UID.
2. Folder UID renamed without updating JSON — dashboard 404s silently. Symptom: folder shows in the sidebar but its contents look empty. Run curl /api/search?folderUid=<expected-uid> to confirm.
3. Variable query returns empty — drill-down link has nothing to substitute, so Grafana navigates to /d/instance-/. Symptom: operator clicks the link and lands on a dashboard with no $instance bound. Validate by opening the variable panel and choosing a value manually.
4. Hard-coded service in PromQL — the panel still shows the old service when $service changes. Symptom: panel does not update after the operator picks another service. Audit panel JSON with grep '"expr"' <file>.json | grep -v '$service'.
5. Provisioning reload after UI edit — somebody clicks “save” in the UI, then the next reload overwrites with the file. The UI edit is gone. Symptom: edits appear and disappear at the next updateIntervalSeconds tick. Set allowUiUpdates: false on production providers.
6. Stale dashboard UID rotation — somebody rotates a UID without checking links. Symptom: drill-down links 404 across every dashboard that points to the old UID. Always grep for the old UID across /var/lib/grafana/dashboards/ before rotating.

How to troubleshoot it

The diagnostic order is the same for every hierarchy problem:

1. Was it working before? Diff the most recent provisioning change. If provisioning changed in the last hour, that is the suspect.
2. What does the platform view say? Check the Grafana server log at /var/log/grafana/grafana.log for errors mentioning provisioning, dashboard UID, or data source UID.
3. What does the dashboard view say? Open the dashboard and look at the variable panel. Are the variables populated? Are they stale?
4. Form a hypothesis, find evidence, test, validate. If the hypothesis is “data source alias broken”, check /api/datasources/uid/${DS_PROMETHEUS}. If the variable is empty, run the variable query directly in Explore.

Security implications

• Folder permissions are the unit of access control in Grafana 11. A team can be granted Editor on Services/Checkout and Viewer everywhere else. There is no per-panel permission.
• Dashboard UID is a stable identifier. Rotating it without updating every cross-dashboard link breaks the hierarchy for everyone who had a bookmark.
• Variables in URL can leak trace IDs, user IDs, or session IDs if a panel or annotation query echoes them. Audit ?var-*= in shared screenshots before posting in public channels.
• Provisioned dashboards should be editable: false in production. A leaked editor credential can otherwise rewrite a provisioned file via the UI for up to one reload interval.

Performance implications

• Overview dashboards run aggregate queries across the whole fleet. Use recording rules (Prometheus) or metrics queries with $__interval step (Loki) to keep load bounded.
• Variable queries add 1–3 round trips on every dashboard open. One variable with a query datasource and a multi-value return is the cheap normal. Five chained variables is the expensive mistake.
• Drill-down dashboards can hit high-cardinality instance panels. A pod-* template with includeAll: true becomes 1000+ series on dashboards that scroll. Always provide a sane default in $instance.current.
• Provisioning reload runs updateIntervalSeconds for every folder. A 5-second interval in a 1000-file directory is the classic “Grafana eats disk” pattern. 30 seconds is the normal for production.

Production guidance

• 1 platform dashboard per cluster or region. 1 service dashboard per service, generated from a Jsonnet / Go template.
• Provision all dashboards. No live edits in production.
• editable: false on every provisioned dashboard.
• disableDeletion: true on production providers.
• One folder per ownership domain (Platform, Services, Business) plus subfolders per team if needed.
• Validate hierarchy with grafana-cli admin provisioning lint before every deploy.
• Tag every dashboard with tier:<platform|service|instance> so a search by tag becomes the inventory.

Verification

You should now be able to answer:

• What are the three logical tiers of a Grafana dashboard hierarchy, and which one does an operator open first?
• Where does the data source UID alias ${DS_PROMETHEUS} fit in the plumbing?
• Why does a variable-driven panel beat a hard-coded one in a service dashboard?
• What breaks when ${DS_PROMETHEUS} is missing from the provisioning files?

Quiz