Lab: Build a Production Dashboard

Objective

By the end of this lab you will have two dashboards that live entirely in files: a twelve-second platform overview and one templated service dashboard that serves three services. Every query will be bound to $service, every panel that shows a ratio will carry a unit, and the red threshold on the latency panel will be the same number as the alert rule that pages on latency.

Then you will break it four times in ways that produce no error message anywhere — the four that account for most of the “the dashboard was green” sentences in incident reviews — and write a lint script that catches each one from the JSON before it reaches a folder.

Architecture

One host, five containers. The dashboards are files; Grafana is a renderer.

   checkout:8000     payment:8000      cart:8000
   /metrics          /metrics          /metrics
        |                 |                 |
        +--------+--------+--------+--------+
                          |  scrape 15s, service= applied as a target label
                          v
                   +--------------+
                   |  Prometheus  |  alert rules: error ratio > 1%
                   |    :9090     |               p99 latency > 1s
                   +------+-------+
                          |
                          | provisioned datasource uid=prom-lab
                          v
   +--------------------------------------------------------------+
   |                          Grafana :3000                        |
   |                                                               |
   |  /var/lib/grafana/dashboards/platform/overview.json           |
   |        |  dashboard link + data link, carries $service        |
   |        v                                                      |
   |  /var/lib/grafana/dashboards/services/service.json            |
   |        annotation query: changes(process_start_time_seconds)  |
   +--------------------------------------------------------------+

The service label is applied by Prometheus, in static_configs, not by the exporters. That is deliberate and it is the setup for the fourth defect: a label that the dashboard depends on and that no application owns can be renamed by someone editing a scrape config, with no application change and no error.

Requirements

• A disposable Linux host with Docker Engine 28.x and Compose v2, 2 GB RAM.
• curl and jq. The lint script in Task 7 is jq over the dashboard JSON; the validation section reads Grafana through its HTTP API.
• Ports 3000 and 9090 free on loopback. Both publishes bind 127.0.0.1.
• A browser you can point at http://localhost:3000. Four checks in this lab are visual and are marked as such — a colour and an axis label are not things an API assertion can judge for you.
• No out-of-band access requirement. Nothing here touches the network configuration of the host.

Scenario

A team has one dashboard. It has 84 panels, it was built in the UI over eighteen months, and three of the panels have been showing “No data” since a migration that nobody can date. During last month’s incident the on-call engineer spent six minutes scrolling it, then gave up and used Explore.

The remit is to replace it with two dashboards that live in the operations repository: one overview that answers “is the fleet healthy, and if not which service”, and one service dashboard that serves every service from one file. The hard requirement from the post-incident review is that a panel must never be able to disagree with the alert that pages on the same condition.

Tasks

Task 1: Stand up the data the dashboards will read

A dashboard built against no data is a layout exercise. Start with three services whose numbers differ enough to be worth looking at.

LAB="$HOME/rb-obs-dashboard"
mkdir -p "$LAB"/{gen,prometheus/rules,grafana/provisioning/datasources}
mkdir -p "$LAB"/grafana/provisioning/dashboards
mkdir -p "$LAB"/dashboards/{platform,services}
cd "$LAB"

gen/gen.py — the exporter deliberately does not emit a service label. prometheus_client also registers the process collectors for free, which is where the deploy annotation in Task 5 gets its metric:

#!/usr/bin/env python3
"""Three-service RED source for the dashboard lab.

Emits a request counter by route and status, and a duration histogram. The
`service` label is applied by Prometheus as a target label, not here - see the
Architecture note.
"""
import os
import random
import time

from prometheus_client import Counter, Histogram, start_http_server

ROUTE = os.environ.get("ROUTE", "/checkout")
RATE = float(os.environ.get("RATE", "25"))
FAIL_RATIO = float(os.environ.get("FAIL_RATIO", "0.002"))
SLOW_MS = float(os.environ.get("SLOW_MS", "120"))

REQUESTS = Counter(
    "http_server_requests_total",
    "Requests handled, by route and outcome.",
    labelnames=("route", "status"),
)
DURATION = Histogram(
    "http_server_request_duration_seconds",
    "Time spent handling a request.",
    labelnames=("route",),
    buckets=(0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10),
)


def main():
    start_http_server(8000)
    interval = 1.0 / RATE
    while True:
        failed = random.random() < FAIL_RATIO
        REQUESTS.labels(ROUTE, "500" if failed else "200").inc()
        centre = SLOW_MS * 8 if failed else SLOW_MS
        DURATION.labels(ROUTE).observe(random.uniform(centre * 0.6, centre * 1.4) / 1000.0)
        time.sleep(interval)


if __name__ == "__main__":
    main()

gen/Dockerfile:

FROM python:3.12-slim
RUN pip install --no-cache-dir prometheus_client==0.21.0
COPY gen.py /app/gen.py
CMD ["python", "-u", "/app/gen.py"]

prometheus/prometheus.yml — the service label is attached here:

# prometheus/prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - /etc/prometheus/rules/*.yml

scrape_configs:
  - job_name: services
    static_configs:
      - targets: ['checkout:8000']
        labels: { service: checkout }
      - targets: ['payment:8000']
        labels: { service: payment }
      - targets: ['cart:8000']
        labels: { service: cart }

prometheus/rules/service.yml — write down the two numbers now. Every threshold in the dashboards has to match one of them, and Task 7 enforces that mechanically:

# prometheus/rules/service.yml
groups:
  - name: service-slos
    rules:
      - alert: ServiceErrorRatioHigh
        # Threshold 0.01. The error-ratio panel turns red at 0.01.
        expr: |
          sum by (service) (rate(http_server_requests_total{status=~"5.."}[5m]))
            / sum by (service) (rate(http_server_requests_total[5m])) > 0.01
        for: 5m
        labels:
          severity: page
        annotations:
          summary: '{{ $labels.service }} error ratio above 1%'

      - alert: ServiceLatencyP99High
        # Threshold 1. The latency panel turns red at 1.
        expr: |
          histogram_quantile(0.99,
            sum by (service, le) (rate(http_server_request_duration_seconds_bucket[5m]))) > 1
        for: 5m
        labels:
          severity: page
        annotations:
          summary: '{{ $labels.service }} p99 latency above 1s'

Task 2: Provision Grafana from disk

grafana/provisioning/datasources/prom.yaml:

# grafana/provisioning/datasources/prom.yaml
apiVersion: 1

datasources:
  - name: Prometheus
    uid: prom-lab
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: false
    jsonData:
      httpMethod: POST
      timeInterval: 15s

grafana/provisioning/dashboards/tiers.yaml — one provider per tier, which is what makes the folder structure a property of the repository rather than of whoever clicked “new folder”:

# grafana/provisioning/dashboards/tiers.yaml
apiVersion: 1

providers:
  - name: platform-tier
    orgId: 1
    folder: Platform
    folderUid: platform
    type: file
    disableDeletion: true
    # 10s so an edit in this lab lands while you are still looking at it.
    # Production uses a longer interval; the reload is a full re-read.
    updateIntervalSeconds: 10
    allowUiUpdates: false
    options:
      path: /var/lib/grafana/dashboards/platform

  - name: service-tier
    orgId: 1
    folder: Services
    folderUid: services
    type: file
    disableDeletion: false
    updateIntervalSeconds: 10
    allowUiUpdates: false
    options:
      path: /var/lib/grafana/dashboards/services

allowUiUpdates: false is the line that decides whether these files are the source of truth or a starting point somebody will edit away from. The provisioning lab covers the drift mechanics; here it simply means every change you make in this lab is a change to a file.

Task 3: Write the service dashboard

dashboards/services/service.json. Read the row order before the JSON: header tiles, then RED, then the process resources, then logs-and-events. That order is eye-scan priority, not taste — the tiles are what changes during an incident, so they sit top-left.

{
  "uid": "svc-lab",
  "title": "Service: $service",
  "tags": ["tier:service", "team:checkout-platform"],
  "schemaVersion": 39,
  "time": { "from": "now-1h", "to": "now" },
  "refresh": "30s",
  "links": [
    {
      "title": "Platform overview",
      "url": "/d/platform-lab",
      "type": "link",
      "icon": "dashboard",
      "includeVars": true,
      "keepTime": true
    },
    {
      "title": "Runbook",
      "url": "https://runbooks.example.com/services/checkout",
      "type": "link",
      "icon": "doc",
      "targetBlank": true
    }
  ],
  "templating": {
    "list": [
      {
        "name": "service",
        "type": "query",
        "datasource": { "type": "prometheus", "uid": "prom-lab" },
        "query": "label_values(http_server_requests_total, service)",
        "refresh": 2,
        "includeAll": false,
        "multi": false
      }
    ]
  },
  "annotations": {
    "list": [
      {
        "name": "Restarts",
        "enable": true,
        "iconColor": "rgba(255, 96, 96, 1)",
        "datasource": { "type": "prometheus", "uid": "prom-lab" },
        "expr": "changes(process_start_time_seconds{service=\"$service\"}[5m]) > 0",
        "titleFormat": "process restart",
        "textFormat": "{{service}}"
      }
    ]
  },
  "panels": [
    {
      "id": 1, "type": "stat", "title": "Error ratio",
      "gridPos": { "x": 0, "y": 0, "w": 6, "h": 4 },
      "datasource": { "type": "prometheus", "uid": "prom-lab" },
      "targets": [{
        "refId": "A",
        "expr": "sum(rate(http_server_requests_total{service=\"$service\",status=~\"5..\"}[5m])) / sum(rate(http_server_requests_total{service=\"$service\"}[5m]))"
      }],
      "fieldConfig": { "defaults": {
        "unit": "percentunit", "decimals": 2, "min": 0,
        "thresholds": { "mode": "absolute", "steps": [
          { "color": "green", "value": null },
          { "color": "yellow", "value": 0.005 },
          { "color": "red", "value": 0.01 }
        ]}
      }}
    },
    {
      "id": 2, "type": "stat", "title": "p99 latency",
      "gridPos": { "x": 6, "y": 0, "w": 6, "h": 4 },
      "datasource": { "type": "prometheus", "uid": "prom-lab" },
      "targets": [{
        "refId": "A",
        "expr": "histogram_quantile(0.99, sum by (le) (rate(http_server_request_duration_seconds_bucket{service=\"$service\"}[5m])))"
      }],
      "fieldConfig": { "defaults": {
        "unit": "s", "decimals": 3,
        "thresholds": { "mode": "absolute", "steps": [
          { "color": "green", "value": null },
          { "color": "yellow", "value": 0.5 },
          { "color": "red", "value": 1 }
        ]}
      }}
    },
    {
      "id": 3, "type": "stat", "title": "Request rate",
      "gridPos": { "x": 12, "y": 0, "w": 6, "h": 4 },
      "datasource": { "type": "prometheus", "uid": "prom-lab" },
      "targets": [{
        "refId": "A",
        "expr": "sum(rate(http_server_requests_total{service=\"$service\"}[5m]))"
      }],
      "fieldConfig": { "defaults": { "unit": "reqps", "decimals": 1 } }
    },
    {
      "id": 4, "type": "timeseries", "title": "Rate by status",
      "gridPos": { "x": 0, "y": 4, "w": 12, "h": 8 },
      "datasource": { "type": "prometheus", "uid": "prom-lab" },
      "targets": [{
        "refId": "A",
        "expr": "sum by (status) (rate(http_server_requests_total{service=\"$service\"}[5m]))",
        "legendFormat": "{{status}}"
      }],
      "fieldConfig": { "defaults": { "unit": "reqps" } }
    },
    {
      "id": 5, "type": "timeseries", "title": "Latency p50 p95 p99",
      "gridPos": { "x": 12, "y": 4, "w": 12, "h": 8 },
      "datasource": { "type": "prometheus", "uid": "prom-lab" },
      "targets": [
        { "refId": "A", "legendFormat": "p50",
          "expr": "histogram_quantile(0.50, sum by (le) (rate(http_server_request_duration_seconds_bucket{service=\"$service\"}[5m])))" },
        { "refId": "B", "legendFormat": "p95",
          "expr": "histogram_quantile(0.95, sum by (le) (rate(http_server_request_duration_seconds_bucket{service=\"$service\"}[5m])))" },
        { "refId": "C", "legendFormat": "p99",
          "expr": "histogram_quantile(0.99, sum by (le) (rate(http_server_request_duration_seconds_bucket{service=\"$service\"}[5m])))" }
      ],
      "fieldConfig": { "defaults": {
        "unit": "s",
        "thresholds": { "mode": "absolute", "steps": [
          { "color": "green", "value": null },
          { "color": "yellow", "value": 0.5 },
          { "color": "red", "value": 1 }
        ]}
      }}
    },
    {
      "id": 6, "type": "timeseries", "title": "Process CPU",
      "gridPos": { "x": 0, "y": 12, "w": 12, "h": 8 },
      "datasource": { "type": "prometheus", "uid": "prom-lab" },
      "targets": [{
        "refId": "A",
        "expr": "rate(process_cpu_seconds_total{service=\"$service\"}[5m])",
        "legendFormat": "{{instance}}"
      }],
      "fieldConfig": { "defaults": { "unit": "percentunit" } }
    },
    {
      "id": 7, "type": "timeseries", "title": "Process resident memory",
      "gridPos": { "x": 12, "y": 12, "w": 12, "h": 8 },
      "datasource": { "type": "prometheus", "uid": "prom-lab" },
      "targets": [{
        "refId": "A",
        "expr": "process_resident_memory_bytes{service=\"$service\"}",
        "legendFormat": "{{instance}}"
      }],
      "fieldConfig": { "defaults": { "unit": "bytes" } }
    }
  ]
}

Three details to look at rather than skim past.

$service appears in every expression. One file, three services. The dashboard has one physical page and as many virtual pages as the variable has values, which is the only shape that stays maintainable past a handful of services.

Every ratio and every duration carries a unit. percentunit renders 0.003 as 0.30%; without it the panel says 0.003 and reads as nothing at all. reqps and bytes are the same argument for two other axes.

The thresholds are the alert’s numbers. 0.01 on the error ratio, 1 on the latency panel. Task 7 turns that into a check, because the agreement is worth nothing if it can drift silently.

Task 4: Write the platform overview

Four panels. The overview’s whole job is a five-second read and a click, and every extra panel is paid for on every load by every operator.

{
  "uid": "platform-lab",
  "title": "Platform overview",
  "tags": ["tier:platform", "team:checkout-platform"],
  "schemaVersion": 39,
  "time": { "from": "now-1h", "to": "now" },
  "refresh": "30s",
  "templating": {
    "list": [
      {
        "name": "service",
        "type": "query",
        "datasource": { "type": "prometheus", "uid": "prom-lab" },
        "query": "label_values(http_server_requests_total, service)",
        "refresh": 2,
        "includeAll": false,
        "multi": false
      }
    ]
  },
  "links": [
    {
      "title": "Service drilldown",
      "url": "/d/svc-lab",
      "type": "link",
      "icon": "external link",
      "includeVars": true,
      "keepTime": true
    }
  ],
  "panels": [
    {
      "id": 1, "type": "stat", "title": "Fleet error ratio",
      "gridPos": { "x": 0, "y": 0, "w": 8, "h": 5 },
      "datasource": { "type": "prometheus", "uid": "prom-lab" },
      "targets": [{
        "refId": "A",
        "expr": "sum(rate(http_server_requests_total{status=~\"5..\"}[5m])) / sum(rate(http_server_requests_total[5m]))"
      }],
      "fieldConfig": { "defaults": {
        "unit": "percentunit", "decimals": 2,
        "thresholds": { "mode": "absolute", "steps": [
          { "color": "green", "value": null },
          { "color": "yellow", "value": 0.005 },
          { "color": "red", "value": 0.01 }
        ]}
      }}
    },
    {
      "id": 2, "type": "stat", "title": "Firing alerts",
      "gridPos": { "x": 8, "y": 0, "w": 8, "h": 5 },
      "datasource": { "type": "prometheus", "uid": "prom-lab" },
      "targets": [{ "refId": "A", "expr": "sum(ALERTS{alertstate=\"firing\"}) or vector(0)" }],
      "fieldConfig": { "defaults": {
        "unit": "short", "decimals": 0,
        "thresholds": { "mode": "absolute", "steps": [
          { "color": "green", "value": null },
          { "color": "red", "value": 1 }
        ]}
      }}
    },
    {
      "id": 3, "type": "stat", "title": "Targets up",
      "gridPos": { "x": 16, "y": 0, "w": 8, "h": 5 },
      "datasource": { "type": "prometheus", "uid": "prom-lab" },
      "targets": [{ "refId": "A", "expr": "sum(up{job=\"services\"})" }],
      "fieldConfig": { "defaults": { "unit": "short", "decimals": 0 } }
    },
    {
      "id": 4, "type": "bargauge", "title": "Error ratio by service",
      "gridPos": { "x": 0, "y": 5, "w": 24, "h": 9 },
      "datasource": { "type": "prometheus", "uid": "prom-lab" },
      "targets": [{
        "refId": "A",
        "expr": "sum by (service) (rate(http_server_requests_total{status=~\"5..\"}[5m])) / sum by (service) (rate(http_server_requests_total[5m]))",
        "legendFormat": "{{service}}"
      }],
      "options": { "displayMode": "gradient", "orientation": "horizontal" },
      "fieldConfig": { "defaults": {
        "unit": "percentunit", "decimals": 2,
        "links": [{
          "title": "Open service dashboard",
          "url": "/d/svc-lab?var-service=${__field.labels.service}&from=${__from}&to=${__to}"
        }],
        "thresholds": { "mode": "absolute", "steps": [
          { "color": "green", "value": null },
          { "color": "yellow", "value": 0.005 },
          { "color": "red", "value": 0.01 }
        ]}
      }}
    }
  ]
}

The data link on panel 4 is the pivot that matters. ${__field.labels.service} resolves to the label value of the bar you clicked, and ${__from} and ${__to} carry the time range across, so the operator arrives at the service dashboard on the window they were already looking at. A pivot that resets the time range makes the operator re-select it every time and is the reason people stop using drill-downs.

Task 5: Start it and confirm the pivot and the annotation

compose.yaml:

# compose.yaml
name: rb-obs-dashboard

x-gen: &gen
  build: ./gen
  restart: unless-stopped

services:
  checkout:
    <<: *gen
    environment: { ROUTE: '/checkout', RATE: '25', FAIL_RATIO: '0.002', SLOW_MS: '120' }
  payment:
    <<: *gen
    environment: { ROUTE: '/pay', RATE: '10', FAIL_RATIO: '0.02', SLOW_MS: '260' }
  cart:
    <<: *gen
    environment: { ROUTE: '/cart', RATE: '40', FAIL_RATIO: '0.001', SLOW_MS: '60' }

  prometheus:
    image: prom/prometheus:v2.55.1
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention.time=6h'
      - '--web.enable-lifecycle'
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./prometheus/rules:/etc/prometheus/rules:ro
      - prom-data:/prometheus
    ports:
      - '127.0.0.1:9090:9090'

  grafana:
    image: grafana/grafana:11.3.0
    environment:
      GF_SECURITY_ADMIN_PASSWORD__FILE: /run/secrets/grafana_admin
      GF_USERS_ALLOW_SIGN_UP: 'false'
      GF_AUTH_ANONYMOUS_ENABLED: 'false'
    volumes:
      - ./grafana/provisioning:/etc/grafana/provisioning:ro
      - ./dashboards:/var/lib/grafana/dashboards:ro
      - ./secrets/grafana_admin:/run/secrets/grafana_admin:ro
      - grafana-data:/var/lib/grafana
    ports:
      - '127.0.0.1:3000:3000'

volumes:
  prom-data:
  grafana-data:

mkdir -p "$LAB/secrets"
openssl rand -base64 24 | tr -d '=/+' > "$LAB/secrets/grafana_admin"
chmod 0600 "$LAB/secrets/grafana_admin"

docker compose config --quiet && echo compose-ok
jq empty dashboards/platform/overview.json dashboards/services/service.json && echo json-ok

$ docker compose up -d --build

Wait six minutes. Every panel uses a [5m] window, and a half-filled window gives numbers that move while you read them.

GF_PW="$(cat "$LAB/secrets/grafana_admin")"

# Both dashboards loaded, in the right folders.
curl -s -u "admin:$GF_PW" 'http://localhost:3000/api/search?type=dash-db' \
  | jq -r '.[] | "\(.folderTitle // "General")/\(.title)  uid=\(.uid)"'

# The variable resolves to three services, from the data.
curl -sG http://localhost:9090/api/v1/label/service/values | jq -r '.data[]'

Now the pivot, which is a browser check. Open http://localhost:3000/d/platform-lab, note the time range, and click the payment bar on “Error ratio by service”. You should land on /d/svc-lab?var-service=payment with the same window, and every panel should be showing payment’s numbers within one refresh.

The annotation next. Restart one generator and watch a marker appear:

docker compose restart payment
sleep 90

# The metric behind the annotation, queried the way Grafana queries it.
curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=changes(process_start_time_seconds{service="payment"}[5m]) > 0' \
  | jq -r '.data.result[] | "\(.metric.service) \(.value[1])"'

A non-empty result is the marker. On the service dashboard with payment selected, a red vertical line now sits on every time-series panel at the restart. That line is the difference between “latency changed at 14:38” and “latency changed when we restarted it”.

Task 6: The four defects that produce no error

Each of these renders. None of them logs anything. Do them one at a time and restore before the next.

Defect 1: the missing unit

Remove "unit": "percentunit" from panel 1’s fieldConfig.defaults in dashboards/services/service.json, wait for the provisioning reload, and look at the tile in the browser.

jq 'del(.panels[0].fieldConfig.defaults.unit)' \
  dashboards/services/service.json > /tmp/svc-nounit.json \
  && mv /tmp/svc-nounit.json dashboards/services/service.json
sleep 15

The tile now reads 0.00 or 0.002 where it read 0.20%. The number is identical; the reading is not. This is the defect that produces the sentence “the error rate looked tiny” in a post-incident review — and the colour still works, because thresholds operate on the raw value regardless of the formatter. A tile can be red and read as nothing.

Restore the unit before continuing.

Defect 2: the hard-coded service

Add a panel whose expression names checkout directly instead of $service:

jq '.panels += [{
      "id": 8, "type": "stat", "title": "Request rate (broken)",
      "gridPos": {"x": 18, "y": 0, "w": 6, "h": 4},
      "datasource": {"type": "prometheus", "uid": "prom-lab"},
      "targets": [{"refId": "A",
        "expr": "sum(rate(http_server_requests_total{service=\"checkout\"}[5m]))"}],
      "fieldConfig": {"defaults": {"unit": "reqps"}}
    }]' dashboards/services/service.json > /tmp/svc-hard.json \
  && mv /tmp/svc-hard.json dashboards/services/service.json
sleep 15

Switch the variable to payment. Every panel changes except one, which keeps showing checkout’s rate under a title that says nothing about it. This is the worst of the four, because the panel is not empty and not wrong-looking — it is confidently displaying another service’s numbers next to this service’s.

The detection is mechanical, and it is the first rule in the lint script:

jq -r '.panels[].targets[]?.expr' dashboards/services/service.json \
  | grep -v '\$service' || echo "all expressions bound to the variable"

Remove the panel before continuing.

Defect 3: the threshold that disagrees with the alert

Change the alert’s error-ratio threshold from 0.01 to 0.05 in prometheus/rules/service.yml, leaving the panel at 0.01:

sed -i 's|\[5m\])) > 0.01|[5m])) > 0.05|' prometheus/rules/service.yml
docker compose exec -T prometheus promtool check rules /etc/prometheus/rules/service.yml
curl -s -X POST http://localhost:9090/-/reload

Nothing complains. Now the payment service — whose FAIL_RATIO is 0.02 — sits in the gap: the panel is red, and no alert fires. The operator sees a red tile, checks the alert list, finds nothing, and concludes the alerting is broken. The next escalation of a red panel gets a little slower.

Restore 0.01, promtool check rules, and reload.

Defect 4: the variable that goes empty

Rename the target label from service to service_name in prometheus/prometheus.yml — a one-word edit in a file that has nothing to do with dashboards:

sed -i 's/service: checkout/service_name: checkout/; s/service: payment/service_name: payment/; s/service: cart/service_name: cart/' \
  prometheus/prometheus.yml
docker compose exec -T prometheus promtool check config /etc/prometheus/prometheus.yml
curl -s -X POST http://localhost:9090/-/reload
sleep 60

curl -sG http://localhost:9090/api/v1/label/service/values | jq -r '.data | length'

The label value list is now empty for the old label and populated for the new one. On the dashboard: the variable dropdown is empty, every panel says “No data”, the annotation query returns nothing, and the platform overview’s bar gauge is blank. No error, no log line, no alert — the alert rules aggregate by (service) too, so they also go quiet.

Restore the original label names, check, and reload before the validation section.

Task 7: Write the lint that catches all four

Three of the four are visible in the JSON, and the fourth is a query against live data. Put them in one script and give it a non-zero exit:

cat > "$LAB/lint-dashboards.sh" <<'LINT'
#!/usr/bin/env bash
# Fails the build on the dashboard defects that render without an error.
set -uo pipefail
SVC=dashboards/services/service.json
OVW=dashboards/platform/overview.json
rc=0

# 1. Every expression on a templated dashboard must be bound to the variable.
if jq -r '.panels[].targets[]?.expr' "$SVC" | grep -qv '\$service'; then
  echo "FAIL hard-coded service in an expression:"
  jq -r '.panels[].targets[]?.expr' "$SVC" | grep -v '\$service'
  rc=1
fi

# 2. Every panel whose expression divides one rate by another is a ratio and
#    must carry a unit, or the reader sees 0.003 and reads it as nothing.
#    Note the assignment, not a pipe into `while`: a loop on the right of a
#    pipe runs in a subshell, and rc=1 set there is discarded on exit.
for f in "$SVC" "$OVW"; do
  bad=$(jq -r '.panels[] | select((.targets[]?.expr // "") | test("\\) / sum"))
              | select(.fieldConfig.defaults.unit == null) | .title' "$f")
  if [ -n "$bad" ]; then
    echo "FAIL ratio panel without a unit in $f:"
    echo "$bad"
    rc=1
  fi
done

# 3. The panel threshold and the alert threshold must be the same number.
panel_err=$(jq -r '.panels[] | select(.title=="Error ratio")
                   | .fieldConfig.defaults.thresholds.steps[-1].value' "$SVC")
alert_err=$(grep -oE '\[5m\]\)\) > [0-9.]+' prometheus/rules/service.yml | head -1 | awk '{print $3}')
if [ "$panel_err" != "$alert_err" ]; then
  echo "FAIL error-ratio threshold drift: panel=$panel_err alert=$alert_err"
  rc=1
fi

# 4. The label the variable depends on must still have values.
n=$(curl -sfG http://localhost:9090/api/v1/label/service/values | jq -r '.data | length')
if [ "${n:-0}" -eq 0 ]; then
  echo "FAIL variable label 'service' has no values in Prometheus"
  rc=1
fi

[ "$rc" -eq 0 ] && echo "PASS all dashboard checks"
exit "$rc"
LINT
chmod +x "$LAB/lint-dashboards.sh"
cd "$LAB" && ./lint-dashboards.sh

Re-introduce any one defect and run it again. A check you have not watched fail is a check you do not know works.

Validation

cd "$LAB"
GF_PW="$(cat "$LAB/secrets/grafana_admin")"

echo "== both dashboards are provisioned, in their folders, and read-only"
curl -s -u "admin:$GF_PW" 'http://localhost:3000/api/search?type=dash-db' \
  | jq -r '.[] | "\(.folderTitle // "General")/\(.title) uid=\(.uid)"'
curl -s -u "admin:$GF_PW" http://localhost:3000/api/dashboards/uid/svc-lab \
  | jq '{provisioned: .meta.provisioned, canSave: .meta.canSave, folder: .meta.folderTitle}'

echo "== the variable resolves, and to the values we expect"
curl -sG http://localhost:9090/api/v1/label/service/values | jq -r '.data | sort | join(" ")'

echo "== every panel expression returns rows for a chosen service"
for q in \
  'sum(rate(http_server_requests_total{service="payment"}[5m]))' \
  'histogram_quantile(0.99, sum by (le) (rate(http_server_request_duration_seconds_bucket{service="payment"}[5m])))' \
  'process_resident_memory_bytes{service="payment"}'
do
  printf '%s -> ' "$(echo "$q" | cut -c1-40)"
  curl -sG http://localhost:9090/api/v1/query --data-urlencode "query=$q" \
    | jq -r '.data.result | length'
done

echo "== panel thresholds and alert thresholds agree"
./lint-dashboards.sh

echo "== ownership metadata is present on both dashboards"
for uid in svc-lab platform-lab; do
  curl -s -u "admin:$GF_PW" "http://localhost:3000/api/dashboards/uid/$uid" \
    | jq -r --arg u "$uid" '"\($u) tags=\(.dashboard.tags | join(",")) links=\(.dashboard.links | length)"'
done

Three checks the API cannot make for you. Open http://localhost:3000/d/svc-lab, select payment, and confirm: the error ratio tile reads as a percentage and not a bare decimal; the latency panel’s axis is labelled in seconds and not in bare numbers; and the restart annotation from Task 5 is drawn on the time-series panels. A unit is a rendering decision, and only rendering proves it.

Expected Outcome

• Two dashboards in two folders, both reporting provisioned: true and canSave: false.
• The service variable resolves to cart, checkout, payment, and every panel follows the selection.
• Clicking a bar on the overview lands on the service dashboard with both the service and the time range carried across.
• A restart of a generator draws an annotation on the service dashboard within one refresh.
• lint-dashboards.sh exits 0 on the restored dashboards, and non-zero for each of the four defects when re-introduced.

Troubleshooting

A dashboard does not appear in the search results. Grafana logs a provisioning error and carries on. docker compose logs grafana | grep -i provision names the file and the reason; the usual causes are invalid JSON (run jq empty on it) and a uid that collides with another dashboard.

Every panel says “No data” but Explore returns rows. The variable is empty or the panel filters on a label that does not exist. Check the variable first: curl -sG http://localhost:9090/api/v1/label/service/values. An empty list here explains the whole dashboard at once.

The data link navigates to /d/svc-lab?var-service= with nothing after it. ${__field.labels.service} had no label to resolve — the panel’s query lost its by (service) grouping, so the field carries no service label. Add the grouping back rather than hard-coding the link.

The annotation never appears. changes(process_start_time_seconds[5m]) > 0 is only non-empty for five minutes after a restart. Restart the container and look within that window, and confirm the metric exists at all: curl -sG http://localhost:9090/api/v1/query --data-urlencode 'query=process_start_time_seconds'.

A panel is red but no alert fires. Before assuming the alerting is broken, compare the two numbers — that is defect 3, and it is far more common than a broken Alertmanager. lint-dashboards.sh answers it in one second.

Cleanup

cd "$LAB"
docker compose down -v --remove-orphans --rmi local

docker volume ls --filter 'name=rb-obs-dashboard' --format '{{.Name}}'
docker ps --format '{{.Names}}'

cd "$HOME" && rm -rf "$LAB"

Production notes

Two of the decisions in this lab change shape at scale, and one does not.

One service dashboard per service does not scale; one templated dashboard does. The file you wrote serves three services and would serve two hundred unchanged. What does not survive the jump is a query variable with includeAll: true on a two-hundred-service fleet: the “All” selection runs every panel against every service and turns a two-second dashboard into a thirty-second one. Leave includeAll off on the service tier and put the fleet-wide view on the overview, where it is a single aggregate query.

Thresholds drift, so make them one artefact. In this lab the check compares two files. In production the same discipline usually means generating both the alert rule and the dashboard threshold from one source — a service catalogue entry, or a Jsonnet or Grafonnet template. Either mechanism is fine; having two hand-maintained numbers and a review that is supposed to notice is not.

Ownership is metadata plus a cadence, and the metadata is the easy half. The team: tag and the runbook link you added cost thirty seconds. The part that decays is the audit: a quarterly pass that lists dashboards with no team tag, panels that have returned no data for a month, and links that 404. Wire that pass into the same CI that runs the lint script, or it will not happen.

What You Learned

• A dashboard is a file, and everything good follows from that. Provisioned with allowUiUpdates: false, the dashboard can be reviewed, diffed, linted and rolled back. Every check in Task 7 exists only because the dashboard is text.
• The unit is not cosmetic. The same value renders as 0.003 or 0.30%, and the second one gets escalated. Thresholds colour the raw value either way, so a tile can be simultaneously red and unreadable.
• A hard-coded label is worse than an empty panel. An empty panel prompts a question; a panel confidently showing another service’s numbers does not. One grep over the expressions catches every instance.
• The panel threshold and the alert threshold are one decision written twice. When they disagree, the operator learns to distrust whichever one they checked second, and that distrust outlives the fix.
• The dashboard’s dependencies extend past the dashboard. A label rename in a scrape config emptied the variable, every panel, and — quietly — the alert rules that grouped by the same label. The dashboard going blank was the friendly half of that failure.