Service Dashboards

The service dashboard is the only dashboard most engineers ever open twice. It is the per-service operating page — what the production owner reads in the morning, what the on-call engineer opens during an incident, what the new hire studies on day two.

A good service dashboard answers four questions without leaving the page:

1. Is the SLO healthy for this service?
2. Where is the time going?
3. Which dependency is the bottleneck?
4. What changed since I last looked?

The dashboard that answers all four takes one hour to design and saves the company dozens of hours per quarter.

What it is

A service dashboard is the per-service slice of the platform overview. One per named service (checkout, payment-svc, cart). It runs RED method (Rate, Errors, Duration) for the service itself and USE method (Utilization, Saturation, Errors) for every resource the service depends on. It also renders dependency-tier health, recent deploys, and recent alerts.

A 50–80 panel dashboard, organised in rows, runs in 2–4 seconds on a typical Prometheus and one Grafana. Anything above 100 panels is wallpaper. Anything below 30 is under-instrumented.

Why a sysadmin cares

A service dashboard gets used in three contexts:

• Steady state — every morning, the service owner opens it, verifies green, closes the tab. If they do not do this, ownership has decayed.
• Incident — the on-call engineer is here. Every panel that is irrelevant lengthens time-to-answer. Every panel that is missing forces a pivot. The structure has to be right.
• Post-incident review — the same engineer reviews what changed. The “recent alerts” and “deploy annotation” rows are the entry point.

The dashboard has to be readable and complete at the same time. Readability comes from row ordering. Completeness comes from never letting a panel that answers an open question be missing.

How it works

The skeleton is rows by purpose:

  +--------------------------------------------------+
  | ROW 0:  HEADER                                   |
  |  service name + SLO tile (colour-coded)          |
  |  current p99, error rate, RPS                    |
  +--------------------------------------------------+
  | ROW 1:  RED (Rate, Errors, Duration)             |
  |  RPS   /  Error rate  /  p50 p95 p99 latency     |
  +--------------------------------------------------+
  | ROW 2:  USE (per pod)                            |
  |  CPU util  /  Memory sat  /  Net drops  /  FDs   |
  +--------------------------------------------------+
  | ROW 3:  DEPENDENCIES (tier visibility)           |
  |  DB latency  /  Cache hit ratio  /  Queue depth  |
  |  external calls p99  /  error budget per dep     |
  +--------------------------------------------------+
  | ROW 4:  RECENT EVENTS                            |
  |  Deploy annotations  /  alert firings (last 24h) |
  +--------------------------------------------------+
  | ROW 5:  LOGS (filtered to $service)              |
  |  Loki log panel  /  Trace search link            |
  +--------------------------------------------------+

Each row pivots to a lower tier. RED is for the service. USE is for the pods that host it. The dependency row reaches up the call chain to the things the service calls. The event row correlates changes with the panels above.

URL-driven selection is what binds the dashboard to a specific service. When the operator clicks a link on the platform overview that says /d/svc-checkout?var-service=checkout, Grafana sets $service=checkout and every panel re-queries. The dashboard has one physical page and N virtual pages, one per service.

Dependency-tier observability is the row most teams get wrong. The common shape is to monitor the service but forget to monitor the things the service depends on. A 5xx spike in checkout is often a 5xx spike in payment-svc, which is itself a slow query in postgres. The dependency row catches the upstream cause before the operator has to click into a separate dashboard.

How to configure it

The skeleton below is a working RED+USE+dependencies row set. The dashboard JSON for a service is templated by service name in the operations repo:

{
  "uid": "svc-checkout",
  "title": "Service: $service",
  "tags": ["tier:service"],
  "templating": {
    "list": [
      {
        "name": "service",
        "type": "query",
        "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
        "query": "label_values(http_server_requests_seconds_count, service)",
        "current": { "text": "checkout", "value": "checkout" },
        "refresh": 2,
        "includeAll": false,
        "multi": false
      },
      {
        "name": "instance",
        "type": "query",
        "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
        "query": "label_values(http_server_requests_seconds_count{service=\"$service\"}, instance)",
        "refresh": 2,
        "includeAll": true,
        "multi": true
      }
    ]
  },
  "panels": [
    {
      "id": 1, "type": "timeseries", "title": "Rate by status",
      "gridPos": { "x": 0, "y": 0, "w": 12, "h": 8 },
      "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
      "targets": [{
        "refId": "A",
        "expr": "sum by (status) (rate(http_server_requests_seconds_count{service=\"$service\"}[5m]))"
      }]
    },
    {
      "id": 2, "type": "timeseries", "title": "Latency p50/p95/p99",
      "gridPos": { "x": 12, "y": 0, "w": 12, "h": 8 },
      "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
      "targets": [
        {
          "refId": "A",
          "expr": "histogram_quantile(0.50, sum by (le) (rate(http_server_requests_seconds_bucket{service=\"$service\"}[5m])))",
          "legendFormat": "p50"
        },
        {
          "refId": "B",
          "expr": "histogram_quantile(0.95, sum by (le) (rate(http_server_requests_seconds_bucket{service=\"$service\"}[5m])))",
          "legendFormat": "p95"
        },
        {
          "refId": "C",
          "expr": "histogram_quantile(0.99, sum by (le) (rate(http_server_requests_seconds_bucket{service=\"$service\"}[5m])))",
          "legendFormat": "p99"
        }
      ],
      "fieldConfig": {
        "defaults": {
          "unit": "s",
          "thresholds": {
            "mode": "absolute",
            "steps": [
              { "color": "green",  "value": null },
              { "color": "yellow", "value": 0.5 },
              { "color": "red",    "value": 1.0 }
            ]
          }
        }
      }
    },
    {
      "id": 3, "type": "timeseries", "title": "CPU saturation per pod",
      "gridPos": { "x": 0, "y": 8, "w": 24, "h": 8 },
      "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
      "targets": [{
        "refId": "A",
        "expr": "max by (instance) (rate(container_cpu_usage_seconds_total{namespace=\"$service\"}[5m]))"
      }],
      "fieldConfig": {
        "defaults": {
          "unit": "percentunit",
          "thresholds": {
            "mode": "absolute",
            "steps": [
              { "color": "green",  "value": null },
              { "color": "yellow", "value": 0.7 },
              { "color": "red",    "value": 0.9 }
            ]
          }
        }
      }
    },
    {
      "id": 4, "type": "timeseries", "title": "DB p99 latency",
      "gridPos": { "x": 0, "y": 16, "w": 12, "h": 8 },
      "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
      "targets": [{
        "refId": "A",
        "expr": "histogram_quantile(0.99, sum by (le) (rate(pg_stat_activity_duration_seconds_bucket{service=\"$service\"}[5m])))"
      }],
      "fieldConfig": {
        "defaults": {
          "unit": "s",
          "thresholds": {
            "mode": "absolute",
            "steps": [
              { "color": "green",  "value": null },
              { "color": "yellow", "value": 0.05 },
              { "color": "red",    "value": 0.2 }
            ]
          }
        }
      }
    },
    {
      "id": 5, "type": "logs", "title": "Logs: $service",
      "gridPos": { "x": 0, "y": 24, "w": 24, "h": 10 },
      "datasource": { "type": "loki", "uid": "${DS_LOKI}" },
      "targets": [{
        "refId": "A",
        "expr": "{service=\"$service\"}"
      }]
    }
  ]
}

Three details to study in the JSON:

• $service everywhere. No hard-coded service name. The same JSON template renders for every service.
• Threshold per row. Each row carries its own threshold semantics. Latency p99 in seconds, CPU in percent, DB latency in seconds. Different rows are different unit namespaces.
• Loki log panel at the bottom. Logs go below the metrics because the eye-flow is: SLO tile, latency, errors, deps, then logs for evidence.

How to validate it

# READ-ONLY: confirm $service variable resolves to more than zero
curl -s -u admin:admin \
  "http://grafana:3000/api/ds/query?ds=prom-prod" \
  -H 'content-type: application/json' \
  -d '{
    "queries": [{
      "refId": "A",
      "datasource": { "type": "prometheus", "uid": "prom-prod" },
      "expr": "label_values(http_server_requests_seconds_count, service)",
      "instant": true
    }],
    "from": 1700000000,
    "to":   1700000600
  }' | jq '.results.A.frames[0].data.values[1][]'

Expected output: a list of service names. An empty list means either the upstream service is not exporting the metric or the label has been renamed in the instrumentation library.

# READ-ONLY: confirm the panels return data after var substitution
curl -s -u admin:admin \
  "http://grafana:3000/api/ds/query?ds=prom-prod" \
  -H 'content-type: application/json' \
  -d '{
    "queries": [{
      "refId": "A",
      "datasource": { "type": "prometheus", "uid": "prom-prod" },
      "expr": "sum by (status) (rate(http_server_requests_seconds_count{service=\"checkout\"}[5m]))"
    }],
    "from": 1700000000,
    "to":   1700000600
  }' | jq '.results.A.frames[0].data.values | length'

A non-zero length means the panel has rows. Zero means the service label does not match the underlying export.

Open the dashboard in the browser. Type ?var-service=checkout in the URL. Every panel must update within one refresh interval. A panel that ignores the var is hard-coded — fix the JSON, not the live dashboard.

How it can fail

The high-frequency failure modes:

1. Empty $service after a label rename — the instrumentation library renamed the service label to service_name. The query returns empty. Symptom: every panel shows “No data”. Fix the query or fix the instrumentation; do not patch the dashboard.
2. Panel does not refresh on var change — the panel was added via the UI with a hard-coded service=checkout in the expression. Symptom: panel keeps showing the previous service. Audit the JSON for hard-coded labels.
3. USE row panels missing instance aggregation — a rate query without by (instance) aggregates across the whole fleet. Symptom: the CPU panel shows one bar summing every pod. Add by (instance) or use max by (instance).
4. Loki log panel unbounded cardinality — {service="$service"} returns every log line ever. Symptom: dashboard times out. Bound the query with |~ "ERROR|WARN" or add a __limit clause.
5. Dependency row points at the wrong service — the dashboard inherits a template whose dependency expressions reference service=postgres regardless of $service. Symptom: DB panel always shows the postgres service even for the cart service. Bind dependencies through a service catalog.
6. Annotations disabled because datasource uid mismatch — the deploy annotation query references ${DS_PROMETHEUS} but the dashboard JSON was edited to use a hard-coded uid. Symptom: deploy markers vanish. Restore the alias.

How to troubleshoot it

1. Open the variable panel. Confirm $service and $instance resolve to expected values.
2. Click one panel, edit, and inspect the response. Hover the panel, expand the inspector, confirm the data source returned rows.
3. Run the panel query in Explore for the same $service and time range. If Explore returns rows and the panel is empty, the dashboard JSON has a mismatched alias or a hard-coded service.
4. Confirm dependency expressions with a service catalog lookup. A dependency row that references an irrelevant service is the most common silent failure.
5. Reload the dashboard after any provisioning change.

Security implications

• Service dashboards expose PII in template variable values. An operator’s shared link to ?var-user_id=... can leak. Audit URL patterns for path parameters that may embed user identifiers.
• Logs panels can leak secrets in line values. A Loki query that returns raw log lines with secret patterns must be redacted at ship time, not at panel time. Audit the Grafana Loki query language and the log redaction pipeline together.
• The service dashboard may be public. A publicly-readable service dashboard exposes dependency topology. Audit the folder permissions for service dashboards the same way the overview audit runs.

Performance implications

• 50-80 panels × 1 wave per ~8 queries = 6-10 waves of evaluation per refresh. That is 30 seconds on a slow Prometheus and 4 seconds on a recording-rule-backed one.
• Variable evaluation runs every time the dashboard opens. Two variables ($service, $instance) are fine. Five are expensive.
• Logs panels are the slowest individual panel because they pull the raw event stream. Always put logs at the bottom of the dashboard so the metrics render first.
• Refresh every 30 seconds is the production default. Lower it (e.g. 10 seconds) and the dashboard load doubles. Raise it (e.g. 60 seconds) and the dashboard lags the SLO.

Production guidance

• One service dashboard per service. Generated from a Jsonnet / Go template; never hand-edited per service.
• 50-80 panels organised in rows: header, RED, USE, dependencies, recent events, logs.
• Every query references $service. No hard-coded service names in expressions.
• Every Prometheus expression that hits raw series is recorded in a recording rule.
• Logs at the bottom of the dashboard. Metrics on top.
• All panels provisioned by the same template; no manual drift between services.

Verification

You should now be able to answer:

• What are the four rows of a service dashboard, top to bottom?
• Why does every PromQL expression on the service dashboard reference $service instead of a hard-coded service name?
• What does the dependency-tier row prevent?
• When does a panel belong on the instance dashboard rather than on the service dashboard?

Quiz