The Presentation Layer

The on-call engineer opens Grafana at 03:00. The dashboard titled “Checkout Service” has three panels. The first panel shows a red error rate. The second panel shows a green CPU usage. The third panel is empty. The on-call engineer clicks on the empty panel and the panel renders a single line: “datasource not found”. The dashboard is correct; the datasource was renamed during a maintenance window two weeks ago. The on-call engineer has a dashboard that renders an operational state that cannot be trusted.

The presentation layer is the operator-facing surface. The discipline of the layer is the discipline of the contract between the operator and the data. When the discipline is absent, the operator is paid to read the wrong answer.

What it is

The presentation layer is the set of components that render the data and notify the operator. In this course:

• Grafana 11.x for dashboards and unified alerting. The component that turns a query into a panel and a panel into a dashboard.
• Alertmanager for alert routing and deduplication. The component that turns a Prometheus alert into a PagerDuty notification, a Slack message, or an email.

The presentation layer is the only layer that the operator sees. The other layers are heard about. The presentation layer is the surface the operator looks at when the system is on fire.

Why a sysadmin cares

The presentation layer is where the system meets the human. The four risks:

1. Unowned dashboards. A dashboard exists with no owner. The queries break. Nobody notices. The on-call engineer clicks on the dashboard and gets a wall of “datasource not found” panels.
2. Alert fatigue. The alertmanager fires every minute. The on-call engineer mutes the channel. The real alert arrives. The channel is muted. The page is missed.
3. Drift between dashboard and data. The dashboard is rendered against a datasource UID that no longer exists. The panel renders a state that is not the data.
4. Paging without context. The alertmanager fires with summary: "CPU above 80%". The on-call engineer has to start the investigation from the page.

The presentation layer is the only layer where the engineering discipline of the platform team directly shapes the operational experience of the operator.

How it works

The mental model: the operator reads a dashboard, the operator receives a page, the operator takes an action. The presentation layer is the bridge between the two.

  +---------- Grafana ----------+         +--------- Alertmanager ---------+
  |                             |         |                                |
  |  datasource  -- query       | <-----  |  prometheus rule  -- alerts   |
  |  panel       -- render      |         |  (silences, inhibitions,       |
  |  dashboard   -- correlate   |         |   routing tree)                |
  |  variable    -- pivot       |         |                                |
  |  alert rule  -- fire        | ----->  |  receiver (Slack, PagerDuty,   |
  |  (Grafana 11 unified)       |         |   email, webhook)              |
  |                             |         |                                |
  +-----------------------------+         +--------------------------------+
            |                                          |
            v                                          v
       operator view                                operator page

The alert path is the most operationally important. A Prometheus rule fires. The alertmanager receives the alert. The alertmanager applies the routing tree. The alertmanager groups the alert. The alertmanager deduplicates repeats. The alertmanager delivers to the receiver that the alert matches. The receiver page the operator.

The dashboard path is the investigation surface. A panel queries a datasource. The datasource returns a result. The panel renders the result. The dashboard correlates panels through variables. The operator pivots from one panel to the next.

Under the hood

Grafana renders a panel by:

1. Loading the panel definition from the dashboard JSON.
2. Resolving the panel’s template variables.
3. Issuing the query to the configured datasource.
4. Receiving the response.
5. Applying the panel’s transformations.
6. Rendering the panel to the operator’s browser.

The datasource is the bridge to the backend. A Grafana datasource is a YAML/JSON definition that maps a UID to a backend URL, an authentication credential, and a default database. The UID is the stable identifier; the URL is the operational detail.

Alertmanager deduplicates alerts by a label fingerprint. The fingerprint is the hash of the alert’s labels. Two alerts with the same labels are the same alert. The grouping key is the set of labels that the routing tree uses to decide the receiver.

The routing tree is a YAML file with a route block and a receivers block. The route matches alerts by label selector and forwards to the receiver. The receiver is the notification target.

How to configure it

Grafana provisioned datasource. The right pattern is provisioned YAML in a repository, not a hand-edited datasource in the UI. The provisioning file is read at startup.

# /etc/grafana/provisioning/datasources/datasources.yaml
apiVersion: 1

datasources:
  - name: Prometheus
    uid: prometheus-prod
    type: prometheus
    access: proxy
    url: http://prometheus.monitoring.svc:9090
    isDefault: true
    basicAuth: true
    basicAuthUser: grafana
    secureJsonData:
      basicAuthPassword: ${PROM_BASIC_AUTH_PASSWORD}
    jsonData:
      timeInterval: 15s
      httpMethod: POST

  - name: Loki
    uid: loki-prod
    type: loki
    access: proxy
    url: http://loki.monitoring.svc:3100
    basicAuth: true
    basicAuthUser: grafana
    secureJsonData:
      basicAuthPassword: ${LOKI_BASIC_AUTH_PASSWORD}

  - name: Tempo
    uid: tempo-prod
    type: tempo
    access: proxy
    url: http://tempo.monitoring.svc:3200
    jsonData:
      httpMethod: GET
      tracesToLogsV2:
        datasourceUid: loki-prod
      serviceMap:
        datasourceUid: prometheus-prod
      nodeGraph:
        enabled: true

The uid is the stable identifier. The dashboard panel references the datasource by uid, not by name. The renames of the datasource do not break the panel.

Grafana provisioned dashboard. The dashboard JSON is committed to a repository and provisioned at startup:

# /etc/grafana/provisioning/dashboards/dashboards.yaml
apiVersion: 1

providers:
  - name: service-dashboards
    orgId: 1
    folder: Services
    type: file
    disableDeletion: false
    updateIntervalSeconds: 30
    options:
      path: /var/lib/grafana/dashboards/services

Alertmanager routing tree. The routing tree is the operational contract. The right pattern is one that separates symptom alerts from cause alerts and routes them to different receivers.

# /etc/alertmanager/alertmanager.yml
global:
  resolve_timeout: 5m
  smtp_smarthost: smtp.internal:25
  smtp_from: alertmanager@monitoring.internal
  smtp_require_tls: false

route:
  receiver: default
  group_by: [alertname, cluster, service]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers:
        - severity="critical"
      receiver: pagerduty-critical
      group_wait: 10s
      repeat_interval: 1h
    - matchers:
        - severity="warning"
      receiver: slack-warnings
      group_wait: 1m
    - matchers:
        - severity="info"
      receiver: slack-info

inhibit_rules:
  - source_matchers: [severity="critical"]
    target_matchers: [severity="warning"]
    equal: [cluster, service]

receivers:
  - name: default
    slack_configs:
      - api_url: https://hooks.slack.com/services/XXX
        channel: "#alerts-default"
        send_resolved: true

  - name: pagerduty-critical
    pagerduty_configs:
      - service_key_file: /etc/alertmanager/pagerduty.key
        severity: critical
        send_resolved: true

  - name: slack-warnings
    slack_configs:
      - api_url: https://hooks.slack.com/services/XXX
        channel: "#alerts-warnings"
        send_resolved: true

  - name: slack-info
    slack_configs:
      - api_url: https://hooks.slack.com/services/XXX
        channel: "#alerts-info"
        send_resolved: true

The inhibit_rules block is the discipline. A critical alert on the same cluster and service suppresses the warning alerts. The on-call engineer receives one page, not ten.

Prometheus alert rule. The rule is the input. The alertmanager is the output. The rule fires a single alert when the predicate holds.

# /etc/prometheus/rules/checkout.yml
groups:
  - name: checkout
    rules:
      - alert: CheckoutHighErrorRate
        expr: |
          sum(rate(http_requests_total{job="checkout", status=~"5.."}[5m]))
          /
          sum(rate(http_requests_total{job="checkout"}[5m]))
          > 0.05
        for: 2m
        labels:
          severity: critical
          service: checkout
          cluster: prod-eu-west-1
        annotations:
          summary: "Checkout error rate above 5%"
          description: "Checkout is returning 5xx for more than 5% of requests for 2 minutes."
          runbook_url: "https://runbooks.internal/checkout-high-error-rate"
          dashboard_url: "https://grafana.internal/d/checkout-prod"

The runbook_url is the canonical reference. The dashboard_url is the investigation surface. The summary is what the on-call engineer sees on the page.

How to validate it

CONFIGURATION — validate the configs before reload.

amtool check-config /etc/alertmanager/alertmanager.yml
# Valid configuration file

READ-ONLY — confirm the alertmanager is routing.

curl -sf http://alertmanager.monitoring.svc:9093/-/ready
# Alertmanager is ready.

curl -sf -u admin:admin http://alertmanager.monitoring.svc:9093/api/v2/silences | jq '.[] | {id, matchers, status}'
# Confirm the silence list is correct.

READ-ONLY — confirm the Grafana datasource is healthy.

curl -sf -u admin:admin \
  http://grafana.monitoring.svc:3000/api/datasources/uid/prometheus-prod/health | jq .
# { "status": "ok", "message": "Data source is working" }

READ-ONLY — confirm the alert is firing.

curl -sf 'http://prometheus.monitoring.svc:9090/api/v1/alerts' | jq '.data.alerts[] | select(.labels.alertname=="CheckoutHighErrorRate")'
# { "labels": { "alertname": "CheckoutHighErrorRate", "severity": "critical", "service": "checkout" },
#   "state": "firing", "activeAt": "2026-08-13T03:14:00Z" }

READ-ONLY — confirm the alertmanager is delivering.

curl -sf -u admin:admin http://alertmanager.monitoring.svc:9093/api/v2/alerts | jq '.[] | {labels, receivers}'
# { "labels": { "alertname": "CheckoutHighErrorRate", "severity": "critical" },
#   "receivers": [{ "name": "pagerduty-critical" }] }

READ-ONLY — confirm the dashboard renders.

curl -sf -u admin:admin \
  http://grafana.monitoring.svc:3000/api/dashboards/uid/checkout-prod | jq '.dashboard.title'
# "Checkout Service"

SERVICE-IMPACT — reload after a config change.

curl -sf -X POST -u admin:admin \
  http://alertmanager.monitoring.svc:9093/-/reload
# Alertmanager reloaded.

curl -sf -X POST -u admin:admin \
  http://grafana.monitoring.svc:3000/api/admin/provisioning/dashboards/reload
# Dashboard provisioning reloaded.

How it can fail

1. Datasource UID renamed. The dashboard panel references the old UID. The panel renders “datasource not found”. The fix is to commit the datasource config with a stable UID and reference the UID from the dashboard.
2. Alertmanager routing tree mismatch. The alert route matches a label that the Prometheus rule does not emit. The alertmanager receives the alert but no route matches. The alert falls through to the default receiver.
3. PagerDuty silent failure. The PagerDuty integration is misconfigured. The alertmanager receives the alert but the notification is dropped. The on-call engineer never receives the page.
4. Alert spam. The Prometheus rule is too tight. The alert fires every minute. The group_wait and repeat_interval are ignored. The on-call engineer mutes the channel.
5. Dashboard drift. The dashboard is provisioned from one repository. The reality is in another. The dashboards in Grafana are the ones in the repository, not the ones the operator edits in the UI.
6. Runbook link broken. The runbook_url annotation points to a document that was renamed. The on-call engineer clicks the link and gets a 404.

How to troubleshoot it

The diagnostic order for “the dashboard is wrong or the alert is missing”:

1. Is the alert firing? The Prometheus /api/v1/alerts endpoint is the source of truth. The alertmanager is the next step.
2. Is the alertmanager delivering? The /api/v2/alerts endpoint shows the active alerts and their receivers. The route that fired is the next step.
3. Is the receiver configured? The amtool config shows the routing tree. The test command sends a synthetic alert.
4. Is the dashboard rendering? The Grafana UI is the source of truth. The /api/datasources/uid/<uid>/health endpoint shows the datasource health.
5. Is the query correct? The Prometheus query returns the result that the operator expects. The Grafana Explore view is the right place to confirm.

Security implications

The presentation layer is the operator-facing surface. The risks:

• Grafana RBAC. The Grafana user has roles. The roles map to organizations, folders, and dashboards. The default admin role is too open for production. The right pattern is a least-privilege role per team.
• Alertmanager receivers. The PagerDuty and Slack integrations hold credentials. The credentials are read from secret files with strict permissions.
• Datasource credentials. The Prometheus and Loki datasources hold credentials. The secureJsonData field is the right place. The credentials are not in the committed YAML.
• Runbook URLs. The runbook_url annotation is visible to the on-call engineer. The URL should not expose internal-only infrastructure to the public internet.
• Dashboard export. The Grafana dashboard export contains the queries. The export is a credential. The repository is the right home.

Performance implications

The presentation layer is the query layer. The four knobs:

• Refresh interval. The default is 30s. The right pattern is 30s for operational dashboards, 5m for long-tail dashboards.
• Query range. The default is 1 hour. The right pattern is the smallest range that answers the question.
• Panel count. The dashboard renders every panel on every refresh. A 50-panel dashboard at 30s refresh is 50 queries every 30 seconds. The right pattern is to split the dashboard into multiple dashboards.
• Variable resolution. A variable with a query that scans the entire TSDB is the most expensive query. The right pattern is to use a small set of bounded values.

Production guidance

• Provision everything. Datasources, dashboards, alert rules, and alertmanager routing. The repository is the source of truth; the UI is a viewer.
• Use stable UIDs. The datasource UID is the contract. Renames are a maintenance event.
• Test the alert path. A synthetic alert that fires every minute is a useful canary. The amtool test command is the right tool.
• Document the runbook. The runbook_url annotation is the contract. The runbook is the document.
• Monitor the dashboard rendering. The grafana_dashboard_panel_render_time metric is the early-warning signal.

Verification

You should now be able to answer:

• What is the difference between a provisioned dashboard and an ad-hoc dashboard?
• Trace an alert from a Prometheus rule to a PagerDuty notification.
• What is the difference between symptom alerts and cause alerts in the routing tree?
• Why is the datasource UID the right contract for a panel reference?
• What is the failure mode when the routing tree does not match the alert labels?

Quiz