Data Source Availability

The on-call engineer opened the SLO dashboard during a customer-impacting incident. Every panel was green. They opened the service overview on the same data source. Every panel was red. They switched back to the SLO dashboard. Still green. They opened the Grafana Explore view and typed the panel’s query by hand. The query returned data. The data source was up. The SLO dashboard was lying.

The cause was a UID mismatch. A provisioning change had renamed the data source UID from prom-prod to prom-prod-v2 and updated every panel that referenced it directly. But one panel still had the literal UID prom-prod in its datasource field. Grafana fell back to the default data source for that panel — a different Prometheus, a staging instance, with no production traffic. The panel rendered green because the staging instance had no failures to report.

This is what data source availability is for. A Grafana 11 dashboard references one or more data sources by UID. Every panel renders against a data source. The dashboard renders correctly only if every referenced UID exists, points at a healthy backend, has working credentials, and returns data for the panel’s query. A dashboard that references a missing or unhealthy UID renders something, but not what the operator expects.

Data source availability is the per-UID contract that every data source referenced by the dashboard exists, is healthy, and has working credentials against the backend.

What it is

A Grafana 11 data source is a plugin-instantiated connection to a backend (Prometheus, Loki, Tempo, OpenTelemetry Collector, MySQL, PostgreSQL, etc.). The data source has:

  +----------------+----------------------------------------------+
  | Field          | Purpose                                      |
  +----------------+----------------------------------------------+
  |  id            | Numeric internal identifier                 |
  |                | (not used by dashboards)                    |
  +----------------+----------------------------------------------+
  |  uid           | String identifier; the dashboard's         |
  |                | reference key                               |
  +----------------+----------------------------------------------+
  |  name          | Human-readable label                       |
  +----------------+----------------------------------------------+
  |  type          | Plugin type (prometheus, loki, tempo, ...) |
  +----------------+----------------------------------------------+
  |  url           | The backend's HTTP endpoint                |
  +----------------+----------------------------------------------+
  |  access        | proxy (Grafana proxies) or direct          |
  |                | (browser calls)                            |
  +----------------+----------------------------------------------+
  |  basicAuth     | Whether basic auth is enabled              |
  +----------------+----------------------------------------------+
  |  isDefault     | Whether this is the default for the org    |
  +----------------+----------------------------------------------+

A panel references the data source by UID:

{
  "type":       "timeseries",
  "title":      "Request rate",
  "datasource": { "type": "prometheus", "uid": "prom-prod" },
  "targets":    [ { "refId": "A",
                    "datasource": { "type": "prometheus",
                                    "uid": "prom-prod" },
                    "expr": "sum(rate(http_requests_total[5m]))" } ]
}

The dashboard JSON also references data sources in templating.list[].datasource (for query variables) and in panels[].datasource (the panel-level default).

Data source availability has four checks:

1. UID exists. The UID is registered in the Grafana data source store.
2. Plugin is healthy. The plugin (e.g., Prometheus) can instantiate a connection to the backend.
3. Backend is reachable. The url field is reachable over the network.
4. Credentials are valid. Basic auth, bearer tokens, or TLS client certs are accepted by the backend.

A failure at check 1 makes the panel render “Data source not found” silently. A failure at check 2 makes the panel show a plugin error. A failure at check 3 makes the panel show a connection error. A failure at check 4 makes the panel show an authentication error.

Why a sysadmin cares

Three operational pains map directly to data source availability:

1. The silent UID drift. A provisioning change renames a data source UID; one panel still references the old UID; Grafana falls back to the default; the panel renders data from a different environment than the dashboard title says.
2. The dead backend. A Prometheus pod restarts; the service comes back; the data source health check eventually recovers; the panel shows “No data” for several minutes while the data source is restarting.
3. The expired credential. A long-lived API token used for a data source’s basic auth expires; every panel shows a 401; the operator opens the panel and sees a credential error; the credential rotation has been forgotten.

The wrong shape shows up as a panel that silently renders data from a different data source than the dashboard intends, or as a dashboard that uniformly fails while the underlying Prometheus is healthy.

Health vs reachability

Two distinct properties are worth distinguishing:

• Health. The data source plugin can talk to the backend and the backend returned a successful response to a probe query. Health is per-UID; the /api/datasources/uid/{uid}/health endpoint returns a JSON status.
• Reachability. The backend’s HTTP endpoint is reachable over the network. Reachability is per-URL; the curl $url/-/ready (Prometheus) or $url/ready (Loki) endpoint returns 200 when ready.

A data source can be healthy but not reachable (network partition); reachable but not healthy (the backend’s storage is broken); neither (the backend is down); or both (the steady state). Dashboard availability requires both.

How it works

The data-source-availability pipeline:

  dashboard JSON
        |
        v
  +-------------------------+
  |  extract every UID      |   jq '.panels[].datasource.uid,
  |  reference              |   .panels[].targets[].datasource.uid,
  |                         |   .templating.list[].datasource.uid'
  +-------------------------+
        |
        v
  +-------------------------+
  |  dedupe UIDs            |   sort -u
  +-------------------------+
        |
        v
  +-------------------------+
  |  for each UID:          |
  |  /api/datasources/      |   curl + jq '.message' for health
  |  uid/{uid}/health       |
  +-------------------------+
        |
        v
  +-------------------------+
  |  for each UID:          |
  |  probe backend readiness|   curl $url/-/ready (Prometheus) or
  |                         |   $url/ready (Loki)
  +-------------------------+
        |
        v
  +-------------------------+
  |  report                 |   list of missing UIDs, unhealthy
  |                         |   UIDs, unreachable backends
  +-------------------------+

How to configure it

The canonical pattern: a CI script that walks every panel and every variable, collects the UID references, and probes each one against the live Grafana.

#!/usr/bin/env bash
# scripts/check-datasources.sh
# Severity: READ-ONLY against staging Grafana.
set -euo pipefail

GRAFANA_URL=${GRAFANA_URL:-https://grafana-staging.example.com}
ADMIN_USER=${ADMIN_USER:-admin}
ADMIN_PASS=${ADMIN_PASS:?admin password required}

extract_uids() {
  jq -r '
    [ .panels[]?.datasource.uid,
      .panels[]?.targets[]?.datasource.uid,
      .templating.list[]?.datasource.uid ]
    | map(select(. != null and . != ""))
    | unique
    | .[]
  ' "$1"
}

check_uid() {
  local json_file=$1
  local uid=$2
  local health
  health=$(curl -s -u admin:$ADMIN_PASS \
    "$GRAFANA_URL/api/datasources/uid/$uid/health")
  if ! echo "$health" | jq -e '.message == "Success"' > /dev/null; then
    echo "FAIL: $json_file references unhealthy UID $uid:"
    echo "      $health"
    return 1
  fi
}

for json_file in dashboards/*.json; do
  for uid in $(extract_uids "$json_file"); do
    check_uid "$json_file" "$uid"
  done
done

A minimal data source provisioning manifest the dashboard expects:

# /etc/grafana/provisioning/datasources/prometheus.yml
apiVersion: 1

datasources:
  - name: 'Prometheus Production'
    uid:  'prom-prod'
    type: 'prometheus'
    access: proxy
    url:   'http://prometheus.monitoring.svc:9090'
    isDefault: true
    json_data:
      timeInterval: '15s'
      httpMethod: 'POST'
  - name: 'Loki Production'
    uid:  'loki-prod'
    type: 'loki'
    access: proxy
    url:   'http://loki.monitoring.svc:3100'
    json_data:
      maxLines: 1000
  - name: 'Tempo Production'
    uid:  'tempo-prod'
    type: 'tempo'
    access: proxy
    url:   'http://tempo.monitoring.svc:3100'

The dashboard’s panels reference these UIDs. The CI step verifies each UID exists, is healthy, and the backend URL is reachable.

How to validate it

Four checks confirm the per-UID contract is live.

Severity: READ-ONLY.

# 1. Every UID the dashboard references is registered
#    in the Grafana data source store.
for uid in $(extract_uids dashboards/svc-overview.json); do
  curl -s -u admin:$ADMIN \
    -o /dev/null -w "%{http_code}" \
    "https://grafana.example.com/api/datasources/uid/$uid"
  echo " $uid"
done
# 200 prom-prod
# 200 loki-prod
# 200 tempo-prod

# 2. Every UID returns a healthy response from the
#    health endpoint.
for uid in $(extract_uids dashboards/svc-overview.json); do
  curl -s -u admin:$ADMIN \
    "https://grafana.example.com/api/datasources/uid/$uid/health" \
    | jq -r ".message"
done
# Success
# Success
# Success

# 3. The backend itself is reachable and ready.
curl -s -o /dev/null -w "%{http_code}\n" \
  http://prometheus.monitoring.svc:9090/-/ready
# 200
curl -s -o /dev/null -w "%{http_code}\n" \
  http://loki.monitoring.svc:3100/ready
# 200

# 4. The panel-level data source UID matches the
#    target-level data source UID. A panel whose
#    datasource field is missing or different from the
#    target's datasource is the canonical UID drift
#    failure shape.
curl -s -u admin:$ADMIN \
  https://grafana.example.com/api/dashboards/uid/svc-overview \
  | jq -r '.dashboard.panels[]?
            | select(.datasource.uid != null)
            | {title,
               panel_uid: .datasource.uid,
               target_uid: .targets[0].datasource.uid}'

How it can fail

Six failure shapes appear repeatedly with data source availability:

1. UID renamed in provisioning. A data source provisioning change renames the UID from prom-prod to prom-prod-v2; one panel still references the old UID. Symptom: the panel renders data from the default data source, which is a different Prometheus; the panel does not error.
2. Backend restarted. A Prometheus pod restarts after a configuration change. The data source health check fails for 30-60 s while the pod comes back. Symptom: every panel shows “No data” or a connection error during the restart window.
3. Network partition. The Grafana instance cannot reach the backend over the network. Symptom: the health check fails with a timeout; the panel shows a “Network error: undefined” message.
4. Credential expired. A long-lived API token used for the data source’s basic auth has expired. Symptom: every panel shows a 401 error; the credential rotation has been forgotten.
5. Plugin stopped. The Grafana plugin process for the data source has crashed or been disabled. Symptom: the health endpoint returns a plugin error; panels show “Plugin not loaded”.
6. Data source deleted. A provisioning change removed the data source. Symptom: the panel shows “Data source not found”; the Grafana UI offers to create a new default data source.

How to troubleshoot it

The diagnostic order:

1. Open the panel. Click Edit. Inspect the datasource.uid field. Copy the UID.
2. Call /api/datasources/uid/{uid}. If the endpoint returns 404, the UID is missing; check the provisioning manifest for the expected UID.
3. Call /api/datasources/uid/{uid}/health. If the endpoint returns a failure, the plugin or the backend is unhealthy; the response body contains the failure reason.
4. Curl the backend directly. curl $url/-/ready for Prometheus, curl $url/ready for Loki, curl $url/status for Tempo. If the backend itself returns non-200, the issue is in the backend, not in Grafana.
5. Check the credential. Open the data source settings; click Save & Test. If the test fails with a 401, the credential has expired; rotate it.
6. Check the panel target. Compare the panel’s datasource.uid to the target’s datasource.uid. If they differ, the panel has drifted; fix the JSON.

Security implications

• Data source credentials are stored in Grafana’s SQL store. The secure_json_data column is encrypted at rest; the encryption key lives in grafana.ini. A compromised key reveals every data source credential.
• The access field controls who reaches the backend. proxy mode means Grafana proxies the request; the browser does not see the backend’s URL. direct mode means the browser calls the backend directly; the backend’s URL must be reachable from the viewer’s network. The right default is proxy for production data sources.
• Basic auth credentials are passed through the proxy. The basic auth user and password are stored alongside the data source; rotate them in the provisioning manifest.
• Bearer tokens and TLS client certs are also stored in the data source record. Treat the data source record as a secret.

Performance implications

• The health endpoint is a round trip. A dashboard with 10 data source references and a 5 s refresh is 120 health-check round trips per minute. A slow backend makes the dashboard slow.
• Grafana caches data source metadata in memory. Adding or removing a data source requires a Grafana reload; the cache does not pick up changes automatically.
• The data source proxy adds latency. proxy mode means every panel query goes through Grafana; the proxy adds one HTTP hop. direct mode skips the hop but exposes the backend to the browser.
• Multiple dashboards referencing the same data source share the proxy. The proxy pool size is bounded by Grafana’s [dataproxy] settings; a slow data source slows every dashboard that references it.

Production guidance

• Alert on data source health, not on panel rendering. A Grafana alert on datasource_health is the operational signal that something is wrong with the dashboard platform itself.
• Use provisioning to declare every data source. The provisioning manifest is the source of truth; UI edits drift.
• Use proxy access for production data sources. The browser does not see the backend’s URL.
• Rotate data source credentials on a calendar. A long-lived token that expires silently is the canonical credential failure.
• Probe data sources in staging, not just in production. The CI step that resolves every UID against staging catches the drift before the dashboard reaches production.
• Document the data source UID contract in the provisioning repo. The UID is a public reference key; a change to it is a breaking change for every dashboard that uses it.

Verification

You should now be able to answer:

• What are the four checks that define data source availability?
• Where do data source UID references live in a Grafana 11 dashboard JSON?
• What is the difference between health and reachability, and why does a data source need both?
• What is the failure shape of a UID renamed in provisioning?
• How does the CI step that resolves every UID against staging catch the drift before the dashboard reaches production?

Quiz