Datasources

A dashboard template assumes a Prometheus data source called “Prometheus”. The new cluster uses the UID “prom-prod”. The panel query up{instance=~"web.*"} returns “datasource not found”. The on-call engineer spends twenty minutes finding out that two UIDs were changed in different pull requests.

This lesson is about that misnaming. A data source is a tiny object in the storage layer (URL, type, UID, credentials, options) plus a plugin that knows how to talk to it. Getting the UID, the URL, and the credentials right is the entire job.

What a data source is

A data source is a Grafana-managed connection to a backend that can answer queries. Each data source has a type — a backend kind such as prometheus, loki, tempo, mysql, or postgres — and a plugin that knows that backend’s wire protocol and query language. The same panel can target any data source whose plugin s the panel’s expected data shape.

Grafana 11 ships the following plugins enabled by default in the official container image:

• Prometheus, Loki, Tempo, Mimir (Grafana Cloud-compatible)
• InfluxDB (v1 + v2), Elasticsearch, OpenSearch, MySQL, MSSQL, Postgres, ClickHouse
• TestData (useful for development; never use in production)
• Alertmanager (legacy; replaced by Grafana Alerting for new installations)
• Zipkin, Jaeger, Pyroscope (legacy tracing)

Enterprise distributions and the community plugin catalogue add hundreds more. The cost of catalogue breadth is that the plugin quality varies; keep your production list small.

Why a sysadmin cares

A misconfigured data source is the difference between a dashboard that answers a question and a dashboard that displays the red triangle. Three things go wrong in production:

1. Wrong URL. The data source points at a Prometheus that exists but is on the wrong port, or at a Prometheus that was rebuilt with a different DNS name.
2. Wrong UID. A developer renames the data source UID in one dashboard but not the others. The dependent panels fail with “datasource not found”.
3. Wrong credentials. The read token expired, the basic-auth user was rotated, or the secure-JSON field was overwritten with an empty value.

Each of these is fixable in minutes once the operator knows which field is wrong. The lesson is the field name and the lookup path.

How it works

The data-source plugin runs in-process inside the Grafana server. The query goes from the panel, to the plugin’s QueryData method, to the backend, and back as a data frame (a columnar, typed result). The data source’s proxy is what makes the Grafana server a useful intermediary: the browser calls Grafana, Grafana calls the backend with the correct credentials, and the browser never sees the secret.

+----------------+      query      +-----------+        HTTPS        +--------------+
| Browser panel  | ---------------> |  Grafana  | -------------------> |  Prometheus  |
| (no credentials) | <--------------- |  server   | <-------------------- |  Loki / Tempo |
+----------------+     data frame  +-----------+     data frame      +--------------+
                          ^              |    ^
                          |              |    |
                   plugin runtime  secureJsonData  |
                                                | basic auth /
                                                | bearer token
                                                v
                                         secrets manager /
                                         keyring file

Three properties of this diagram matter operationally:

• The browser never holds a data source credential. The proxy is the only thing that does. This is why the secureJsonData.password field exists and why passwords do not appear in /api/datasources responses.
• The UID is the stable handle. Dashboards reference the UID; renaming the name does not break the dashboard but renaming the UID does. UIDs are alphanumeric, lowercase, and unchanged by the UI’s “rename” action on the data source.
• The HTTP routing is by name in the URL. The GET /api/datasources proxy endpoint is at /api/datasources/<uid>/proxy/<path>. Clients normally should not need this; dashboard panels route through the plugin, not the proxy.

How to configure it

File-based provisioning is the only configuration that survives a container restart and the only one that belongs in version control. The directory /etc/grafana/provisioning/datasources/ holds one YAML per data-source kind.

# /etc/grafana/provisioning/datasources/metrics.yaml
apiVersion: 1
datasources:
  - name: Prometheus
    uid:  prom-prod
    type:  prometheus
    access: proxy
    orgId: 1
    url:    http://prometheus.monitoring.svc:9090
    isDefault: true
    version: 1
    editable: false
    jsonData:
      httpMethod: POST
      timeInterval: 30s
      queryTimeout: 60s
      manageAlerts: true
      prometheusType: Prometheus
      prometheusVersion: 2.55.0
    secureJsonData:
      basicAuthPassword: ${PROM_READ_TOKEN}

# /etc/grafana/provisioning/datasources/logs.yaml
apiVersion: 1
datasources:
  - name: Loki
    uid:  loki-prod
    type:  loki
    access: proxy
    url:    http://loki.monitoring.svc:3100
    editable: false
    jsonData:
      maxLines: 1000
      timeout: 60
      httpMethod: GET
    secureJsonData:
      basicAuthPassword: ${LOKI_READ_TOKEN}

# /etc/grafana/provisioning/datasources/traces.yaml
apiVersion: 1
datasources:
  - name: Tempo
    uid:  tempo-prod
    type:  tempo
    access: proxy
    url:    http://tempo.monitoring.svc:3200
    editable: false
    jsonData:
      httpMethod: GET
      tracesToLogsV2:
        datasourceUid: loki-prod
        tags: [job, instance]
        spanStartTimeShift: -1s
        spanEndTimeShift:   1s
      serviceMap:
        datasourceUid: prom-prod
      nodeGraph:
        enabled: true
    secureJsonData: {}

editable: false is the configuration call an operator cares about. It prevents a developer from saving a one-off data source through the UI that then quietly overrides the file-based one on the next provisioning poll.

Environment variables in secureJsonData are expanded by Grafana at load time; they are not stored as plain text. The syntax ${VAR} reads from the process environment. The expression ${VAR:-default} is supported and is the form used for non-secret defaults in jsonData.

How to validate it

Three checks confirm a data source is up from the operator’s view.

Severity: READ-ONLY.

# 1. List every data source Grafana knows about.
curl -sf -u "grafana-admin:$GF_ADMIN_PASSWORD" \
  http://grafana:3000/api/datasources | jq '.[] | {uid, type, url}'

{
  "uid":  "prom-prod",
  "type": "prometheus",
  "url":  "http://prometheus.monitoring.svc:9090"
}
{
  "uid":  "loki-prod",
  "type": "loki",
  "url":  "http://loki.monitoring.svc:3100"
}
{
  "uid":  "tempo-prod",
  "type": "tempo",
  "url":  "http://tempo.monitoring.svc:3200"
}

# 2. Run the data source health check.
curl -sf -u "grafana-admin:$GF_ADMIN_PASSWORD" \
  http://grafana:3000/api/datasources/uid/prom-prod/health | jq

{
  "message": "Data source is working",
  "status":   "success"
}

# 3. From a Grafana host, confirm the backend is reachable on
#    its own protocol. For Prometheus:
curl -sf -o /dev/null -w "%{http_code}\n" \
  http://prometheus.monitoring.svc:9090/-/ready
# 200

A 200 from /api/datasources/uid/<uid>/health proves Grafana can talk to the backend. A 200 from the backend’s own health endpoint proves the backend is healthy. The two together prove the data path the panel will take is open.

How it can fail

Five high-frequency failure shapes:

1. UID collision. Two data sources share the same UID — the second is rejected at provisioning time. Symptom: log line data source with uid <x> exists and the missing source in /api/datasources.
2. Wrong URL. url points at a host or port that exists but is the wrong service — for example, a Loki URL ending in :9090 (Prometheus). Symptom: /health returns an unparseable response, panels display “Unexpected error”, /api/datasources/uid/<uid>/health returns "status": "error" with a 200 response.
3. Missing credentials. The backend requires basic auth and the password is unset or expired. Symptom: health returns "status": "error" with HTTP 401 propagated from the backend. The Grafana log includes 401 Unauthorized from the plugin.
4. CORS or direct-access mismatch. The panel uses a backend that the browser should reach directly, but the backend has no CORS headers. Symptom: the browser’s network tab shows a blocked request; Grafana’s /health is fine. The fix is to set access: proxy rather than direct.
5. secureJsonData wiped by env var expansion. The ${PROM_READ_TOKEN} is unset in the container environment, so Grafana interprets it as the empty string and stores an empty password. Symptom: /health returns "status": "error"; logs/grafana/grafana.log does not log the secret (correctly) but does log Using empty value for $\{...\} at startup (after the relevant config reload).
6. Plugin removed after upgrade. A plugin was uninstalled or is incompatible with the new Grafana version. Symptom: datasources listing skips the entry; the provisioned source is dropped on every reload.

How to troubleshoot it

The order works because each step eliminates a layer.

1. Does Grafana see the source? /api/datasources. Missing means provisioning failed; check the Grafana log for YAML errors.
2. Does the plugin pass? /api/datasources/uid/<uid>/health. status: error here isolates the failure to the network or credentials, not to the JSON configuration.
3. Is the backend reachable? curl http://.../-/ready (or /ready for Loki, /status/version for Tempo). 5xx or connection refused means the data source is fine; the backend is down.
4. Are the credentials correct? Tail Grafana’s logs for the 401 / 403 line. Verify with a manual curl -u user:pass https://....
5. Is the plugin version compatible with Grafana? grafana cli plugins ls and check version. Plugin manifest requires the runtime version number to be in its dependencies.grafanaDependency range.
6. Is the datasource proxy route blocked? From the Grafana process, tcp <host> <port> to confirm the kernel’s view of the destination. Containerised Grafana without sidecar DNS is the most common cause of “connection refused” on a *.svc URL.

Security implications

• The browser never holds the secret. Confirm by viewing any panel in the browser’s developer tools; the outbound network requests go to Grafana, not the backend. Setting access: direct breaks this property and exposes the secret to anyone who can read the dashboard URL.
• secureJsonData is encrypted at rest with the server’s secret_key. The encryption is reversible by the server, so it is not a credential vault — it is a secret-on-disk barrier. Treat it as “the secret is on the server” and not as “the secret is safe”.
• Read-only tokens. Every backend in this stack supports a read-only credential. Issue one per Grafana; never reuse an admin token. The Prometheus prom-prod data source should hold a token with read permission only; alerting writes go through Prometheus’s own /api/v1/admin/* API with a separate credential.
• mTLS / TLS termination. If the backend is on https://... and the certificate is internal (acquired from a private CA), jsonData.tlsSkipVerify: true is a temptation; the right control is to install the CA into the Grafana container’s trust store.

Performance implications

• A panel makes one request per data source per refresh. The timeInterval field caps the rate at which Grafana may ask the backend. The default (1s for Prometheus) suits most dashboards; tight intervals (200ms) amplify load and should be reserved for short auto-refresh bursts.
• Loki queries are streamed; the maxLines cap prevents a panel from requesting hundreds of thousands of lines at once. Set it to what a human can read (100 to 1000).
• Tempo’s tracesToLogs and serviceMap cause cross-data-source fetches. The first log search inside a trace view makes a Loki query; the first service-map render makes a Prometheus query. These are unbounded in production graphs; govern with managedAlert/managedDashboard policies, not by patching every panel.
• ${PROM_READ_TOKEN} reads from the environment on every poll. A slow secret-manager read on each poll is a known pattern; cache the value into the environment once on container start.

Production guidance

• Provision, do not UI-edit. editable: false belongs on every production data source.
• Pin a UID once. Add a CI check that fails when a UID changes.
• Secure-JSON fields reference a secrets manager via env vars. Commit a .env.example, not the secrets.
• One read token per backend. Issue a new one when an operator leaves the team; rotate on a fixed cadence (90 days is common).
• Keep the plugin set small. Each plugin is an attack surface and a version-pinning liability.
• Treat “no data” on a panel as a data-source-health event first and a panel-config event second.

Verification

You should now be able to answer:

• What is the UID for, and why is it different from the name?
• What does secureJsonData buy you that jsonData does not?
• Why is access: proxy the secure default for internet-exposed Grafana?
• Which two endpoints together prove a panel can render?
• How does Grafana stop you from racing the provisioning file with a UI edit?

Quiz