Prometheus as a Data Source

A panel goes red at 03:00. The dashboard says “No data.” The on-call engineer opens the Explore view in Grafana, types up into the Prometheus query editor, and gets a yellow banner:

Network Error: 502 Bad Gateway
datasource: prom-prod-eu, uid: prom-prod-eu

The datasource is configured. The query is the simplest query in PromQL. The data source health check — the very check Grafana runs before it sends a panel query — already says the connection is unreachable. The lesson this time is about that connection: the YAML that declares it, the data-source proxy that mediates it, the health check that audits it, and the timeout interaction that makes a “working” Prometheus silently lie about whether it is working.

What it is

A Prometheus data source in Grafana is a named, configured client of the Prometheus HTTP API. The provisioning file declares the data source’s URL, type, credentials, TLS posture, and a small set of behavioural options. Once provisioned, Grafana runs every panel query through a server-side proxy at /api/datasources/proxy/uid/<uid>/..., never directly from the browser. The proxy is the only component that holds the credentials.

In Grafana 11.x the canonical Prometheus data source type is prometheus. The legacy alias Prometheus (capitalised) still appears in older dashboards and is resolved identically at query time; new provisioning should use the lowercase form. The url value points at the Prometheus /api/v1 endpoint, not at the metrics root.

Why a sysadmin cares

The Prometheus data source is the most-used data source in any metrics stack and the one whose silent failures cost the most time. Three production shapes appear repeatedly:

• The credential rotates upstream and Grafana does not know. Prometheus’ basic-auth password was changed in the secret manager. Grafana still holds the old secureJsonData.password. Every panel renders 401 Unauthorized and the dashboards are treated as “broken” by the on-call. The fix is not a restart; it is a redeploy with a new secureJsonData value.
• The TLS posture changes. A Prometheus instance behind a sidecar gets a new certificate, or its CA bundle rotates. Grafana’s tlsConfig still pins the old CA. Panels either fail the handshake silently (Grafana reports “no data”) or surface an x509: certificate signed by unknown authority error in the data-source health check.
• Scrape interval drift. The Grafana query timeout defaults to 60 seconds. The Prometheus scrape interval is 15 seconds. A panel that asks for a six-hour window with a 1-second step sends a query that Prometheus cannot answer inside 60 seconds. Grafana reports context deadline exceeded and the on-call concludes Prometheus is “broken” — when Prometheus is, in fact, working correctly and the panel is the wrong shape.

How it works: the request path

   browser        grafana-server           prometheus
   -------        --------------           ----------
      |                |                        |
      |--Panel query-->|                        |
      |                |--GET /api/v1/query_range|
      |                |  (BasicAuth, TLS, ...) |
      |                |                        |
      |                |<--200 OK + samples-----|
      |<--Panel data---|                        |

Three observations:

1. The browser never sees the credentials. Every panel query from the browser is rewritten by Grafana into a server-side call to /api/datasources/proxy/uid/<uid>/api/v1/query_range. The credentials live in secureJsonData on the server.
2. The data source is identified by UID, not by name. A panel that says “Prometheus” by name silently breaks the moment the data source is renamed; a panel that says prom-prod-eu by UID keeps working through renames.
3. The HTTP path through the proxy mirrors the upstream path. /api/v1/query, /api/v1/query_range, /api/v1/series, /api/v1/labels, /api/v1/metadata — Grafana proxies them all. A query against an unknown path returns 404 from the proxy and the same 404 from Prometheus; the round trip is not free.

How to configure it

# /etc/grafana/provisioning/datasources/prometheus.yml
apiVersion: 1

datasources:
  - name: prom-prod-eu
    uid: prom-prod-eu           # stable; never reuse across instances
    type: prometheus
    access: proxy               # Grafana proxies; the browser never sees creds
    orgId: 1
    url: https://prom-prod-eu.internal:9090
    isDefault: true
    editable: false             # prevent click-ops from drifting from Git

    # Basic auth. Username in jsonData, password in secureJsonData.
    basicAuth: true
    basicAuthUser: grafana-reader
    jsonData:
      tlsAuth: false
      tlsAuthWithCACert: true
      tlsSkipVerify: false
      # Query timeout: 60s matches the Grafana default.
      # Set below the Prometheus --query.timeout (default 2m).
      timeInterval: 15s          # the step Grafana assumes for "last X" queries
      httpMethod: POST           # large query_range bodies exceed GET limits
    secureJsonData:
      tlsCACert: |
        -----BEGIN CERTIFICATE-----
        MIIDazCCAlOgAwIBAgIUJx...
        -----END CERTIFICATE-----
      basicAuthPassword: ${PROM_PASSWORD}    # injected by the provisioner

A few production notes on the options:

• access: proxy is the right default. The direct mode sends credentials to the browser; only use it for unauthenticated, loopback-only Prometheis.
• uid is the stable identifier. Use a name that describes the environment and region, not the URL.
• isDefault makes this the implicit target of new panels. Setting it on more than one Prometheus per Grafana instance produces surprising behaviour when an Explore query has no explicit target.
• timeInterval is the step Grafana uses when the user picks “Last 6 hours” without specifying a step. Setting it lower than the Prometheus scrape interval produces “duplicate sample timestamp” warnings.
• httpMethod: POST is required when the query or the label selector exceeds a few hundred characters; the default GET caps at 8 KB on most proxies.
• tlsAuthWithCACert plus the CA cert in secureJsonData is the right posture for a Prometheus with a private CA. Skipping verification (tlsSkipVerify: true) is acceptable only for development.

How to validate it

# READ-ONLY: the data source is provisioned and reachable.
curl -fsS -u grafana-admin:$GRAFANA_ADMIN \
  http://grafana.internal:3000/api/datasources/uid/prom-prod-eu
# {
#   "id": 1,
#   "uid": "prom-prod-eu",
#   "name": "prom-prod-eu",
#   "type": "prometheus",
#   "url": "https://prom-prod-eu.internal:9090",
#   "access": "proxy",
#   "isDefault": true,
#   ...
# }

# READ-ONLY: the health check returns one of four states.
curl -fsS -u grafana-admin:$GRAFANA_ADMIN \
  http://grafana.internal:3000/api/datasources/uid/prom-prod-eu/health
# {"message":"Data source is working","status":"success"}

# READ-ONLY: a minimal query against the proxy.
curl -fsS -u grafana-admin:$GRAFANA_ADMIN \
  --data-urlencode 'query=up' \
  --data-urlencode 'time=2026-08-13T10:00:00Z' \
  http://grafana.internal:3000/api/datasources/proxy/uid/prom-prod-eu/api/v1/query
# {"status":"success","data":{"resultType":"vector","result":[...]}}

# READ-ONLY: Prometheus-side query log to confirm the request landed.
ssh prom-host 'tail -n 5 /var/log/prometheus/query.log'
# ts=2026-08-13T10:00:00 caller=queryable.go:... component=tsdb
#   method=query endpoint=/api/v1/query ...

# CONFIGURATION: reload Grafana provisioning.
sudo systemctl reload grafana-server

A clean validation: the uid is present, the health check returns "status": "success", the proxy returns Prometheus data, and the Prometheus query log shows the Grafana query. Anything less is a symptom; each failure mode below maps to one of these four signals failing.

How it can fail

The most expensive Prometheus data source failure modes from real production incidents.

1. Stale basicAuthPassword. The credential rotated in the secret store; Grafana still holds the previous value. The health check returns 401 Unauthorized, every panel shows Network Error, and the on-call concludes Prometheus is down. The symptom is status: 401 in /api/datasources/uid/<uid>/health.
2. CA bundle drift. The private CA that signs Prometheus’ certificate rotated. Grafana’s tlsCACert is stale. The health check returns the same success shell with a x509: certificate signed by unknown authority error in the underlying message. The symptom is the health-check message field containing the cert error string.
3. Proxy URL points at the metrics root, not at /api/v1. The url value is https://prom.internal:9090 rather than the explicit API endpoint. The health check sometimes succeeds (Prometheus returns its own /-rooted page) and every panel fails with parse error. The symptom is panels returning parse error: unexpected EOF while /api/datasources/.../health is green.
4. Scrape interval / step mismatch. Prometheus scrapes every 15 seconds. Grafana’s timeInterval is 5 seconds. The query engine asks for 1-second steps and Prometheus returns duplicate timestamps. The symptom is error: duplicate sample timestamp warnings at every panel load.
5. HTTP method left at GET. A panel with a 12 KB label selector (large cardinality, deep service-name regex) hits the GET URL-length limit and is rejected by the proxy before it ever reaches Prometheus. The symptom is intermittent panel failures correlated with selector length.
6. DNS split-horizon. The data source URL is prom.internal but the Grafana host resolves it via a different DNS path than the application hosts. Health checks pass from Grafana’s perspective but the upstream Prometheus reports the query against a different replica. The symptom is “data is correct but timestamps lag the rest of the platform by 30 seconds”.

How to troubleshoot it

The diagnostic order is “is the data source provisioned?”, “is the proxy reachable?”, “is the upstream responding?”, “is the credential right?”.

1. Confirm the data source exists. GET /api/datasources/uid/<uid>. If 404, the provisioning file did not reload; check grafana.log for parse errors.
2. Confirm the health check. GET /api/datasources/uid/<uid>/health. The four states are success, error, notfound, and the rarely-seen forbidden. Each maps to a different upstream condition.
3. Reproduce the request through the proxy. curl against /api/datasources/proxy/uid/<uid>/api/v1/query?query=up. This is the exact request Grafana issues for the simplest panel. If this fails, the failure is in the proxy chain.
4. Reproduce the request directly. curl against the upstream Prometheus URL with the same credentials Grafana uses. If this succeeds and the proxy call fails, the failure is in Grafana’s secureJsonData or tlsConfig.
5. Inspect Grafana’s logs. /var/log/grafana/grafana.log records every proxy call and its outcome, with the UID and the upstream status. A 401 from Prometheus appears as a single line with status=401; a TLS error appears with the x509 string.
6. Cross-check the credential in the secret store. A vault read or equivalent that confirms the value Grafana is using matches the value Prometheus expects.

Security implications

• The proxy holds the credentials. A Grafana with a misconfigured access: direct would expose the password to every browser. The fix is access: proxy and a server-side network ACL that limits /api/datasources/.../secure/* to service-account tokens.
• The CA cert is a credential too. A Grafana that pins the wrong CA cannot distinguish a legitimate Prometheus from a network attacker with a self-signed cert. Verify the chain out-of-band at every rotation.
• The secureJsonData payload lives in memory. A Grafana process dump or a pprof heap profile can leak the password. Disable the debug endpoints on production Grafana ([security] disable_gravatar = true is unrelated, but [users] allow_sign_up = false and the [analytics]-related public endpoints should be off).
• The health check leaks the upstream status. A 401 from Prometheus is visible to any Grafana admin. Treat the /api/datasources/uid/<uid>/health endpoint as authenticated-and-internal-only.

Performance implications

• Query latency is on the proxy. The path is browser → Grafana server → Prometheus. The Grafana server hop is non-negotiable for credentialed sources. Place Grafana and Prometheus in the same availability zone to keep the additional hop under 1 ms.
• Query timeout is on Grafana. Default 60 seconds. Set it below the Prometheus --query.timeout so Grafana fails fast on slow queries and Prometheus does not accumulate abandoned work.
• httpMethod: POST removes the GET URL-length cap. A panel with a 12 KB selector is rejected by GET; POST is required at scale.
• scrape_interval and timeInterval should match. A Prometheus with a 15-second scrape and a Grafana with a 5-second timeInterval produces duplicate sample timestamp warnings at every panel render.

Production guidance

• Pin the uid. Renaming a data source is fine; changing the UID breaks every panel that references it.
• Use access: proxy for every Prometheus with credentials.
• Align timeInterval to the scrape interval. Document both in the team’s instrumentation guide.
• Reload Grafana provisioning through the GitOps pipeline, not by click-ops in the UI.
• Verify the health check from the alerting path. A Grafana that flips a Prometheus to error should page the on-call before a panel renders red.
• Test credential rotation in staging. The cost of a “broken” production Grafana at the moment of a planned credential rotation is a real and recurring outage shape.

Verification

You should now be able to answer:

• What is the operational difference between access: proxy and access: direct for a Prometheus data source in Grafana?
• What does the GET /api/datasources/uid/<uid>/health endpoint actually verify, and what does it not verify?
• How does Grafana’s query timeout interact with Prometheus’ --query.timeout, and which one should be smaller?
• Why does a “Data source is working” health check sometimes coexist with broken panels?

Quiz