Data Source Credentials

A Grafana provisioning YAML file lands in a public Git repository. The file contains the Loki data source with secureJsonData: { basicAuthPassword: hunter2 } written in cleartext. A scanner finds the repository three hours later. The scanner does not need to authenticate to Loki; the password is in the commit history. The next morning the Loki ingest path shows a 4x spike from IPs in three different countries, and the on-call engineer is reading the audit log looking for the breach vector that does not exist inside Grafana.

This lesson is about closing that gap. Data source credentials are the secrets that turn a Grafana from a read-only dashboard into a read-write attack surface. The shape they take on disk, in the Grafana database, and on the wire decides who can read them.

What it is

Data source credentials in Grafana are the per-data-source secrets Grafana uses to authenticate against upstream backends (Prometheus, Loki, Tempo, Mimir, Elasticsearch, MySQL, Postgres, and dozens more). Three families of credentials matter in production:

1. Basic auth — a username and password pair, sent to the upstream on every query. The Grafana fields are basicAuthUser and basicAuthPassword.
2. TLS client certificate — a client certificate, key, and CA bundle, presented to the upstream to authenticate the Grafana process. The Grafana fields are tlsClientCert, tlsClientKey, and tlsCACert.
3. Custom HTTP headers — bearer tokens and arbitrary Authorization headers, used for OIDC-protected upstreams or for custom auth schemes. The Grafana fields are httpHeaderName1 / httpHeaderValue1 and similar.

All three live in two distinct fields in the data-source configuration:

• jsonData — non-sensitive configuration. Plain text. Visible in the HTTP API.
• secureJsonData — sensitive configuration. Encrypted at rest. Never returned by the HTTP API.

   Grafana                            Loki / Prometheus / Tempo
   -------                            ------------------------
      |                                       |
      |--GET /api/datasources/proxy/1/...-->|
      |                                       |
      |<-- 200 OK (response body)------------|
      |                                       ^
      |  (Grafana adds the credentials)-------|

   No credentials ever cross to the browser when the access mode
   is `proxy`. With `direct`, the browser holds the credentials.

The single most important property of the data-source credential model: with access = proxy (the default), Grafana proxies the query and the credentials never leave the Grafana process. With access = direct, the credentials are sent to the browser so it can query the upstream directly. Production Grafana uses proxy for everything except the cases where browser-side rendering is the explicit goal.

Why a sysadmin cares

Data source credentials are the keys to the upstream telemetry store. A Grafana with Editor or Admin can read these credentials via the API, and a Grafana with anonymous Viewer can issue queries that the upstream serves. A credential leak from Grafana is a credential leak from the upstream.

The four production failure shapes:

• The cleartext YAML. A data source is provisioned through conf/provisioning/datasources/ with secureJsonData written in plain text. The provisioning file is committed to Git. The secret is now in the commit history. The fix is environment-variable interpolation.
• The browser-side credential. A data source is configured with access = direct because the operator wanted low-latency queries. The credentials are now in the browser, in the page source, in the network tab, in any browser extension that reads localStorage.
• The shared secret with no rotation. A Loki data source is provisioned with a long-lived basicAuthPassword that nobody has rotated since install. The secret is in the YAML, in the database, and in any backup of either. There is no expiry.
• The credential that grants too much. A Grafana data source credential for Loki or Mimir is a tenant-wide credential. The Grafana user with Editor can issue any query the credential permits, including queries that read other tenants’ streams if the upstream is not configured to scope them.

How it works

Every data source is a row in the data_source table. The row has two JSON columns: json_data (plaintext, returned by the API) and secure_json_data (encrypted, never returned). The encryption uses the [security] secret_key and an authenticated encryption scheme; the encrypted value never leaves the database.

   Provisioning YAML        Grafana database            HTTP API
   -----------------        ----------------            --------
   jsonData:                data_source.json_data:      GET /api/datasources/...
     basicAuth: true          basicAuth: true             returns json_data,
     ...                      ...                          never secure_json_data
   secureJsonData:          data_source.secure_json_data:
     basicAuthPassword:       <encrypted>                 never returned
     "hunter2"

When a dashboard renders a panel, Grafana receives the query from the browser, appends the credentials from secure_json_data, and forwards the query to the upstream through /api/datasources/proxy/<id>/.... The response is returned to the browser without the credentials.

   Browser                Grafana                 Loki / Prometheus
   -------                -------                 ------------------
      |                      |                            |
      |--POST /api/ds/query->|                            |
      |   {"query":...}      |--with credentials-------->|
      |                      |                            |
      |                      |<--200 OK (data)-----------|
      |<--200 OK (data)------|                            |
      |                      |                            |
      # Credentials stay inside Grafana; the browser sees
      # only the response body.

For access = direct, the credentials are serialised into the panel’s data structure and sent to the browser, which then queries the upstream directly. The browser holds the credentials; browser extensions can read them; the network tab reveals them to anyone with browser dev tools.

How to configure it

Basic auth with env-var interpolation

# /etc/grafana/provisioning/datasources/loki.yaml
apiVersion: 1
datasources:
  - name: Loki
    type: loki
    uid: loki-prod
    orgId: 1
    url: https://loki.internal.example.com
    access: proxy
    isDefault: false
    jsonData:
      httpMethod: POST
      tlsAuth: false
      tlsAuthWithCACert: false
      timeout: 60
      # Grafana 11.x resolves ${VAR} from the environment at provisioning time.
      # The value never lands in the YAML on disk.
      basicAuth: true
    secureJsonData:
      basicAuthUser: ${LOKI_USERNAME}
      basicAuthPassword: ${LOKI_PASSWORD}

TLS client certificate

# /etc/grafana/provisioning/datasources/mimir.yaml
apiVersion: 1
datasources:
  - name: Mimir
    type: prometheus
    uid: mimir-prod
    orgId: 1
    url: https://mimir.internal.example.com/prometheus
    access: proxy
    jsonData:
      tlsAuth: true
      tlsAuthWithCACert: true
      tlsServerName: mimir.internal.example.com
    secureJsonData:
      tlsClientCert: ${MIMIR_CLIENT_CERT}
      tlsClientKey:  ${MIMIR_CLIENT_KEY}
      tlsCACert:      ${MIMIR_CA_BUNDLE}

Bearer token (custom header)

# /etc/grafana/provisioning/datasources/cortex.yaml
apiVersion: 1
datasources:
  - name: Cortex
    type: prometheus
    uid: cortex-prod
    orgId: 1
    url: https://cortex.internal.example.com
    access: proxy
    jsonData:
      httpHeaderName1: Authorization
    secureJsonData:
      httpHeaderValue1: ${CORTEX_BEARER_TOKEN}

Provisioning the secret through the API (Admin only)

# CONFIGURATION: write a data source credential through the HTTP API.
# The secureJsonData is encrypted at rest; only the Admin who wrote it
# can read it again through the provisioning shape.
curl -fsS -X POST https://grafana.example.com/api/datasources \
  -H "Authorization: Bearer ${GF_ADMIN_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Loki",
    "type": "loki",
    "url": "https://loki.internal.example.com",
    "access": "proxy",
    "basicAuth": true,
    "basicAuthUser": "grafana",
    "secureJsonData": {
      "basicAuthPassword": "'"${LOKI_PASSWORD}"'"
    }
  }'

How to validate it

# READ-ONLY: confirm a data source is configured with access = proxy.
curl -fsS -H "Authorization: Bearer ${GF_SA_TOKEN}" \
  https://grafana.example.com/api/datasources/uid/loki-prod | jq '.access'
# "proxy"

# READ-ONLY: confirm secureJsonData is not returned.
curl -fsS -H "Authorization: Bearer ${GF_SA_TOKEN}" \
  https://grafana.example.com/api/datasources/uid/loki-prod | jq '.secureJsonData'
# null                 # good; the field is hidden
# {"basicAuthPassword":"hunter2"}    # bad; the secret is leaking

# READ-ONLY: a query through the proxy succeeds with the credentials.
curl -fsS -X POST -H "Authorization: Bearer ${GF_SA_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"queries":[{"refId":"A","datasource":{"uid":"loki-prod"},"expr":"{job=\"varnish\"}"}]}' \
  https://grafana.example.com/api/ds/query
# {"results":{"A":{"frames":[...]}}}

# READ-ONLY: a query directly through the browser without auth fails.
curl -fsS -o /dev/null -w "%{http_code}\n" \
  https://loki.internal.example.com/loki/api/v1/query?query='{job="varnish"}'
# 401            # the upstream requires credentials; good

# READ-ONLY: the same query through Grafana returns 200 with data.
curl -fsS -o /dev/null -w "%{http_code}\n" \
  -H "Authorization: Bearer ${GF_SA_TOKEN}" \
  -X POST -H "Content-Type: application/json" \
  -d '{"queries":[{"refId":"A","datasource":{"uid":"loki-prod"},"expr":"{job=\"varnish\"}"}]}' \
  https://grafana.example.com/api/ds/query
# 200            # good; Grafana held the credential, the proxy added it

# READ-ONLY: confirm the TLS client cert is loaded.
curl -fsS -H "Authorization: Bearer ${GF_SA_TOKEN}" \
  https://grafana.example.com/api/datasources/uid/mimir-prod/health
# {"message":"data source is working","status":"success"}

# READ-ONLY: confirm the data source ACL.
curl -fsS -H "Authorization: Bearer ${GF_ADMIN_TOKEN}" \
  https://grafana.example.com/api/datasources/uid/loki-prod | jq '.permissions'
# [{"id":1,"role":"Viewer","userId":0},{"id":2,"role":"Editor","teamId":1}]

A clean validation: the API returns access = proxy, secureJsonData is null in the response, a query through the proxy returns data, and a direct curl to the upstream without credentials returns 401.

How it can fail

The high-frequency data-source credential failure modes from real Grafana installs.

1. access = direct for a Loki data source. The browser holds the bearer token or the basic-auth credential. A browser extension that reads localStorage finds it. A developer with browser dev tools sees it in the network tab. The fix is access = proxy.
2. Cleartext password in committed YAML. The provisioning file was committed with basicAuthPassword: hunter2. The scanner finds the commit; the password is now in the commit history of every fork. The fix is ${LOKI_PASSWORD} interpolation and a secret rotation, then a history rewrite or a new repo.
3. tlsAuth: true with no tlsClientCert provided. The data source is configured to use TLS client auth but the cert is missing from secureJsonData. The symptom is every query returning tls: failed to find any PEM data in certificate input.
4. Expired upstream client certificate. The certificate was valid for 90 days and was issued once at install. The 91st day produces tls: failed to verify client certificate on every query. The symptom is the dashboard going red across the org.
5. tlsCACert not updated when the upstream CA rotates. The client cert is valid but the CA bundle in Grafana does not include the new issuer. The symptom is x509: certificate signed by unknown authority on every query.
6. Credential scoping too wide. The Grafana Loki credential is a Cortex tenant admin, not a tenant reader. A Grafana Editor can issue admin queries (delete series, change retention) through the proxy. The fix is a read-only credential scoped to the specific tenant.

How to troubleshoot it

The diagnostic order matters: a credential failure can be at the upstream, at Grafana, or in between.

1. Probe the upstream directly. curl -fsS https://upstream.example.com/api/v1/query?query=up with the expected credentials confirms the upstream is healthy and the credential works.
2. Inspect the data source health endpoint. GET /api/datasources/<id>/health runs the proxy health check. A 200 means the credential is being applied; a 401/403 means it is not.
3. Read the Grafana log at debug. log.level = debug produces a line for every data-source query, including the TLS handshake state and the credential match.
4. For TLS client certs: confirm the cert, key, and CA bundle are valid with openssl x509 -in client.crt -noout -dates and openssl verify -CAfile ca.crt client.crt.
5. For access = direct: the browser network tab is the diagnostic tool. Look for the request to the upstream URL; the credential appears in the request headers.
6. For env-var interpolation: GRAFANA_LOG_LEVEL=debug shows the provisioning path. A missing env var produces failed to load data source: basicAuthPassword: ${LOKI_PASSWORD}: environment variable not found.

Security implications

• secureJsonData is the only safe shape for secrets. Writing a secret to jsonData puts it in every API response, every backup of the database, every log line that mentions the data source.
• access = direct puts credentials in the browser. A Grafana that legitimately needs access = direct (rare; mostly for browser-side caching or for visualisation-only upstreams) accepts that the credential is exposed to anyone who can reach the dashboard URL.
• Provisioning YAML is the most-leaked shape. A Grafana with 200 data sources has 200 YAML files; each is a potential cleartext credential. The fix is ${VAR} interpolation and a secrets manager.
• The data-source credential is upstream-scoped. A Grafana Editor can issue any query the credential permits. The fix is to scope the credential at the upstream: a read-only Loki tenant, a read-only Mimir tenant, a read-only Elasticsearch user.

Performance implications

• access = proxy adds one Grafana hop per query. The browser sends the query to Grafana, Grafana sends it to the upstream, Grafana returns the response. The cost is one extra network round-trip and the Grafana CPU to format the request. For most upstreams, the cost is negligible.
• access = direct removes the Grafana hop but adds browser CPU. The browser formats the upstream request, sends it, and renders the response. The performance benefit is real but the security cost is usually higher.
• TLS client auth adds handshake cost. Every query through tlsAuth: true does a full TLS handshake with client cert presentation. With connection reuse (keepalive) on the upstream, the cost is amortised.

Production guidance

• Every data source uses access = proxy.
• Every credential lives in secureJsonData only, sourced from the secrets manager via ${VAR} interpolation.
• The data-source credential at the upstream is read-only and tenant-scoped.
• A rotation cadence (90 days is a common starting point) for every credential, with a documented runbook.
• A list of every data source and its credential type, kept in a security baseline document; the on-call engineer can answer “what does this data source use for auth?” without grepping YAML.
• A scanner (gitleaks, trufflehog, or equivalent) in the CI pipeline that fails the build on cleartext credentials in provisioning YAML.

Verification

You should now be able to answer:

• What is the difference between jsonData and secureJsonData in a Grafana data source, and which one is safe to write secrets into?
• Why does access = proxy keep credentials out of the browser while access = direct does not, and when is the trade-off worth it?
• Why is ${LOKI_PASSWORD} interpolation in a provisioning YAML safer than writing the cleartext password, and where does the value actually come from at provisioning time?
• What is the failure shape of an expired upstream TLS client certificate, and how do you distinguish it from an expired CA bundle?
• Why should the Grafana data-source credential at the upstream be scoped read-only and tenant-limited, and what is the blast radius of a Grafana Editor with an admin-scoped credential?

Quiz