Datasource Credentials

A new starter joins the platform team and runs grafana-cli datasource ls in the dev Grafana to enumerate what is available. The output includes the production Prometheus URL with the production basic-auth password in cleartext, because a teammate copied the data source configuration from the prod Grafana UI into the dev Grafana six months earlier and the secure field was exported as part of the JSON. The new starter copies the password into a Notion page labelled grafana-credentials because not having to look it up is worth the convenience. Three months later, an audit reveals the Notion page is readable by every contractor in the company. The secret has been on three systems and two wikis. None of them are designed to hold it.

This lesson is about closing that gap. Grafana data source credentials have one home, one transport, and one audit surface. The production failure shape is the credential existing in more than one of those.

What it is

A Grafana data source is a record that tells Grafana how to talk to a backend (Prometheus, Loki, Tempo, Elasticsearch, a Postgres database, an HTTP API). The record has two halves:

• jsonData — non-sensitive configuration. URLs, timeouts, custom HTTP headers, query interval, derived field mappings. Stored as JSON in the data_source table; readable through the Grafana API; safe to export to a dashboard JSON.
• secureJsonData — sensitive configuration. Basic-auth passwords, bearer tokens, TLS client keys, private CA certificates. Stored as JSON in the data_source table too, but encrypted at rest with a key derived from [security] secret_key. Never returned by the Grafana API. Never exported in dashboard JSON.

The two halves travel together; the secure half is never visible to a reader of the non-secure half. A Grafana admin who reads the data source by uid sees the jsonData but the secureJsonData fields are masked with ***. A reader of the provisioning YAML who copies jsonData into Git commits nothing sensitive; secureJsonData sourced from a secrets manager commits nothing at all.

   Browser ----> Grafana (sees jsonData only)
                    |
                    |  access = proxy
                    v
                Backend (Prometheus / Loki / Tempo)
                (receives credential via Grafana)

The second property is access = proxy. With access = proxy, Grafana makes the request to the backend on the browser behalf and returns only the response. The browser never receives the credential. With access = direct, Grafana returns the credential to the browser and the browser makes the request to the backend directly. Production uses proxy; direct is a debugging escape hatch only.

Why a sysadmin cares

The data source credential decides four things the operator is going to be asked about.

• What can be read. A Grafana with the production Prometheus read-token can read every metric that token can see. A Grafana with the production Loki write-token can write to Loki. The credential is the boundary, not the data source record.
• Who can read it. Server Admin and Org Admin can both update a data source. Only Org Admin can see the secureJsonData on read. The Editor and Viewer roles cannot reach the data source admin API at all.
• Where it is stored. The data_source.secure_json_data column is encrypted at rest with a key derived from secret_key. A snapshot of the database without secret_key is useless for the credentials.
• How it is rotated. A change to secureJsonData via the UI or the API takes effect on the next request Grafana makes to the backend. A change via provisioning YAML takes effect on the next provisioning sync.

The most expensive credential failures in real Grafana installs are all in this list. A committed YAML, a rotated token that Grafana does not know about, a data source on access = direct that hands the browser a credential it never should have — every one of these is a security incident waiting on a curious user.

How it works: the credential path

The credential path is four steps. Each one is a place where the credential can leak.

1. Operator sets secureJsonData in provisioning YAML or via the API
   |
   v
2. Grafana writes the encrypted value to data_source.secure_json_data
   |
   v
3. On every backend request, Grafana decrypts the field, attaches it
   to the outbound HTTP request, and discards the cleartext in memory
   |
   v
4. Grafana returns only the response body to the browser; the
   credential never crosses the Grafana/browser boundary

The encryption is AES-GCM with a key derived from secret_key through a per-data-source salt. Rotating secret_key rotates the encryption key for every secureJsonData field in the install; Grafana re-encrypts the fields on next read.

The audit trail is two-tier:

• Data source admin API. GET /api/datasources lists every data source with non-sensitive fields only. POST/PUT to /api/datasources records the actor and the timestamp.
• Grafana audit log. The audit log records every data source create, update, and delete with the actor, the data source UID, and a diff of the non-sensitive fields. Secure field changes are recorded as changed fields without the values.

How to configure it

A Prometheus data source with basic auth

# /etc/grafana/provisioning/datasources/prometheus.yaml
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    uid: prometheus-prod
    orgId: 1
    url: http://prometheus.monitoring.svc:9090
    isDefault: true
    version: 1
    editable: false
    jsonData:
      timeInterval: 15s
      httpMethod: POST
      tlsAuthWithCACert: true
      tlsAuth: false
    secureJsonData:
      basicAuthPassword: ${PROM_BASIC_AUTH_PASSWORD}
      tlsCACert: ${PROM_CA_CERT}

The two secrets are read from the Grafana process environment. Grafana substitutes them at provisioning time and stores the encrypted values in the database.

A Loki data source with bearer token

# /etc/grafana/provisioning/datasources/loki.yaml
apiVersion: 1
datasources:
  - name: Loki
    type: loki
    access: proxy
    uid: loki-prod
    orgId: 1
    url: http://loki.monitoring.svc:3100
    jsonData:
      httpHeaderNames: ['X-Scope-OrgID']
      maxLines: 1000
    secureJsonData:
      httpHeaderValue1: ${LOKI_TENANT_HEADER}
      tlsClientCert: ${LOKI_CLIENT_CERT}
      tlsClientKey: ${LOKI_CLIENT_KEY}

A Tempo data source with mTLS

# /etc/grafana/provisioning/datasources/tempo.yaml
apiVersion: 1
datasources:
  - name: Tempo
    type: tempo
    access: proxy
    uid: tempo-prod
    orgId: 1
    url: https://tempo.monitoring.svc:3100
    jsonData:
      httpMethod: GET
      tlsAuthWithCACert: true
      tlsAuth: true
    secureJsonData:
      tlsCACert: ${TEMPO_CA_CERT}
      tlsClientCert: ${TEMPO_CLIENT_CERT}
      tlsClientKey: ${TEMPO_CLIENT_KEY}

systemd EnvironmentFile

# /etc/grafana/grafana.env
PROM_BASIC_AUTH_PASSWORD=actual-password-from-vault
LOKI_TENANT_HEADER=actual-tenant-from-vault
LOKI_CLIENT_CERT=actual-cert-pem
LOKI_CLIENT_KEY=actual-key-pem
TEMPO_CA_CERT=actual-ca-pem
TEMPO_CLIENT_CERT=actual-cert-pem
TEMPO_CLIENT_KEY=actual-key-pem

# /etc/systemd/system/grafana-server.service.d/secrets.conf
[Service]
EnvironmentFile=/etc/grafana/grafana.env
PermissionsStartOnly=true

How to validate it

# READ-ONLY: list every data source. The output is sanitised.
curl -fsS -H "Authorization: Bearer ${GF_SA_TOKEN}" \
  https://grafana.example.com/api/datasources | jq '.[] | {name, type, access}'
# {"name":"Prometheus","type":"prometheus","access":"proxy"}
# {"name":"Loki","type":"loki","access":"proxy"}
# {"name":"Tempo","type":"tempo","access":"proxy"}

# READ-ONLY: confirm secureJsonData is masked on read.
curl -fsS -H "Authorization: Bearer ${GF_SA_TOKEN}" \
  https://grafana.example.com/api/datasources/uid/prometheus-prod | jq
# {
#   "id": 1,
#   "uid": "prometheus-prod",
#   "type": "prometheus",
#   "access": "proxy",
#   "url": "http://prometheus.monitoring.svc:9090",
#   "jsonData": { "timeInterval": "15s" },
#   "secureJsonData": {}
# }

# READ-ONLY: confirm the data source actually works through Grafana.
curl -fsS -o /dev/null -w "%{http_code}\n" -G \
  -H "Authorization: Bearer ${GF_SA_TOKEN}" \
  --data-urlencode 'query=up' \
  https://grafana.example.com/api/datasources/proxy/uid/prometheus-prod/api/v1/query
# 200

# READ-ONLY: confirm access = direct is NOT used in production.
curl -fsS -H "Authorization: Bearer ${GF_SA_TOKEN}" \
  https://grafana.example.com/api/datasources | jq '.[] | select(.access == "direct")'
# (no output)

# READ-ONLY: confirm a backend health check passes through the proxy.
curl -fsS -H "Authorization: Bearer ${GF_SA_TOKEN}" \
  https://grafana.example.com/api/datasources/uid/loki-prod/health
# {"message":"ok","status":"success"}

# READ-ONLY: enumerate audit log entries for data source mutations.
curl -fsS -H "Authorization: Bearer ${GF_SA_TOKEN}" \
  'https://grafana.example.com/api/audit?from=2026-08-01&to=2026-08-14' \
  | jq '.[] | select(.action | test("datasource"; "i")) | {action, actor}'
# {"action":"datasource.created","actor":"alice@example.com"}
# {"action":"datasource.updated","actor":"ops-bot"}

How it can fail

The high-frequency credential failure shapes from real Grafana installs.

1. Literal credential in Git. A teammate commits a provisioning YAML with the password inline rather than sourced from the environment. The commit is in Git history forever; the password rotates but the leak remains.
2. Rotated secret in the IdP, not in the secrets manager. The credential is updated in the secrets manager but Grafana is not restarted and the new value is not re-loaded. Grafana keeps using the stale credential until the next provisioning sync.
3. Data source on access = direct. A debugging escape hatch left enabled in production. The credential is returned to the browser; it can be logged in the browser console, exported in a HAR file, or captured by an extension.
4. Environment variable not loaded by systemd. The EnvironmentFile is set but the systemd unit does not pick it up on a reload; Grafana starts with an empty value for the secret and the data source test fails on first request.
5. Multiple data sources with overlapping secrets. Two data sources for the same backend, configured six months apart, with different credentials, both still in use. An operator rotates one and the other silently breaks.
6. secureJsonData updated via the UI but provisioning overrides it. A provisioning sync runs every 30 seconds; a credential pasted into the UI is overwritten by the next sync from YAML. Operators do not know whether their change survived.

How to troubleshoot it

The diagnostic order matters: a 401 from a backend can come from five different places in the chain.

1. Pick the boundary. Is the failure at Grafana (the data source test fails), at the backend (Grafana forwards the request but the backend rejects it), or at the network (Grafana cannot reach the backend)?
2. Test the data source in the Grafana UI. Administration -> Data sources -> Test. The error message names the backend response: 401, 403, connection refused.
3. Inspect the provisioning log. tail -F /var/log/grafana/grafana.log | grep -E ‘datasource|provisioning’. A line of failed to load datasource shows the YAML file and the line number.
4. Confirm the environment variable is set in the Grafana process. Run ps eww $(pgrep -f grafana-server) | tr ' ' '\n' | grep PROM_BASIC. The variable must be present at process start; systemd EnvironmentFile changes require a restart.
5. Inspect the encrypted value at the database level. SELECT id, name, secure_json_data FROM data_source; on a staging copy. The values are opaque; you cannot decrypt them without the secret_key.
6. Verify the backend rejects only on bad credentials, not on network reachability. curl -fsS -i https://prometheus.example.com/api/v1/query?query=up with no credential returns 200 if the backend is reachable; 401 means the backend is up and the credential is the problem.

Security implications

• secureJsonData is encrypted at rest with secret_key. A snapshot of the database without secret_key is useless for the credentials. Rotate secret_key with the same discipline as a database password.
• access = proxy is the only safe mode. access = direct sends the credential to the browser and the browser becomes part of the trust boundary. Production should reject direct via configuration review.
• Server Admin and Org Admin both see the data source record. Org Admin sees secureJsonData on read; Server Admin sees it through the database. Limit both to named operators.
• The provisioning YAML is the source of truth. Operators who update secureJsonData via the UI will lose the change at the next provisioning sync.
• The audit log records the actor, the action, and the field names. It does not record the values. A log review is a who changed what review, not a what-was-the-password review.

Performance implications

• Decryption is per-request. Every backend request decrypts the secureJsonData field, attaches the credential, and discards the cleartext. The overhead is small but it appears on every query.
• The data source proxy adds one network hop. Browser -> Grafana -> backend. With Grafana on the same network as the backend, the hop is sub-millisecond. With Grafana across a WAN from the backend, every query pays the latency twice.
• Provisioning sync is bounded. The loader reads YAML files every 30 seconds by default; the cost is one read per file per interval. A Grafana with hundreds of data sources should raise the interval or move to a push-based provisioner.
• The audit log grows with every data source mutation. A team that updates data sources frequently produces a noisy audit log. Filter on action=datasource.* during review.

Production guidance

• access = proxy on every data source; reject access = direct in configuration review.
• secureJsonData sourced from the secrets manager; never literal in the provisioning YAML.
• systemd EnvironmentFile managed by the secrets manager; restart Grafana to pick up changes.
• Data sources provisioned as code; UI edits to secureJsonData fail at the next sync.
• Rotate every backend credential on a fixed cadence; align the cadence with the IdP signing key rotation.
• Audit log reviewed on a fixed cadence: every data source mutation has an actor, every actor is a named operator, every operator has the role they should have.

Verification

You should now be able to answer:

• What is the difference between jsonData and secureJsonData, and where is each one stored?
• Why does access = direct hand the credential to the browser, and what is the blast radius of that hand-off?
• What is the right way to source a data source password from the secrets manager into a provisioning YAML?
• How do you confirm that the credential is encrypted at rest and that the audit log records the actor without the value?
• What is the recovery path if a credential is found in Git history?

Quiz