Configuration Propagation

An operator updates a data source through the Admin UI at 11:00. The URL is changed from http://prometheus:9090 to http://prometheus-new:9090. The data source saves. The dashboard panels routed to grafana-01 (where the change was made) work. Panels routed to grafana-02 and grafana-03 (the other replicas) still hit the old URL. The user opens a ticket. The on-call engineer inspects the data source on each replica. The three replicas show different URLs. The cause is that the change was made through the Admin UI, but a provisioning file in /etc/grafana/provisioning/datasources/prometheus.yaml is also defined with the same data source UID. The provisioning file is unedited. At the next reload, the file’s URL overwrites the operator’s URL. The user is back to the old URL but on every replica this time. The drift cycle is now a known shape.

Configuration propagation is the mechanism by which a configuration change reaches every replica. There are two paths: the API/UI writes to the shared database; the provisioning files write to the database at boot or on reload. The right shape is to define the canonical configuration in provisioning files sourced from git, and to lock the data source from UI edits with editable: false.

What it is

Grafana configuration propagation is the pathway by which a configuration change reaches every replica. There are two paths:

1. The shared database. A change via the HTTP API or the Admin UI is written to the shared database. Every replica reads the database on the next query. The change is visible on every replica.
2. Provisioning files. A YAML file in /etc/grafana/provisioning/ is read on boot and on reload. Grafana parses the file, applies the configuration to the database, and the change is visible on every replica.

The two paths converge on the same database. The difference is who triggers the write: the API user or the file loader. The right shape is to use the file loader as the canonical source and the API for additions that are not yet in the file.

   Grafana API / UI
        |
        v
   shared database <-- reads on every query
        ^
        |
   /etc/grafana/provisioning/**/*.yaml
        |
        +-- read on boot
        +-- read on SIGHUP
        +-- applied to DB if editable: false

Why a sysadmin cares

The five operational pains that disappear once the propagation path is canonical:

1. Drift between replicas. A change made via the UI is visible on every replica (because the database is shared) until the next reload, when the provisioning file overwrites the change. The user sees the new URL for 10 minutes, then the old URL.
2. Lost change on reimage. A change made via the UI is in the database. A replica is reimaged. The provisioning file is fresh. The provisioning file is re-applied. The UI change is replaced. The change is lost.
3. No audit trail. A change via the UI is recorded in the audit log. A change via the file is a git commit. The combined audit trail is the git log; the file is the source of truth.
4. Rollback complexity. A change via the UI requires a database restore or a second API call to revert. A change via the file is a git revert and a reload.
5. Cross-environment drift. The staging data source is different from the production data source. The version field in the provisioning file is the lever that prevents silent re-application.

How it works

The provisioning loader reads every YAML file under the configured provisioning directory. For each file, it parses the YAML and applies the configuration to the database. The flow on a Grafana boot:

1. Read the [paths] provisioning directory. Default is /etc/grafana/provisioning.
2. Read every .yaml file in the directory.
3. For each file, parse the YAML and validate the schema.
4. For each defined resource, look up the database by UID.
5. If the UID exists, update the row.
6. If the UID does not exist, insert the row.
7. If editable: false, set the lock flag on the row.

The reload signal is SIGHUP or the HTTP call POST /admin/provisioning/dashboards/reload. The reload re-runs the same flow without restarting Grafana.

The version field is the lever that prevents silent drift. Bumping the version field forces the loader to re-apply the file even if the parsed configuration is identical. The convention is to bump the version on every intentional change.

How to configure it

The Grafana provisioning directory structure for a production deployment:

/etc/grafana/provisioning/
  datasources/
    prometheus.yaml
    loki.yaml
    tempo.yaml
  dashboards/
    default.yaml
    platform.yaml
  alerting/
    contact-points.yaml
    policies.yaml
    rules.yaml
  plugins/
    apps.yaml

Sample data source provisioning file:

# /etc/grafana/provisioning/datasources/prometheus.yaml
apiVersion: 1

datasources:
  - name: Prometheus
    uid: prometheus-prod
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    version: 1
    editable: false
    jsonData:
      timeInterval: 15s
      httpMethod: POST
      manageAlerts: true
    secureJsonData:
      basicAuthPassword: ${PROMETHEUS_BASIC_AUTH_PASSWORD}

The fields, annotated:

• uid — required for sourced provisioning. The UID is the lookup key. Two data sources with the same UID conflict.
• name — display name. Multiple data sources can share the same name but the lookup is by UID.
• type — prometheus, loki, tempo, mysql, postgres, influxdb, etc. Must match a loaded plugin.
• access — proxy (Grafana queries the data source) or direct (browser queries the data source). proxy is the default and the HA choice.
• isDefault — exactly one data source per organisation is the default. Multiple defaults are an error.
• version — bump to force re-application. Lever for the “I changed a file but the database is already at this version” problem.
• editable: false — the lock. The UI cannot edit the data source. The provisioning file is the only source.
• jsonData — non-secret data source configuration.
• secureJsonData — secret configuration. Source from environment variables; never commit.

Sample dashboards provisioning file:

# /etc/grafana/provisioning/dashboards/default.yaml
apiVersion: 1

providers:
  - name: default
    orgId: 1
    folder: General
    type: file
    disableDeletion: true
    editable: false
    updateIntervalSeconds: 30
    allowUiUpdates: false
    options:
      path: /var/lib/grafana/dashboards
      foldersFromFilesStructure: true

The provisioning directory is set in [paths]:

[paths]
provisioning = /etc/grafana/provisioning

The reload is triggered via SIGHUP:

# SERVICE-IMPACT
# Reload the provisioning without restarting Grafana.
kill -HUP 1

Or via the HTTP API:

# CONFIGURATION
# Reload the dashboards provisioning only.
curl -X POST -u admin:REDACTED \
  http://g1:3000/admin/provisioning/dashboards/reload

How to validate it

Confirm the runtime configuration matches the provisioning file on every replica.

# READ-ONLY
# Query the data source on each replica. The payload should
# be identical.
curl -s -u admin:REDACTED http://g1:3000/api/datasources/uid/prometheus-prod
curl -s -u admin:REDACTED http://g2:3000/api/datasources/uid/prometheus-prod
curl -s -u admin:REDACTED http://g3:3000/api/datasources/uid/prometheus-prod

{
  "id": 1,
  "uid": "prometheus-prod",
  "name": "Prometheus",
  "type": "prometheus",
  "url": "http://prometheus:9090",
  "readOnly": true
}

The readOnly: true field confirms editable: false is in effect. Compare the response across replicas.

# READ-ONLY
# Diff the data source list across replicas.
diff \
  <(curl -s -u admin:REDACTED http://g1:3000/api/datasources | jq -S .) \
  <(curl -s -u admin:REDACTED http://g2:3000/api/datasources | jq -S .)

An empty diff is the right answer.

# CONFIGURATION
# Trigger a reload and confirm the API response is 200.
curl -i -X POST -u admin:REDACTED \
  http://g1:3000/admin/provisioning/dashboards/reload

HTTP/1.1 200 OK

The reload is cheap. Confirm the loaded set is still identical.

How to fail

Six failure modes hit configuration propagation in production.

1. UI change overwritten by provisioning file. The editable: false flag is missing. Symptom: the user makes a change, the change is visible on every replica until the next reload, then the file overwrites the change.
2. YAML parse error. A typo in the file. Symptom: the Grafana log shows “provisioning: failed to parse /etc/grafana/provisioning/… yaml: …”; the file is ignored; the previous configuration stays.
3. Version not bumped. A manual change to the database (via the API) drifts from the file. Symptom: the database is at the new value, the file is at the old value, the reload does not re-apply because the version is unchanged.
4. Provisioning path differs across replicas. The mount in the container is different. Symptom: each replica has a different configuration set.
5. Two operators provisioning the same data source. One writes the file, the other writes the API. Symptom: last write wins; the audit trail is in two places.
6. No API permission to provisioning. The operator’s service account lacks the Admin role. Symptom: the API call to reload returns 403; the file is not re-applied.

How to troubleshoot it

Diagnose from the API inward.

1. Is the data source on every replica? curl /api/datasources from each replica. The list must match.
2. Is the data source read-only? The readOnly field in the payload. true means the UI cannot edit; false means the file is missing the editable: false flag.
3. Is the file loaded? The Grafana log on boot. Look for “Provisioning loaded” entries.
4. Did the reload succeed? The HTTP status of the /admin/provisioning/dashboards/reload call. 200 is healthy; 403 means permission; 500 means parse error.
5. Is the YAML valid? yq or yamllint against the file. The Grafana parser is strict; small mistakes break the file.
6. Is the UID unique? Two files with the same UID conflict. The loader logs the conflict and the second file is rejected.

Distinguish “is the file on disk?” (the file is mounted) from “is the file loaded?” (Grafana knows about it). The file can be present but unloaded because of a YAML parse error.

Security implications

Provisioning files contain secrets. The right deployment discipline is to source the secrets from a secret manager and to never commit them.

• Secrets in secureJsonData. Source from environment variables. Mount as a secret. Never commit.
• API tokens for the provisioning reload. The HTTP call requires the Admin role. Use a service account with the minimum role.
• File permissions. The provisioning directory must be readable by the Grafana process only. Mode 0750 owned by the Grafana user.
• Audit trail. Every provisioning change is a git commit. The git log is the audit trail. The database audit log records the API changes; the file changes are in git.

Performance implications

Provisioning is a one-off cost on boot and on reload. The runtime cost is the same as the API path: every query and mutation hits the shared database.

• Boot time. The provisioning loader reads, parses, and writes to the database. A large provisioning set (hundreds of data sources, thousands of dashboards) adds seconds to the cold start.
• Reload time. SIGHUP reload is fast. The HTTP reload endpoint re-runs the loader for the named resource type.
• Connection pool. The loader uses the same pool as the API. A reload during a peak load can transiently compete for connections.

Production guidance

The right approach is to source the provisioning files from git, set editable: false on every defined resource, and trigger reloads via the pipeline.

• All dashboards, data sources, alert rules, and contact points defined in provisioning files.
• Files sourced from git. The pipeline is the deployment mechanism.
• editable: false on every data source. allowUiUpdates: false on every dashboard provider.
• Reload via SIGHUP or the HTTP API after every commit.
• Drift detection. A periodic check that the runtime configuration matches the git state.
• Secrets in environment variables or secret files. Never in the provisioning file.

Verification

You should now be able to answer:

• What is the relationship between the provisioning file and the database?
• Which flag prevents UI edits from overwriting a sourced data source, and why is it critical in HA?
• How do you trigger a reload without restarting Grafana?
• What is the role of the version field in the provisioning file, and what happens when it is not bumped?
• Which file path is the default provisioning directory, and how is it configured?

Quiz