Provisioning Compatibility

At 14:08 the team restarts a Grafana instance after a routine config push. Within thirty seconds the alert rules in production stop firing. The on-call engineer opens the UI and finds that the alert rule list is empty.

The cause is a provisioning drift. The team manages alert rules through both the UI and the file-based provider. The provisioning files declare 47 rules; the database has 52. The provisioning tick rewrites the database to match the files, and the 5 rules that were only in the database disappear. Two of those rules were paging the on-call rotation. They had been “owned” by a team that left the company nine months ago, and nobody had migrated them to the provisioning files.

This is the failure shape that provisioning discipline prevents: a silent rewrite that erases the team’s work because the file-based provider and the database have diverged.

What provisioning compatibility is

Grafana provisioning is the mechanism by which the team manages datasources, dashboards, alert rules, and other configuration as code. The provisioning files are YAML documents that the Grafana binary reads on a timer (default every 30 seconds) and applies to the database.

Provisioning has two distinct compatibility shapes:

1. Schema stability. The YAML schema for each provider (datasources, dashboards, alert rules) is mostly stable across Grafana major versions. The most common change is a new optional field; a breaking change is rare. When a breaking change happens, it is announced in the release note.
2. UID stability. A datasource’s uid is the stable identifier that dashboards reference. A provisioning change that alters the uid of a datasource breaks every dashboard that references it. The team must not let the provisioning files change a uid accidentally.

The two shapes interact. A provisioning schema change can require a UID-aware migration; a UID change can require a schema update. The team tracks both in the provisioning discipline.

Why a sysadmin cares

Three operational pains the discipline prevents:

• Provisioning overwrites database edits. The provisioning tick rewrites the database row for every object the file declares. If the team has been editing through the UI, the UI edits silently revert. Symptom: dashboards or alert rules that the team edited through the UI disappear or revert on the next provisioning tick.
• UID change breaks every referencing dashboard. A provisioning change that alters a datasource uid produces a “Datasource not found” error on every panel that referenced the old uid. Symptom: panels that worked before the change render with “Datasource query error”.
• Provisioning path validation tightens. A Grafana minor release can tighten the validation of the paths the provisioning provider reads. A path that was silently ignored under the old binary becomes an error under the new one. Symptom: Grafana refuses to start, or refuses to load certain provisioning files, after the upgrade.

The cost of the discipline is roughly thirty minutes per upgrade. The cost of skipping it is a silent rewrite that erases alert rules the team depends on.

How it works: the provisioning tick

The provisioning tick runs every 30 seconds by default. Each provider walks its configured path, reads every YAML file, and applies the declared objects to the database:

1. Grafana binary starts
       |
2. Read /etc/grafana/grafana.ini [paths] section
       |
3. Read /etc/grafana/grafana.ini [provisioning] section
       |
4. For each provider:
       |   - datasources: /etc/grafana/provisioning/datasources
       |   - dashboards: /etc/grafana/provisioning/dashboards
       |   - alert rules: /etc/grafana/provisioning/alerting
       |   - plugins: /etc/grafana/plugins
       |
5. For each file in the path:
       |   - Parse the YAML
       |   - Validate against the provider schema
       |   - For each declared object:
       |       - If it has a uid, look up the database row
       |       - If the row exists, overwrite it
       |       - If the row does not exist, insert it
       |
6. Sleep 30 seconds, repeat

The dangerous step is the overwrite. The provider does not merge the YAML with the database; it replaces the database row with the YAML’s contents. A field that is in the database but not in the YAML is preserved; a field that is in the YAML but not in the database is added; a field that is in both is overwritten with the YAML’s value.

How to configure it: the provisioning discipline

The provisioning discipline has four steps. The team runs all four before every Grafana upgrade that crosses a minor boundary.

Step 1: pin the schema version in the file path. The team organises provisioning files into directories that record the Grafana version they target:

/etc/grafana/provisioning/
├── datasources/
│   ├── prometheus.yaml
│   ├── loki.yaml
│   └── tempo.yaml
├── dashboards/
│   ├── prom-overview.yaml
│   ├── loki-logs.yaml
│   └── tempo-traces.yaml
└── alerting/
    ├── high-priority.yaml
    └── low-priority.yaml

The directory layout is stable across Grafana versions. The team does not rename directories when upgrading Grafana; they rename them only when the provisioning schema for a provider changes in a breaking way.

Step 2: dry-run the provisioning on the canary. After the Grafana upgrade on the canary host, the team inspects the Grafana log for provisioning warnings or errors. The canary host has its own database; the provisioning tick rewrites the canary’s database from the files, but the canary’s database is isolated from the fleet.

# READ-ONLY: scan the canary's grafana log for provisioning
# warnings since the upgrade.
journalctl -u grafana-server --since "1 hour ago" | \
  grep -i "provisioning\|provision" | \
  grep -i "warn\|error"
# (empty output means the provisioning files parsed and
#  loaded cleanly)

Step 3: snapshot the production database. The team takes a snapshot of the production database before the provisioning tick has a chance to rewrite it. The snapshot is the rollback artefact.

# DATA-LOSS-RISK-adjacent: snapshot the production Grafana
# database. The team does this BEFORE the provisioning tick
# has rewritten it.
sqlite3 /var/lib/grafana/grafana.db \
  ".backup '/var/backups/grafana/grafana.$(date +%Y%m%d-%H%M%S).db'"

Step 4: commit the migrated provisioning files. Once the canary host has loaded the provisioning files cleanly and the team has confirmed the database has been rewritten to match, the team commits the migrated files to the provisioning repo. The fleet loads the migrated files on the next provisioning tick.

A representative provisioning file. The datasources provider:

# /etc/grafana/provisioning/datasources/prometheus.yaml
apiVersion: 1
datasources:
  - name: Prometheus
    uid: prom-1
    type: prometheus
    access: proxy
    url: http://prometheus.internal:9090
    isDefault: true
    editable: false
    jsonData:
      timeInterval: 30s
      httpMethod: POST
    secureJsonData:
      basicAuthPassword: ${PROM_BASIC_AUTH_PASSWORD}

The uid is the stable identifier. A dashboard that references this datasource uses uid: prom-1 in the panel’s targets[].datasource.uid field. Changing the uid here breaks every referencing dashboard.

The alert rules provider:

# /etc/grafana/provisioning/alerting/high-priority.yaml
apiVersion: 1
groups:
  - orgId: 1
    name: high-priority
    folder: Production
    interval: 1m
    rules:
      - uid: high-cpu-usage
        title: High CPU usage
        condition: A
        data:
          - refId: A
            datasourceUid: prom-1
            model:
              expr: avg(rate(node_cpu_seconds_total[5m])) > 0.9
              intervalMs: 1000
              maxDataPoints: 43200
        noDataState: NoData
        execErrState: Error
        for: 5m
        annotations:
          summary: 'CPU usage above 90% for 5 minutes'
          dashboard: 'https://grafana.internal/d/prom-overview'
        labels:
          severity: critical
          team: platform

How to validate it

The minimum validation set for a provisioning discipline after a Grafana upgrade. Every command is READ-ONLY unless flagged otherwise:

# READ-ONLY: enumerate every datasource and its uid. The
# team cross-references this against the provisioning files.
curl -fsS -u admin:admin \
  http://grafana-canary-01:3000/api/datasources | \
  jq '.[] | {id, uid, name, type}'
# {"id":1,"uid":"prom-1","name":"Prometheus","type":"prometheus"}
# {"id":2,"uid":"loki-1","name":"Loki","type":"loki"}
# {"id":3,"uid":"tempo-1","name":"Tempo","type":"tempo"}

# READ-ONLY: enumerate every alert rule and its uid. The
# team cross-references against the provisioning files.
curl -fsS -u admin:admin \
  http://grafana-canary-01:3000/api/v1/provisioning/alert-rules | \
  jq '.[] | {uid, title, folder}'
# {"uid":"high-cpu-usage","title":"High CPU usage","folder":"Production"}
# {"uid":"low-disk-space","title":"Low disk space","folder":"Production"}

# READ-ONLY: confirm the provisioning path is healthy. The
# /api/admin/provisioning/dashboards/reload endpoint forces
# a reload; a 200 response means the files parsed.
curl -fsS -u admin:admin -X POST \
  http://grafana-canary-01:3000/api/admin/provisioning/dashboards/reload
# (empty body on success)

# READ-ONLY: confirm the Grafana log has no provisioning
# warnings or errors since the upgrade.
journalctl -u grafana-server --since "1 hour ago" | \
  grep -i "provisioning\|provision" | \
  grep -i "warn\|error" || echo "no warnings"

The validation order matters: datasource enumeration first, alert rule enumeration second, force-reload third, log scan fourth. The team should not consider the provisioning discipline complete until every object in the database has a matching declaration in the provisioning files.

How it can fail

Five failure modes recur in Grafana provisioning upgrades.

1. Provisioning overwrites database edits. The provisioning tick rewrites the database row for every object the file declares. Symptom: dashboards or alert rules the team edited through the UI disappear or revert on the next provisioning tick.
2. UID change breaks referencing dashboards. A provisioning change that alters a datasource uid produces “Datasource not found” on every panel. Symptom: panels that worked before the change render with “Datasource query error”.
3. Provisioning path validation tightens. A Grafana minor release tightens the validation of the paths the provisioning provider reads. Symptom: Grafana refuses to start, or refuses to load certain provisioning files, after the upgrade.
4. Schema field renamed. A provisioning schema change renames a field. Symptom: the old field is silently dropped on parse; the new field is required for full feature support.
5. Provisioning files reference deleted folders. A dashboard that was moved or deleted in the UI but is still in the provisioning files produces a “folder not found” warning on every provisioning tick. Symptom: the dashboard is not loaded; the Grafana log records a warning.

How to troubleshoot it

When provisioning goes wrong after an upgrade, the diagnostic order matters. Start at the file view and move toward the database view.

1. What does the provisioning log say? Read journalctl -u grafana-server. Look for provisioning, failed to parse, folder not found, unknown field.
2. What does the file say? Read the YAML file directly. Confirm the schema matches the target Grafana version. Look for renamed fields, missing required fields, and undeclared secureJsonData fields.
3. What does the database say? Run the datasource and alert-rule enumeration queries against the API. Cross- reference against the provisioning files.
4. What does the diff say? Diff the provisioning files against the database contents. The diff shows what the provisioning tick will change on the next tick.
5. Form a hypothesis. Pin the failure to one of the five failure modes above. The most common is “provisioning overwrites database edits”.
6. Find evidence. Compare the database contents before and after the provisioning tick. The diff shows what was rewritten.
7. Test the hypothesis. Edit the provisioning files to add the missing objects. Force a reload. Confirm the database is rewritten correctly.

The diagnostic order is “did the file parse before asking whether the database matches.”

Security implications

Three security implications are specific to Grafana provisioning work:

• secureJsonData in version control. The provisioning YAML may contain secureJsonData fields with credentials. These must come from environment variable substitution, not from committed values. The provisioning repo’s .gitignore must exclude any file that contains a literal credential.
• Provisioning path permissions. The directories the provisioning provider reads must be readable only by the Grafana user. A world-readable provisioning directory is a disclosure vector for datasource URLs and (if the team is careless) credentials.
• Provisioning reload endpoint. The /api/admin/provisioning/dashboards/reload endpoint forces a reload. The endpoint must be protected by the same authentication as the rest of the admin API. A reload triggered by an unauthenticated request can be used to timing-attack the provisioning files.

Performance implications

Performance implications of a Grafana provisioning upgrade are not symmetric with the upgrade’s risk:

• Provisioning tick frequency. The default 30-second tick means a change to a provisioning file takes up to 30 seconds to apply. A team that needs faster turnaround can lower the interval, at the cost of more disk I/O on the Grafana host.
• YAML parse cost. A directory with hundreds of YAML files takes longer to parse than one with a single file. The team should consolidate provisioning files where possible without losing readability.
• Database rewrite cost. Every provisioning tick rewrites the database rows for every declared object. A fleet with thousands of dashboards will see measurable database load on every tick. The team should benchmark the database load before and after a provisioning change.

The release note will not call out performance implications of provisioning work specifically. The validation step is where the team notices.

Production guidance

• Pin provisioning schema versions in the file path. The directory layout records the Grafana version the files target. A breaking schema change is announced in the release note; the team responds by renaming the directory and updating the files.
• Snapshot the database before the provisioning tick. The database snapshot is the rollback artefact. The tick has the ability to rewrite the database in ways the team cannot undo.
• Treat the database as derived state. The provisioning files are the source of truth. The database is a cache that the provisioning tick refreshes. The team should not edit through the UI and expect the change to survive the next tick.
• Audit the database against the files quarterly. A drift between the files and the database is a drift between what the team has provisioned and what the team is running. The audit is the only reliable check.

Verification

You should now be able to answer:

• What is the difference between the database and the provisioning files in terms of which is the source of truth?
• Why must the database be snapshotted before the provisioning tick has a chance to rewrite it?
• What does a UID change in a provisioning file break?
• Why is the audit between the files and the database a quarterly discipline rather than a per-upgrade one?

Quiz