Drift Prevention

A junior engineer cannot find a panel in the production Grafana. The senior engineer opens the dashboard, sees the panel, and explains what it does. The junior engineer leaves the call, opens the Grafana UI, edits the panel, and clicks “Save”. The next provisioning poll, 60 s later, reverts the change. The junior engineer is confused. The senior engineer is annoyed.

This lesson is about the discipline that prevents the confusion. The discipline is “edit only in code”. The mechanism is the file-based source of truth, the allowUiUpdates: false flag, and the code-review workflow.

What drift prevention is

Drift prevention is the combination of operational policies and technical controls that keeps the file-based provisioning source in lockstep with the live Grafana state. The policies are “edit only in code”, “every change is a pull request”, and “no UI edits after the first provisioning”. The technical controls are editable: false, allowUiUpdates: false, and the provisioned flag on the dashboard metadata.

The two sources of drift are:

• A UI edit that is reverted at the next poll.
• A file edit that is not picked up by the loader.

The first is the dominant failure mode. The second is rarer and is usually a path or permissions error.

Why a sysadmin cares

The audit trail is the operational reason. A Grafana change that is “in the UI” is invisible to a code review. A Grafana change that is “in the file” is a commit. The commit is the audit; the diff is the change; the reviewer is the gate; the revert is the rollback.

Three operational disciplines only drift prevention supplies:

1. Reproducibility. A second Grafana (a staging replica, a regional instance, a disaster-recovery host) is the same files. The UI-edited state is not reproducible.
2. Audit. Every change is in a pull request with a reviewer. The change log is git log -- provisioning/. The default of UI edits is no log at all.
3. Rollback. git revert undoes a dashboard change in the same workflow as a code change. The rollback is also versioned: a later git log reveals what was rolled back and when.

Without drift prevention, a Grafana instance is a hand-built artefact. With drift prevention, it is a deployable.

How it works

Drift prevention is two layers. The technical layer is the loader reconciling declared to live; the policy layer is the code review.

   +--------------------------+
   |  Operator opens a panel  |
   |  in the Grafana UI       |
   +------------+-------------+
                |
                v
   +--------------------------+
   |  editable: false?        |
   +------------+-------------+
                |
        yes     |     no
        |       v       |
        |   +----------+
        |   |  Allow   |
        |   |  the UI  |
        |   |  edit    |
        |   +----+-----+
        |        |
        |        v
        |   +--------------------------+
        |   |  allowUiUpdates: false?  |
        |   +------------+-------------+
        |                |
        |         yes    |     no
        |         |      v       |
        |         |   +----------+
        |         |   |  Revert  |
        |         |   |  at the  |
        |         |   |  next    |
        |         |   |  poll    |
        |         |   +----------+
        |         v
        |   +--------------------+
        |   |  provisioned flag  |
        |   |  on the dashboard  |
        |   |  metadata is true  |
        |   +---------+----------+
        |             |
        v             v
   +------------------------------+
   |  UI edit rejected at save   |
   |  with "this dashboard is    |
   |  read-only" message         |
   +------------------------------+

The technical layer is the loader’s reconcile loop. The policy layer is the code review. The two together are the only reliable defence.

The editable: false flag is set on the data source or on the dashboard JSON. The flag is enforced at save time: the UI displays the “Save” button as disabled or rejects the save with a clear message. The operator experiences the rejection immediately.

The allowUiUpdates: false flag is set on the dashboard provider block. The flag is enforced at the next poll: the loader reverts the UI edit. The operator experiences the revert as a “my change vanished” event.

The provisioned flag on the dashboard metadata is the runtime proof of the source. A provisioned: true value confirms the dashboard is file-based. A provisioned: false value (the default for UI-created dashboards) confirms the dashboard is not file-based.

How to configure it

The policy is enforced by the technical controls. The controls are present in the YAML files.

# /etc/grafana/provisioning/datasources/metrics.yaml
apiVersion: 1
datasources:
  - name:           Prometheus
    uid:            prom-prod
    type:           prometheus
    access:         proxy
    url:            http://prometheus.monitoring.svc:9090
    editable:       false
    jsonData:
      timeInterval: 30s
    secureJsonData:
      basicAuthPassword: ${PROM_READ_TOKEN}

# /etc/grafana/provisioning/dashboards/prod.yaml
apiVersion: 1
providers:
  - name:               prod-sre
    orgId:              1
    folderUid:          sre
    folder:             SRE
    type:               file
    disableDeletion:    false
    updateIntervalSeconds: 30
    allowUiUpdates:     false
    options:
      path:             /etc/grafana/provisioning/dashboards/prod-sre

{
  "uid":          "checkout-error-rate",
  "title":        "Checkout error rate",
  "schemaVersion": 39,
  "editable":     false,
  "panels":       []
}

The three flags together: editable: false on the data source blocks the UI from saving a data source override; allowUiUpdates: false on the provider reverts any UI edit on a dashboard at the next poll; editable: false on the JSON blocks the UI from saving a panel edit.

The CI check is the policy enforcement. The check fails the build if a provisioning file is changed without a corresponding reviewer, and a separate check fails the build if the provisioned flag is set on a UI-created dashboard.

The check is a script in the CI pipeline:

#!/bin/bash
# bin/check-drift.sh
set -euo pipefail

# 1. No UI-edited dashboards are present in the live state.
#    The /api/dashboards/uid/{uid} endpoint returns the
#    provisioned flag. A false value means the dashboard is
#    UI-edited.
curl -sf -u "grafana-admin:$GF_ADMIN_PASSWORD" \
  "http://grafana:3000/api/search?folderUIDs=sre" \
  | jq -r '.[] | select(.uid != null) | .uid' \
  | while read uid; do
    provisioned=$(curl -sf -u "grafana-admin:$GF_ADMIN_PASSWORD" \
      "http://grafana:3000/api/dashboards/uid/$uid" \
      | jq -r '.meta.provisioned')
    if [[ "$provisioned" != "true" ]]; then
      echo "Drift detected: dashboard $uid is not provisioned"
      exit 1
    fi
  done

echo "No drift detected."

The script is run by the CI pipeline on every pull request. A failure blocks the merge. The discipline is to never merge a pull request that introduces drift.

How to validate it

Five checks confirm drift prevention is in place.

# 1. The provisioned flag is true on every dashboard.
curl -sf -u "grafana-admin:$GF_ADMIN_PASSWORD" \
  "http://grafana:3000/api/search?folderUIDs=sre" \
  | jq -r '.[] | .uid' \
  | while read uid; do
    curl -sf -u "grafana-admin:$GF_ADMIN_PASSWORD" \
      "http://grafana:3000/api/dashboards/uid/$uid" \
      | jq -e '.meta.provisioned == true' > /dev/null \
      || echo "Drift: $uid is not provisioned"
  done

# 2. The data source is editable: false.
curl -sf -u "grafana-admin:$GF_ADMIN_PASSWORD" \
  "http://grafana:3000/api/datasources/uid/prom-prod" \
  | jq '.readOnly'
# true

Note: readOnly in the API response is the inverse of editable. The API returns readOnly: true when the source has editable: false.

# 3. The provider block has allowUiUpdates: false.
grep -A5 'name: prod-sre' \
  /etc/grafana/provisioning/dashboards/prod.yaml \
  | grep -i 'allowUiUpdates'
# allowUiUpdates:     false

# 4. The JSON-level editable flag is false.
jq '.editable' \
  /etc/grafana/provisioning/dashboards/prod-sre/checkout-error-rate.json
# false

# 5. The git log shows the change as a commit.
git log --oneline -- provisioning/dashboards/prod.yaml \
  | head -5
# a3f2d1c (HEAD -> main) feat: add checkout-error-rate dashboard
# 8b71e29 fix: update prom-prod UID in metrics.yaml
# 4c93faa chore: initial commit of provisioning directory

A missing commit for a current dashboard is a drift. The discipline is that every provisioning change is a commit; the git log is the audit trail.

How it can fail

Six high-frequency failure shapes:

1. UI edit raced by the loader. An operator edits a panel in the UI. The next provisioning poll reverts the change. Symptom: the operator’s “fix” vanishes within a minute.
2. allowUiUpdates: true left on by mistake. The provider block has allowUiUpdates: true (the default in some legacy files). Symptom: a UI edit is accepted; the dashboard displays “edited by the operator” when it was really reverted by the poll.
3. Data source created through the UI. An operator creates a new data source in the UI to test a connection. The next provisioning poll deletes the data source. Symptom: the test data source vanishes within a minute.
4. Dashboard created through the UI. A new dashboard is created in the UI. The next provisioning poll deletes the dashboard. Symptom: the new dashboard vanishes.
5. Drift undetected in CI. The CI check is missing or skipped. Symptom: a pull request merges with a UI-edited dashboard in the live state.
6. The provisioned flag is stale. A dashboard is UI-edited; the flag is still true. The CI check passes; the dashboard is drifted. Symptom: the dashboard displays the file-based value at the next poll, which is the value the operator thought they had overwritten.

How to troubleshoot it

The diagnostic order, designed to isolate which side of the drift prevention is broken.

1. Is the provisioned flag true? curl -sf http://grafana:3000/api/dashboards/uid/\{uid\} | jq .meta.provisioned. A false value means the dashboard is UI-edited or UI-created.
2. Is the data source editable: false? curl -sf http://grafana:3000/api/datasources/uid/\{uid\} | jq .readOnly. A false value means the data source is editable; UI edits are allowed.
3. Is the provider allowUiUpdates: false? grep allowUiUpdates /etc/grafana/provisioning/dashboards/*.yaml. A true value means UI edits are accepted.
4. Is the JSON editable: false? jq '.editable' {file}. A true value means the UI can edit the dashboard.
5. Is the CI check running? git log --oneline -- bin /check-drift.sh. A missing entry means the check is not in CI.
6. Is the git log audited? git log --oneline -- provisioning/. A missing commit for a current state means a UI edit slipped through.

Security implications

• The UI is a privilege escalation surface. A UI edit on a provisioned dashboard is a privilege escalation if the operator has the Editor role but the team policy is “edit only in code”. The editable: false flag closes the surface.
• The provisioned flag is a security signal. A provisioned: false value is a dashboard that is not in version control. The dashboard is invisible to the audit. The discipline is to enforce provisioned: true on every production dashboard.
• The git log is the audit trail. A missing commit for a current state is a drift. The discipline is to audit the git log on every provisioning change.
• The CI check is the gate. A missing CI check is a drift vector. The discipline is to never merge a pull request without the drift check.

Performance implications

• The CI check is a series of curl calls. The cost is a few hundred milliseconds of network I/O.
• The drift check is run on every pull request. The cost is approximately 5 s of CI time per pull request.
• The allowUiUpdates: false flag reverts UI edits at the next poll. The cost is a single database write per UI edit.
• The editable: false flag rejects UI edits at save time. The cost is a single HTTP round-trip.

Production guidance

• Set editable: false on every production data source.
• Set allowUiUpdates: false on every production provider.
• Set editable: false on every production dashboard JSON.
• Run the drift check in CI on every pull request.
• Audit the git log on every provisioning change.
• Enforce the “edit only in code” policy in the team handbook.

Verification

You should now be able to answer:

• What is the difference between a UI edit and a git edit of a Grafana resource?
• What is the role of editable: false versus allowUiUpdates: false in drift prevention?
• How does the CI check detect drift between the file-based source and the live state?
• Why is the git log the audit trail for a Grafana change?
• What is the right discipline for a developer who needs to test a dashboard change against the production data source?

Quiz