Testing Provisioning Locally

A developer writes a new dashboard on the staging Grafana. The dashboard works. The pull request is merged. The production Grafana is rebuilt and the loader complains: data source with uid prom-prod exists. The reason is that the developer referenced the staging data source UID, not the production one. The dashboard loads in staging because the UID matches; the dashboard fails in production because the UID does not match.

This lesson is about the test setup that catches this kind of mistake before the production roll-out. The setup is a disposable Grafana with the same provisioning directory, a synthetic data source, and a sandbox that exercises the loaders end-to-end.

What testing provisioning locally is

Testing provisioning locally is the practice of running a disposable Grafana with the same provisioning directory as production, validating that the loaders apply every file, and confirming the API endpoints report the expected state. The test is performed in a sandbox — a Docker container, a separate namespace, or a CI runner — that is destroyed after the test. The cost is a few minutes of setup; the benefit is the absence of “what changed in production” surprises.

The test setup is:

• A Grafana container with the same apiVersion: 1 YAML files.
• A synthetic data source (the testdata plugin) that produces predictable values without a real backend.
• A curl-based script that calls the admin API and validates the response.
• A docker compose down -v at the end to return the host to working state.

The discipline is to never commit a provisioning change without running the test. The test is the contract between the developer’s laptop and the production Grafana.

Why a sysadmin cares

Two reasons, each one a class of production failure:

1. Loader failures are silent. A malformed YAML, a UID typo, a missing folder UID, a plugin version mismatch — all of these are logged at WARN level and do not crash the loader. The dashboard is dropped, the data source is rejected, the alert rule is skipped. The production error is the absence of a dashboard, not a stack trace.
2. Cross-resource dependencies are invisible. A dashboard that references a missing UID is provisioned successfully; the panel fails at query time. The dashboard is in the database; the panel is broken. The test must check the cross-reference, not just the file presence.

The sandbox is the only place the operator can run the test with confidence. A staging Grafana is the wrong place: the staging data sources and dashboards are different from production. The test must be a self-contained environment.

How it works

The test setup is a Docker Compose stack with one Grafana container, one synthetic data source, and a validation script.

+--------------------+
|  Test runner       |
|  (shell + curl)    |
|  runs:             |
|   docker compose up|
|   curl /api/...    |
|   validate         |
|   docker compose   |
|     down -v        |
+---------+----------+
          |
          v
+--------------------+
|  Grafana sandbox   |
|  image: grafana/   |
|   grafana:11.2.0   |
|  volumes:          |
|   - provisioning/  |
|   - testdata.yaml  |
|  env:              |
|   GF_AUTH_ANON:    |
|    true            |
+---------+----------+
          |
          v
+--------------------+
|  TestData plugin   |
|  (built-in)        |
|  yields:           |
|   CSV / random     |
|   series for       |
|   dashboards       |
+--------------------+

The synthetic data source is the testdata plugin that ships with the Grafana image. The plugin has no external dependencies and produces deterministic values through CSV content and Random walk data shapes. The dashboard JSON in the test references the test data source; the validation script confirms the panels render.

For the validation, the script calls the admin API:

• /api/datasources — the data sources are loaded.
• /api/search?folderUIDs={x} — the dashboards are loaded.
• /api/dashboards/uid/{uid} — the provisioned flag is true.
• /api/v1/provisioning/alert-rules — the alert rules are loaded.
• /api/admin/provisioning/dashboards/reload — the reload endpoint is reachable.

The test is a green bar when every endpoint returns the expected value. The test is a red bar when any endpoint returns an unexpected value or a non-200 status.

How to configure it

A production-grade test setup is a Docker Compose file with two services and a validation script.

# docker-compose.test.yaml
services:
  grafana-test:
    image: grafana/grafana:11.2.0
    ports:
      - "3001:3000"
    environment:
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
      - GF_USERS_DEFAULT_THEME=light
      - GF_LOG_LEVEL=warn
    volumes:
      - ./provisioning:/etc/grafana/provisioning:ro
      - ./testdata.yaml:/etc/grafana/provisioning/datasources/testdata.yaml:ro

The sandbox provisioning directory mirrors the production layout. The testdata.yaml is the synthetic data source:

# testdata.yaml
apiVersion: 1
datasources:
  - name:           TestData
    uid:            testdata
    type:           testdata
    access:         proxy
    orgId:          1
    editable:       false
    jsonData:
      timeInterval: 30s

The dashboard JSON in the test references the test data source by the canonical UID:

{
  "uid":          "checkout-error-rate-test",
  "title":        "Checkout error rate (test)",
  "schemaVersion": 39,
  "editable":     false,
  "tags":         ["test"],
  "panels": [
    {
      "type":       "timeseries",
      "title":      "Random walk",
      "datasource": {
        "type": "testdata",
        "uid":  "testdata"
      },
      "targets": [
        {
          "refId": "A",
          "datasource": {
            "type": "testdata",
            "uid":  "testdata"
          },
          "scenarioId": "random_walk"
        }
      ]
    }
  ]
}

The validation script:

#!/bin/bash
# bin/test-provisioning.sh
set -euo pipefail

echo "Starting Grafana sandbox..."
docker compose -f docker-compose.test.yaml up -d

echo "Waiting for Grafana to be ready..."
for i in {1..30}; do
  if curl -sf http://localhost:3001/api/health > /dev/null; then
    break
  fi
  sleep 1
done

echo "Validating data sources..."
curl -sf http://localhost:3001/api/datasources \
  | jq -e '.[] | select(.uid == "testdata")' > /dev/null

echo "Validating dashboards..."
curl -sf "http://localhost:3001/api/search?query=checkout-error-rate-test" \
  | jq -e '.[0].uid == "checkout-error-rate-test"' > /dev/null

echo "Validating the provisioned flag..."
curl -sf "http://localhost:3001/api/dashboards/uid/checkout-error-rate-test" \
  | jq -e '.meta.provisioned == true' > /dev/null

echo "Validating the reload endpoint..."
curl -sf -X POST http://localhost:3001/api/admin/provisioning/dashboards/reload \
  | jq -e '.message | test("reloaded")' > /dev/null

echo "Tearing down the sandbox..."
docker compose -f docker-compose.test.yaml down -v

echo "All checks passed."

The script is the contract. The script returns a non-zero exit code on the first failure; the CI run fails; the pull request is blocked.

How to validate it

Five checks confirm the test setup is correct.

# 1. The Grafana sandbox is up.
docker compose -f docker-compose.test.yaml ps
# NAME                  SERVICE         STATUS
# xxx-grafana-test-1    grafana-test    Up 5 seconds (healthy)

# 2. The Grafana health endpoint is reachable.
curl -sf http://localhost:3001/api/health | jq
# {"database":"ok","version":"11.2.0"}

# 3. The data sources are loaded.
curl -sf http://localhost:3001/api/datasources | jq '.[] | {uid, type}'
# {"uid":"testdata","type":"testdata"}

# 4. The dashboards are loaded.
curl -sf "http://localhost:3001/api/search?query=test" \
  | jq '.[] | {uid, title}'
# {"uid":"checkout-error-rate-test","title":"Checkout error rate (test)"}

# 5. The provisioned flag is true.
curl -sf "http://localhost:3001/api/dashboards/uid/checkout-error-rate-test" \
  | jq '.meta.provisioned'
# true

A false value at the last check means the dashboard is in the database but is UI-edited, not file-based. The cause is usually a UI edit that raced the loader; the fix is to reset the dashboard to the file-based version by deleting the row and reloading.

How it can fail

Six high-frequency failure shapes:

1. Wrong image tag. The test uses an older Grafana image that does not accept the dashboard schemaVersion. Symptom: the loader logs Dashboard schemaVersion 41 is newer than the supported 40; the dashboard is dropped.
2. Wrong port. The test uses a port that is already bound on the host. Symptom: the container exits with bind: address already in use; the validation script fails on the curl http://localhost:3001/api/health call.
3. Volume mount read-only. The test mounts the provisioning directory read-only. The reload endpoint cannot write to the database; the test fails. Symptom: the loader logs permission denied.
4. TestData plugin disabled. The Grafana image is built without the testdata plugin. Symptom: the data source is rejected with plugin not found: testdata; the provisioning fails.
5. UID collision with the production UID. The test uses the same UID as the production data source. Symptom: the loader logs data source with uid {x} exists; the data source is dropped.
6. Sandbox not torn down. The docker compose down -v is skipped. The next test runs against a stale database. The fix is to always run down -v in the script’s teardown.

How to troubleshoot it

The diagnostic order, designed to isolate which side of the test setup is broken.

1. Is the container running? docker compose -f docker-compose.test.yaml ps. A missing or crashed container is the dominant cause.
2. Is the health endpoint reachable? curl -sf http://localhost:3001/api/health. A connection refused means the container is not listening on the port.
3. Is the data source loaded? curl -sf http://localhost:3001/api/datasources. A missing source is a YAML error or a UID collision.
4. Is the dashboard loaded? curl -sf "http://localhost:3001/api/search?query=test". A missing dashboard is a loader error or a schemaVersion mismatch.
5. Is the provisioned flag true? curl -sf http://localhost:3001/api/dashboards/uid/\{uid\} | jq .meta.provisioned. A false value is a UI edit raced the loader, or the dashboard is not in the file.
6. Is the reload endpoint reachable? curl -sf -X POST http://localhost:3001/api/admin/provisioning/dashboards /reload. A 401 or 403 means the anonymous role is not Admin.

Security implications

• The sandbox uses anonymous admin. The test setup configures GF_AUTH_ANONYMOUS_ENABLED=true and GF_AUTH_ANONYMOUS_ORG_ROLE=Admin. The container is exposed on localhost only. The discipline is to never expose the sandbox on a public interface.
• The sandbox has the testdata plugin. The plugin is development-only. The discipline is to never deploy the testdata plugin in a production YAML.
• The provisioning directory is mounted read-only. The test container cannot write to the directory. The discipline is to keep the directory read-only on the container, even though the source is writable on the host.
• The validation script runs in CI. The CI runner has access to the source code. The discipline is to keep the validation script in version control and to never include production credentials in the script.

Performance implications

• The sandbox is a single container. The test runs in approximately 30 s of CI time. The cost is small.
• The docker compose down -v returns the host to working state. The cost is a few seconds of disk I/O.
• The validation script is a series of curl calls. The cost is a few hundred milliseconds of network I/O.
• The total cost of a test run is approximately 35 s. The benefit is the absence of “what changed in production” surprises.

Production guidance

• Run the test on every pull request that touches the provisioning directory. The test is the contract.
• Use the TestData plugin for synthetic data. The discipline is to never expose the production backends to the test.
• Mount the provisioning directory read-only on the container. The discipline is to keep the source on the host and the destination on the container separate.
• Tear down the sandbox at the end of the test. The discipline is to never leave a sandbox running.
• Pin the Grafana version in the test setup. The discipline is to test against the same version as production.

Verification

You should now be able to answer:

• What is the role of the TestData plugin in the test setup?
• Why is the test setup a disposable Grafana with the same provisioning directory as production?
• Which endpoint confirms the provisioned flag is true on a dashboard, and what does a false value mean?
• Why is the disableDeletion semantic relevant when the sandbox is torn down?
• What is the right discipline for the anonymous role in the sandbox configuration?

Quiz