Lab: Provision Grafana from Disk

Objective

By the end of this lab you will have a Grafana whose data source, folder and dashboard exist only because of four files on disk — destroy the database and they all come back — and you will have reproduced, from the command line, the four ways provisioning fails without ever returning an error: a variable that was not set, a UID that resolves to nothing, a save that should have been rejected, and a deletion nobody asked for.

Architecture

Two containers. Prometheus scrapes itself, which gives the dashboard something real to draw; Grafana reads its entire configuration from a bind-mounted directory.

  host filesystem                          containers
  ~/grafana-provisioning-lab/
    prometheus.yml  ------------------->  prometheus :9090
    provisioning/                          (scrapes itself)
      datasources/                              ^
        metrics.yaml  ---------+                |
      dashboards/              |                | proxy query
        lab.yaml  ------+      |                |
        json/           |      |                |
          overview.json |      |                |
                        |      |                |
                        v      v                |
                    grafana :3000 ---------------+
                      |
                      +-- loader: datasources  (poll)
                      +-- loader: dashboards   (updateIntervalSeconds)
                      +-- /var/lib/grafana/grafana.db  (derived state)

The direction of that last arrow is the whole subject. grafana.db is downstream of the files: it is a cache of a declaration, not a source of truth. Every failure in this lab comes from somebody treating it as the other way round.

Requirements

A disposable Linux host with Docker Engine 28.x and the Compose v2 plugin. Everything lives under one directory and two named volumes.
Ports 3000 and 9090 free on loopback. Task 1 checks.
curl, jq and openssl on the host.
About 700 MB of free disk for two images.
No out-of-band access requirement. Nothing here touches host networking, the firewall, or SSH.

Scenario

The Grafana you inherited has forty dashboards and nobody can say where any of them came from. Two are duplicates with slightly different queries. One references a data source that was deleted in March and renders “datasource not found” on every panel, which the team has learned to ignore. There is no staging copy, because there is no way to build one: the instance is the artefact.

The remit is to make the next Grafana a deployable. Everything in files, everything reviewable, everything reproducible from an empty database. The part of that remit people underestimate is the second half of this lab: the loader is deliberately forgiving, and almost every way this goes wrong goes wrong quietly.

Tasks

Task 1: Lay out the tree the loader expects

The directory layout is not a convention you can rearrange. Each subdirectory under the provisioning root is owned by exactly one loader, and a file in the wrong one is not an error — it is simply never read.

LAB="$HOME/grafana-provisioning-lab"
mkdir -p "$LAB"/provisioning/{datasources,dashboards/json} "$LAB/secrets"
cd "$LAB"

for port in 3000 9090; do
  ss -ltn "sport = :$port" | grep -q LISTEN && echo "PORT $port IN USE"
done

openssl rand -base64 24 | tr -d '=/+' > secrets/grafana_admin
chmod 0600 secrets/grafana_admin

find "$LAB" -type d | sort

Task 2: Give the dashboard something real to draw

# prometheus.yml
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']

Prometheus scraping itself is not a toy: prometheus_http_requests_total and up are real series produced by a real process, which is exactly what you need to tell “the dashboard is wired correctly” apart from “the dashboard has no data because nothing is producing any”.

# docker-compose.yaml
name: rb-obs-provisioning

services:
  prometheus:
    image: prom/prometheus:v2.55.1
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.path=/prometheus
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus-data:/prometheus
    ports:
      - "127.0.0.1:9090:9090"

  grafana:
    image: grafana/grafana:11.3.0
    environment:
      GF_SECURITY_ADMIN_PASSWORD__FILE: /run/secrets/grafana_admin
      GF_AUTH_ANONYMOUS_ENABLED: "false"
      GF_USERS_ALLOW_SIGN_UP: "false"
      # Read by the provisioning file in Task 3. Deliberately the only
      # place the backend URL is written down.
      PROM_URL: http://prometheus:9090
    volumes:
      # Read-only on purpose: the container must never be able to edit
      # its own declaration. You edit these files on the host.
      - ./provisioning:/etc/grafana/provisioning:ro
      - ./secrets/grafana_admin:/run/secrets/grafana_admin:ro
      - grafana-data:/var/lib/grafana
    ports:
      - "127.0.0.1:3000:3000"

volumes:
  prometheus-data:
  grafana-data:

Task 3: Declare the data source

# provisioning/datasources/metrics.yaml
apiVersion: 1
datasources:
  - name: Prometheus
    # The UID is the only stable handle. Every dashboard panel refers
    # to this string. Set it explicitly: a generated UID is derived
    # from the name and changes when somebody renames the source.
    uid: prom-lab
    type: prometheus
    access: proxy
    orgId: 1
    # Substituted from the process environment at load time - once,
    # not per query. A rotated value needs a reload or a restart.
    url: ${PROM_URL}
    isDefault: true
    # The UI cannot save an override on this source.
    editable: false
    jsonData:
      httpMethod: POST
      timeInterval: 15s

Task 4: Declare the folder, the provider and the dashboard

The provider file says where dashboards come from and what the loader is allowed to do with them. The dashboard JSON is the content. Two files, because you want to move the content without rewriting the declaration.

# provisioning/dashboards/lab.yaml
apiVersion: 1
providers:
  - name: lab-sre
    orgId: 1
    folderUid: lab-sre
    folder: Lab SRE
    # `file` is the provider type. The declaration lives here; the
    # content lives wherever `options.path` points.
    type: file
    # 10s so this lab does not spend its life waiting. Production
    # values are 30-60s; the poll is a full directory scan.
    updateIntervalSeconds: 10
    # A dashboard removed from the source is deleted from the database
    # on the next poll. Task 8 makes this happen on purpose.
    disableDeletion: false
    # A UI save on these dashboards is refused rather than reverted.
    allowUiUpdates: false
    options:
      path: /etc/grafana/provisioning/dashboards/json
      foldersFromFilesStructure: false

{
  "uid": "lab-overview",
  "title": "Lab overview",
  "schemaVersion": 39,
  "version": 1,
  "editable": false,
  "timezone": "browser",
  "time": { "from": "now-30m", "to": "now" },
  "refresh": "30s",
  "tags": ["lab"],
  "templating": { "list": [] },
  "annotations": { "list": [] },
  "panels": [
    {
      "id": 1,
      "type": "timeseries",
      "title": "Prometheus HTTP requests",
      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 },
      "datasource": { "type": "prometheus", "uid": "prom-lab" },
      "targets": [
        {
          "refId": "A",
          "datasource": { "type": "prometheus", "uid": "prom-lab" },
          "expr": "sum by (handler) (rate(prometheus_http_requests_total[5m]))"
        }
      ]
    },
    {
      "id": 2,
      "type": "stat",
      "title": "Targets up",
      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 0 },
      "datasource": { "type": "prometheus", "uid": "prom-lab" },
      "targets": [
        { "refId": "A", "datasource": { "type": "prometheus", "uid": "prom-lab" },
          "expr": "sum(up)" }
      ]
    }
  ]
}

Write that JSON to provisioning/dashboards/json/overview.json. Note the "uid": "prom-lab" inside each panel: that string, and only that string, joins the dashboard to the data source. Nothing checks it at provisioning time.

Two loaders, one UID, no negotiation

The datasource loader and the dashboard loader are separate reconcilers running in the same process on their own intervals. They do not talk to each other.

The datasource loader keys on the data source UID. The dashboard loader keys on folderUid plus the dashboard uid. The panel’s datasource.uid is not a key for either of them — it is opaque content inside a JSON blob that the dashboard loader stores verbatim.

Resolution happens at query time, in the browser, when somebody opens the dashboard. So a dashboard referencing a data source that has never existed provisions cleanly, logs a success, and reports provisioned: true. Task 7 does exactly that, and it is the mechanism behind the “renders datasource not found and everyone ignores it” dashboard in the scenario.

Task 5: Start it and read the loader, not the UI

Configuration changelab host— Starts Prometheus and Grafana; Grafana provisions from the bind mount on boot

$ cd ~/grafana-provisioning-lab && docker compose up -d

cd "$HOME/grafana-provisioning-lab"

for i in $(seq 1 30); do
  curl -sf http://127.0.0.1:3000/api/health >/dev/null &&
  curl -sf http://127.0.0.1:9090/-/ready   >/dev/null && break
  sleep 2
done

# What the loaders did, in their own words. Read this before the API:
# it is the only place a skipped file is mentioned.
docker compose logs grafana 2>&1 | grep -i provisioning | tail -20

Now confirm through the API. readOnly: true is the API’s way of reporting editable: false — the field is inverted, which trips people up:

GF_PW=$(cat secrets/grafana_admin)
API=http://127.0.0.1:3000

curl -sf -u "admin:$GF_PW" "$API/api/datasources" \
  | jq '.[] | {uid, type, url, readOnly}'

curl -sf -u "admin:$GF_PW" "$API/api/datasources/uid/prom-lab/health" \
  | jq '{status, message}'

curl -sf -u "admin:$GF_PW" "$API/api/search?folderUIDs=lab-sre" \
  | jq '.[] | {uid, title, type}'

curl -sf -u "admin:$GF_PW" "$API/api/dashboards/uid/lab-overview" \
  | jq '.meta | {provisioned, provisionedExternalId, folderTitle}'

Prove the panel query returns data, rather than assuming it because the dashboard loaded:

curl -sfG http://127.0.0.1:9090/api/v1/query \
  --data-urlencode 'query=sum by (handler) (rate(prometheus_http_requests_total[5m]))' \
  | jq '.data.result | length'

Task 6: Break it with an unset variable

The ${VAR} substitution has one property worth knowing before you rely on it: an undefined variable is not an error. It is an empty string.

cd "$HOME/grafana-provisioning-lab"

# Remove the variable the data source depends on and restart Grafana.
sed -i 's|^      PROM_URL: .*|      PROM_URL: ""|' docker-compose.yaml
docker compose up -d grafana
sleep 8

GF_PW=$(cat secrets/grafana_admin)
curl -sf -u "admin:$GF_PW" http://127.0.0.1:3000/api/datasources/uid/prom-lab \
  | jq '{uid, url}'

curl -sS -u "admin:$GF_PW" \
  http://127.0.0.1:3000/api/datasources/uid/prom-lab/health | jq '{status, message}'

The data source is still there. It is still provisioned. Its type is right, its UID is right, and its URL is the empty string. The health check is the only probe in the ladder that notices, which is why a data source health check belongs in the same pipeline as the provisioning apply.

Put it back:

sed -i 's|^      PROM_URL: .*|      PROM_URL: http://prometheus:9090|' docker-compose.yaml
docker compose up -d grafana
sleep 8
curl -sf -u "admin:$GF_PW" \
  http://127.0.0.1:3000/api/datasources/uid/prom-lab/health | jq -r '.status'

Task 7: Break it with a UID that resolves to nothing

cd "$HOME/grafana-provisioning-lab"
cp provisioning/dashboards/json/overview.json overview.json.good

sed -i 's/"uid": "prom-lab"/"uid": "prom-typo"/g' \
  provisioning/dashboards/json/overview.json
sleep 15

GF_PW=$(cat secrets/grafana_admin)
API=http://127.0.0.1:3000

# The loader is happy.
docker compose logs --since 30s grafana 2>&1 | grep -i provisioning | tail -5
curl -sf -u "admin:$GF_PW" "$API/api/dashboards/uid/lab-overview" \
  | jq '.meta.provisioned'

# The panel is not. This is the reference that never resolves.
curl -sf -u "admin:$GF_PW" "$API/api/dashboards/uid/lab-overview" \
  | jq -r '.dashboard.panels[].datasource.uid' | sort -u

curl -s -o /dev/null -w 'lookup of prom-typo: HTTP %{http_code}\n' \
  -u "admin:$GF_PW" "$API/api/datasources/uid/prom-typo"

provisioned: true, a clean log, and a panel pointing at a data source that returns 404. Nothing in Grafana will ever tell you about this; the only place it becomes visible is a human opening the dashboard.

The check that catches it belongs in your pipeline, and it is four lines:

GF_PW=$(cat secrets/grafana_admin)
API=http://127.0.0.1:3000
KNOWN=$(curl -sf -u "admin:$GF_PW" "$API/api/datasources" | jq -r '.[].uid' | sort -u)

jq -r '.. | .datasource? // empty | .uid? // empty' \
  provisioning/dashboards/json/*.json | sort -u \
  | grep -vxF "$KNOWN" | sed 's/^/UNRESOLVED datasource uid: /'

Restore the good file:

cd "$HOME/grafana-provisioning-lab"
mv overview.json.good provisioning/dashboards/json/overview.json
sleep 15

GF_PW=$(cat secrets/grafana_admin)
curl -sf -u "admin:$GF_PW" \
  http://127.0.0.1:3000/api/dashboards/uid/lab-overview \
  | jq -r '.dashboard.panels[0].datasource.uid'

Task 8: Break it with a save, and with a deletion

First the save. With allowUiUpdates: false the API refuses the write outright rather than accepting it and reverting later — a loud failure, which is the whole point of the flag:

cd "$HOME/grafana-provisioning-lab"
GF_PW=$(cat secrets/grafana_admin)
API=http://127.0.0.1:3000

curl -s -o /tmp/save-attempt.json -w 'save attempt: HTTP %{http_code}\n' \
  -u "admin:$GF_PW" -H 'Content-Type: application/json' \
  -X POST "$API/api/dashboards/db" \
  -d '{"dashboard":{"uid":"lab-overview","title":"Edited in the UI","schemaVersion":39,"panels":[]},"overwrite":true}'
jq -r '.message // .' /tmp/save-attempt.json

# The declared title is unchanged.
curl -sf -u "admin:$GF_PW" "$API/api/dashboards/uid/lab-overview" \
  | jq -r '.dashboard.title'

Now the deletion. disableDeletion: false means the file set is authoritative in both directions: adding a file adds a dashboard, and removing one removes it.

cd "$HOME/grafana-provisioning-lab"
mv provisioning/dashboards/json/overview.json ./overview.json.parked
sleep 15

GF_PW=$(cat secrets/grafana_admin)
curl -s -o /dev/null -w 'dashboard after removal: HTTP %{http_code}\n' \
  -u "admin:$GF_PW" http://127.0.0.1:3000/api/dashboards/uid/lab-overview

# Put it back. It returns because the file is the source of truth.
mv ./overview.json.parked provisioning/dashboards/json/overview.json
sleep 15
curl -sf -u "admin:$GF_PW" \
  http://127.0.0.1:3000/api/dashboards/uid/lab-overview | jq -r '.dashboard.title'

That round trip is the property worth taking away: the dashboard came back without anyone restoring a backup, because the declaration never left the disk.

Task 9: Prove the whole thing rebuilds from an empty database

Data-loss risklab host— Destroys Grafana's database. Everything must come back from the files alone.

$ cd ~/grafana-provisioning-lab && docker compose down && docker volume rm rb-obs-provisioning_grafana-data

cd "$HOME/grafana-provisioning-lab"
docker compose up -d
for i in $(seq 1 30); do
  curl -sf http://127.0.0.1:3000/api/health >/dev/null && break
  sleep 2
done

GF_PW=$(cat secrets/grafana_admin)
API=http://127.0.0.1:3000
curl -sf -u "admin:$GF_PW" "$API/api/datasources" | jq -r '.[].uid'
curl -sf -u "admin:$GF_PW" "$API/api/search?folderUIDs=lab-sre" | jq -r '.[].uid'
curl -sf -u "admin:$GF_PW" "$API/api/dashboards/uid/lab-overview" \
  | jq -r '.meta.provisioned'

If the data source, the folder and the dashboard are all back, the provisioning directory is complete. If anything is missing, it was created by hand at some point in this lab and you have just found the gap — which is the entire reason to run this destructive test on purpose rather than discovering it during a rebuild.

Validation

Two scripts. The first is the provisioning test from the testing lesson: it asserts every declaration landed. The second is the drift check: it asserts nothing exists that the files do not own.

cat > "$HOME/grafana-provisioning-lab/test-provisioning.sh" <<'TEST'
#!/usr/bin/env bash
set -euo pipefail
cd "$HOME/grafana-provisioning-lab"
API=http://127.0.0.1:3000
GF_PW=$(cat secrets/grafana_admin)
fail() { echo "FAIL: $1"; exit 1; }

curl -sf -u "admin:$GF_PW" "$API/api/datasources" \
  | jq -e '.[] | select(.uid == "prom-lab")' >/dev/null \
  || fail "data source prom-lab was not provisioned"

curl -sf -u "admin:$GF_PW" "$API/api/datasources/uid/prom-lab" \
  | jq -e '.readOnly == true' >/dev/null \
  || fail "prom-lab is UI-editable; editable:false did not apply"

curl -sf -u "admin:$GF_PW" "$API/api/datasources/uid/prom-lab/health" \
  | jq -e '.status == "OK" or .status == "success"' >/dev/null \
  || fail "prom-lab is provisioned but cannot answer a query"

curl -sf -u "admin:$GF_PW" "$API/api/dashboards/uid/lab-overview" \
  | jq -e '.meta.provisioned == true' >/dev/null \
  || fail "lab-overview is not file-backed"

# Note the shape: a `while read` inside a pipeline runs in a subshell, so
# an exit from it would not end this script. Collect first, assert after.
KNOWN=$(curl -sf -u "admin:$GF_PW" "$API/api/datasources" | jq -r '.[].uid' | sort -u)
DECLARED=$(jq -r '.. | .datasource? // empty | .uid? // empty' \
  provisioning/dashboards/json/*.json | sort -u)
UNRESOLVED=$(printf '%s\n' "$DECLARED" | grep -vxF "$KNOWN" || true)
[ -z "$UNRESOLVED" ] || fail "panel references unknown datasource(s): $UNRESOLVED"

echo "PASS: every declaration resolved"
TEST

chmod +x "$HOME/grafana-provisioning-lab/test-provisioning.sh"
"$HOME/grafana-provisioning-lab/test-provisioning.sh"

The third assertion is the one worth defending in review. Checking that the data source exists proves the loader ran; checking that it can answer proves the declaration is correct. They fail for different reasons and both belong in the gate.

cat > "$HOME/grafana-provisioning-lab/check-drift.sh" <<'DRIFT'
#!/usr/bin/env bash
set -euo pipefail
cd "$HOME/grafana-provisioning-lab"
API=http://127.0.0.1:3000
GF_PW=$(cat secrets/grafana_admin)
rc=0

for uid in $(curl -sf -u "admin:$GF_PW" "$API/api/search?type=dash-db" | jq -r '.[].uid'); do
  p=$(curl -sf -u "admin:$GF_PW" "$API/api/dashboards/uid/$uid" | jq -r '.meta.provisioned')
  if [ "$p" != "true" ]; then
    echo "DRIFT: dashboard $uid exists but no file declares it"
    rc=1
  fi
done

for uid in $(curl -sf -u "admin:$GF_PW" "$API/api/datasources" | jq -r '.[].uid'); do
  ro=$(curl -sf -u "admin:$GF_PW" "$API/api/datasources/uid/$uid" | jq -r '.readOnly')
  if [ "$ro" != "true" ]; then
    echo "DRIFT: data source $uid is UI-editable"
    rc=1
  fi
done

[ "$rc" -eq 0 ] && echo "No drift detected."
exit "$rc"
DRIFT

chmod +x "$HOME/grafana-provisioning-lab/check-drift.sh"
"$HOME/grafana-provisioning-lab/check-drift.sh"

Expected Outcome

/api/datasources lists exactly prom-lab, with readOnly: true and a URL substituted from the environment.
/api/datasources/uid/prom-lab/health reports a healthy status against a Prometheus that is genuinely answering queries.
/api/dashboards/uid/lab-overview reports provisioned: true and sits in the Lab SRE folder.
A save against that dashboard is refused rather than accepted.
After destroying grafana-data entirely, all three come back unchanged.
test-provisioning.sh and check-drift.sh both exit 0.

Troubleshooting

A file you added is ignored and the log says nothing. The loader reads *.yaml and *.yml only. An editor backup (metrics.yaml.bak), a swap file, or a .yml.tmp from an atomic write is skipped with no message, because from the loader’s point of view it is not a provisioning file. ls -la the directory and look at the extensions, not the names.

A malformed YAML file breaks one resource and nothing else. That is by design: a parse failure is logged at error level and the file is skipped, while every other file in the directory still applies. The result is a half-applied directory that looks like a partially-working Grafana. Search the log for the filename — docker compose logs grafana 2>&1 | grep -i provisioning — before concluding that the loader did not run.

The dashboard loads into the wrong folder. folderUid is the key and folder is the display name. Renaming the folder in the UI changes the label and not the UID, so a provider whose folderUid does not match creates a second folder with the same visible name. Compare jq -r '.meta.folderUid' on the dashboard against the provider file.

schemaVersion too new. A dashboard exported from a newer Grafana than the one loading it is dropped with a log line naming the version. The fix is to export from a Grafana on the target version, not to hand-edit the number down — the number describes the JSON model, and lowering it does not change the model.

The health check reports an error but Prometheus is fine. Check the URL Grafana holds, not the one you wrote: curl -sf -u admin:PW /api/datasources/uid/prom-lab | jq -r '.url'. The substitution happens at load time, so the file and the live value can differ whenever the environment changed after the last reload.

The dashboard reverts to an older version after a docker compose up. The file is authoritative and the file is what you edited last. If the change you expect is missing, you edited the copy in the container’s read-only mount path in your head and the copy on the host in reality — or the other way round. The host path is the one that matters; the mount is :ro precisely so that this ambiguity cannot exist.

Cleanup

cd "$HOME/grafana-provisioning-lab"
docker compose config --volumes
docker compose down -v

docker compose ps -a
docker volume ls | grep rb-obs-provisioning || echo "no lab volumes remain"

cd "$HOME"
rm -rf "$HOME/grafana-provisioning-lab"

Remove the images with docker image rm grafana/grafana:11.3.0 prom/prometheus:v2.55.1 if you want the disk back.

What You Learned

The database is downstream of the files. You destroyed grafana-data and the data source, the folder and the dashboard all returned. That property is what makes a Grafana a deployable rather than an artefact, and it is testable in one command.
The loaders are forgiving by design, and that is the risk. An unset variable becomes an empty string. A panel UID is stored without being resolved. A malformed file is skipped so the others can apply. Every one of those choices keeps a single mistake from taking the whole instance down, and every one of them turns a mistake into something you have to look for.
provisioned: true means “a file declared this”, not “this works”. In Task 7 the flag was true while the panel pointed at nothing. The flag answers a question about provenance; the health check and the UID cross-check answer the question about correctness.
A loud rejection beats a quiet revert. allowUiUpdates: false refuses the save. The operator learns immediately that this dashboard is code. The alternative — accept, then silently revert 60 seconds later — teaches them that Grafana is unreliable.
disableDeletion: false makes the file set authoritative in both directions. Adding a file adds a dashboard and removing one removes it. That is usually what you want and it means a misdirected git mv is a deletion.

Taking this to a real change window

Provisioning changes are among the lowest-risk changes you will make to a Grafana and among the easiest to make badly. The change window has three stages, and the middle one is the one people skip.

Stage one, in CI on every pull request: bring up a throwaway Grafana with the proposed provisioning directory and run test-provisioning.sh against it. This is the whole point of the lab’s sandbox shape — the loader is the contract, and YAML syntax checking does not exercise the loader.

Stage two, at apply time: reload rather than restart (/api/admin/provisioning/dashboards/reload), then re-run the data source health checks. A reload is seconds and does not drop sessions.

Stage three, continuously: run check-drift.sh on a schedule. Its job is not to catch your change, it is to catch the dashboard somebody created in the UI during an incident three weeks from now — the one that works, that nobody declared, and that will vanish the next time the instance is rebuilt.

Objectives

Prerequisites