Skip to main content
RunBook Academy

← All checklists in Observability

Before deploymentgrafana-readiness

Grafana Production Readiness

23 items ·16 critical ·7 warn ·0 info

Run this before Grafana carries production traffic, and again after any change to the database, the identity provider, the provisioning tree, or the set of people who hold an administrator role. Those four are where the risk lives; the rest of the configuration can drift without anybody getting hurt.

The list is deliberately ordered as storage, then identity, then permissions, then provisioning, then recovery — because each layer only means something if the one before it holds. Folder permissions are decoration on an install where sign-up is open, and a backup is decoration on an install whose dashboards were never written to a file in the first place.

What “ready” means here

Two things, and neither is “the login page loads”. First, that a lost host is a rebuild rather than an archaeology project: the provisioning tree plus a verified database dump reconstruct the install. Second, that the answer to “who can see this, and who can change it” is a configuration somebody wrote down rather than an accident of defaults.

Access this needs

Read access to the Grafana configuration files and provisioning tree on the host, plus a service-account token with administrative read for the API items. Substitute your own Grafana address, provisioning path and repository location; the paths below are the conventional ones. Nothing here writes. The direct header-authentication probe is the one item that must be run from somewhere other than the proxy, because running it through the proxy tests the wrong path.

Sign-off

  • Reviewer: ________________ Date: ___________
  • Platform owner: ___________ Date: ___________
  • Security owner: ___________ Date: ___________

Critical16 items

  1. awk '/^\[database\]/{f=1;next} /^\[/{f=0} f && /^(type|host|name|user)/' /etc/grafana/grafana.ini
  2. sqlite3 /var/backups/grafana/latest.sqlite "PRAGMA integrity_check;"; sqlite3 /var/backups/grafana/latest.sqlite "SELECT count(*) FROM dashboard;"
  3. diff -r /etc/grafana/provisioning /srv/observability/grafana-provisioning
  4. curl -sf -H "Authorization: Bearer $GF_SA_TOKEN" http://grafana:3000/api/datasources | jq -r '.[] | "\(.uid) \(.type) \(.url) readOnly=\(.readOnly)"'
  5. grep -L '"uid"' /srv/observability/grafana-provisioning/dashboards/json/*.json
  6. awk '/^\[auth.anonymous\]/{f=1;next} /^\[/{f=0} f && /enabled/' /etc/grafana/grafana.ini
  7. grep -nE '^(allow_sign_up|auto_assign_org_role|allow_org_create)' /etc/grafana/grafana.ini
  8. awk '/^\[auth\./{s=$0} /^enabled[[:space:]]*=[[:space:]]*true/{print s}' /etc/grafana/grafana.ini
  9. grep -nE '^(allowed_domains|allowed_organizations|allowed_groups|org_mapping|search_base_dns)' /etc/grafana/grafana.ini /etc/grafana/ldap.toml
  10. grep -nE 'team_sync|groups_attribute_path|role_attribute_path' /etc/grafana/grafana.ini
  11. awk '/^\[auth.proxy\]/{f=1;next} /^\[/{f=0} f' /etc/grafana/grafana.ini; curl -s -o /dev/null -w "%{http_code}\n" -H "X-WEBAUTH-USER: root" http://grafana.internal:3000/api/org
  12. grep -nE '^(protocol|cert_file|cookie_secure|cookie_samesite|strict_transport_security|tls_skip_verify)' /etc/grafana/grafana.ini
  13. grep '^secret_key' /etc/grafana/grafana.ini | awk -F= '{print length($2)}'
  14. for f in $(curl -sf -H "Authorization: Bearer $GF_SA_TOKEN" http://grafana:3000/api/folders | jq -r '.[].uid'); do echo "== $f"; curl -sf -H "Authorization: Bearer $GF_SA_TOKEN" "http://grafana:3000/api/folders/$f/permissions" | jq -r '.[] | "\(.team // .userLogin // .role) \(.permissionName)"'; done
  15. curl -sf http://grafana:3000/api/health; for u in $(curl -sf -H "Authorization: Bearer $GF_SA_TOKEN" http://grafana:3000/api/datasources | jq -r '.[].uid'); do printf "%s " "$u"; curl -sf -H "Authorization: Bearer $GF_SA_TOKEN" "http://grafana:3000/api/datasources/uid/$u/health" | jq -r '.status'; done
  16. cat /var/backups/grafana/last-drill.txt

Warning7 items

  1. grep -nE 'disableDeletion|allowUiUpdates|updateIntervalSeconds|foldersFromFilesStructure' /etc/grafana/provisioning/dashboards/*.y*ml
  2. grep -nE 'login_maximum_lifetime_duration|login_maximum_inactive_lifetime_duration' /etc/grafana/grafana.ini
  3. grep -nE 'allow_assign_grafana_admin' /etc/grafana/grafana.ini; curl -sf -H "Authorization: Bearer $GF_SA_TOKEN" http://grafana:3000/api/admin/users | jq -r '.[] | select(.isAdmin == true) | .login'
  4. grep -rnE '(password|basicAuthPassword|token|secureJsonData)' /srv/observability/grafana-provisioning/datasources/
  5. ls -la /etc/grafana/provisioning/alerting/ 2>/dev/null; grep -nE '^(enabled|execute_alerts)' /etc/grafana/grafana.ini
  6. grafana-cli plugins ls