Grafana Anatomy

A user opens Grafana at 09:00. Three dashboard tabs load. Each panel renders in under a second. To the user, Grafana is “the dashboard page”. To the operator, Grafana is a single Go binary holding five sub-systems, a database, and an unbounded set of plugins, any one of which can be the reason a panel is slow or empty.

This lesson opens the hood. The goal is not to teach every option key. The goal is to name the pieces so that the next three lessons (datasources, panels, provisioning) attach to a known shape.

What Grafana is

Grafana is a single-server application written in Go. It serves a single-page web UI, accepts arbitrary data-source plugins over a stable plugin protocol, queries those plugins on the operator’s behalf, renders the returned frames into panels, and persists dashboards, alert rules, users, and audit data in a SQL database.

Grafana is not a data store. It does not own Prometheus metrics, Loki logs, or Tempo traces. It owns the views of those things. This distinction is the single most common source of operational confusion: a missing dashboard is almost never a Grafana outage; it is almost always a missing or unreachable data source.

Why a sysadmin cares

Three operational pain points disappear once you understand the anatomy.

1. “Grafana is broken” tickets. Most arrive because a panel shows no data, not because the web UI failed. Knowing where the failure can live (HTTP, plugin, proxy, storage) shortens the diagnostic from hours to minutes.
2. Capacity arguments. Disk used by the grafana.db file. CPU spent in plugin health checks. Memory held by panel render cache. These three numbers answer the “how big should this host be” question honestly, which means knowing which subsystem owns each one.
3. Upgrade planning. Storage migrations, plugin-breaking changes, and licence-mode switches each map to a specific subsystem. A targeted upgrade plan starts there.

How it works

The mental model has five layers. Each layer has its own configuration, its own failure modes, and its own monitoring signal.

+-----------------------------------------------------------------+
|                       Web UI (React SPA)                         |
|   Dashboards  |  Explore  |  Alerting  |  Connections  |  Admin  |
+-----------------------------------------------------------------+
                              |
                              v
+-----------------------------------------------------------------+
|                    HTTP server (Go net/http)                    |
|    /api/health   /api/datasources   /api/folders   /api/...     |
+-----------------------------------------------------------------+
                              |
        +---------------------+----------------------+
        |                     |                      |
        v                     v                      v
+---------------+   +-------------------+   +------------------+
| Query engine  |   | Alert scheduler   |   | Provisioning     |
| (frontend     |   | (rule evaluator,  |   | loader (yaml     |
|  data frames) |   |  contact points,  |   | polling on       |
|               |   |  silences)        |   |  /etc/grafana)   |
+---------------+   +-------------------+   +------------------+
        |
        v
+-----------------------------------------------------------------+
|                Plugin runtime  (backend + frontend)             |
|   prometheus  | loki | tempo | mysql | postgres | influx | ... |
+-----------------------------------------------------------------+
                              |
                              v
+-----------------------------------------------------------------+
|                        Storage layer                            |
|       sqlite (file)  |  postgres  |  mysql                      |
|       dashboards, users, orgs, alert rules, audit log            |
+-----------------------------------------------------------------+

The query engine is the part operators interact with most. It receives a query, dispatches it to the relevant data-source plugin (over the plugin’s QueryData method), receives a data frame back (a columnar, typed structure that has replaced the older TimeSeries type in Grafana 11), and applies the panel’s transformations before the rendering pipeline turns it into pixels.

How to configure the server

The Grafana configuration block below is a realistic starting point for a small production deployment. Everything outside the [paths], [database], and [security] sections has sensible defaults; changing them without reason is how incidents start.

# /etc/grafana/grafana.ini
[paths]
data    = /var/lib/grafana
logs    = /var/log/grafana
plugins = /var/lib/grafana/plugins
provisioning = /etc/grafana/provisioning

[server]
http_port   = 3000
domain      = grafana.internal.example
enforce_domain = true
root_url    = https://grafana.internal.example/

[security]
admin_user     = grafana-admin
secret_key     = a-long-random-string-from-secrets-manager
disable_gravatar = true
cookie_secure  = true
cookie_samesite = lax

[database]
# sqlite is the default. Postgres / MySQL are documented options
# for HA. The url format follows the underlying sql driver.
type = sqlite3
path = grafana.db

[users]
allow_sign_up = false
auto_assign_org_role = Viewer

[log]
level = info
mode  = console

# Mandatory for alerting.
[unified_alerting]
enabled = true

# Image rendering is a separate container in production.
[rendering]
server_url = http://grafana-image-renderer:8081/render
callbacks  = http://grafana:3000/

For a high-availability pair, swap [database] for a managed Postgres and add a [session] block pointing at Redis. The [database] and [session] blocks are the only two that must match across all nodes in the cluster; everything else can drift freely without harm.

How to validate it

A fresh Grafana should pass three checks in order. The first proves the process is up. The second proves the database is reachable. The third proves the API is answering authenticated requests.

Severity: READ-ONLY. All commands below inspect; none change state.

# 1. Process is up and listening.
curl -sf -o /dev/null -w "%{http_code}\n" http://grafana:3000/login
# 3000

# 2. Health endpoint distinguishes "up" from "ready".
curl -s http://grafana:3000/api/health | jq

{
  "version": "11.1.0",
  "commit": "abc1234",
  "database": "ok",
  "datasources": {
    "Prometheus": {
      "status": 1,
      "message": "Data source is working"
    }
  },
  "unifiedAlerting": {
    "status": 1,
    "message": "ok"
  }
}

The four sub-statuses that matter:

• database must be "ok". Anything else means Grafana cannot read its own state.
• each data source’s status must be 1. 0 is unreachable.
• unifiedAlerting.status must be 1 if alerting is enabled.
• version is the running build — confirm it matches the one deployed.

# 3. Authenticated API answers. Use a service account token in
#    real use; the basic-auth form is for diagnosis only.
curl -sf -u "grafana-admin:$GF_ADMIN_PASSWORD" \
  http://grafana:3000/api/org | jq

{
  "id": 1,
  "name": "Main Org.",
  "address": { "address1": "", "address2": "", "city": "", ... },
  "created": "2026-08-01T00:00:00Z"
}

A 401 here against a known-good credential indicates the cookie / token signing key (secret_key) has changed between the previous session and now — sessions are invalidated and must be re-issued.

How it can fail

The five most common failure shapes, with their observable symptoms:

1. Database is not reachable. Symptom: curl http://grafana:3000/api/health returns "database": "ok" or "database": "failing". The login page renders but /api/* returns 500. Logs contain database is not healthy.
2. Data source is unreachable but Grafana is fine. Symptom: api/health returns "database": "ok" and the data source’s status is 0. Panels display “No data” or the red triangle state. The login page, dashboards listing, and admin pages all load — only panels are broken. This is the operationally dangerous case: the page loads, the user clicks a panel, and nothing useful is shown.
3. Plugin is missing. Symptom: dashboard panel shows “Plugin not found” or the panel error of an unspecified kind. Logged at start: Failed to load plugin: <id>. Caused by removing the plugin directory or by a version mismatch between Grafana and the plugin manifest.
4. secret_key rotated unintentionally. Symptom: every authenticated session returns 401. The OIDC and LDAP sign-in paths break. Operators are signed out and cannot sign back in if the IdP flow uses the same key. This is the failure caused by redeploying with a fresh secret_key rather than restoring it from secrets management.
5. Disk full on /var/lib/grafana. Symptom: write paths (dashboard save, alerting silences, audit log writes) return 500. Reads continue to work. The sqlite file may corrupt on a hard restart. Diagnose with du -sh /var/lib/grafana/*.
6. Image render service is unreachable. Symptom: PDF / PNG exports hang or return 502. The dashboard itself renders fine in the browser. Check the rendering_service env var points at a reachable sibling.

How to troubleshoot it

The diagnostic order is reproducible across the failure shapes above:

1. Process up? systemctl status grafana-server (or docker ps for a container). If not up, the host or container runtime is the fault. Read stdout/stderr. Restart only after the cause is found.
2. API up? curl -sf http://grafana:3000/api/health. Returns 200 for “process is up”. Read the JSON body for database, unifiedAlerting, and per-data-source status. This single call resolves three of the six common failures above.
3. Database reachable? database: ok from the call above. If not, reach the database directly. For sqlite: sqlite3 /var/lib/grafana/grafana.db ".schema". For Postgres: psql -h $PGHOST -U grafana -c "select 1".
4. Plugin load? grafana-cli plugins ls (binary) or ls /var/lib/grafana/plugins/ (container). For each plugin, inspect the manifest at <plugin-dir>/<plugin>/plugin.json and confirm version matches the runtime requirement of Grafana 11.
5. Disk pressure? du -sh /var/lib/grafana/* then df -h /var/lib/grafana. The png directory is the render cache; it is safe to truncate when full.
6. Network to data sources? From the Grafana host, curl -sf -o /dev/null -w "%\{http_code\}\n" http://prometheus:9090/-/ready. If this fails the proxy is the source. If it succeeds, the data source plugin is the source.

Security implications

Grafana’s attack surface is the HTTP listener. Treat it as an internet-facing service. The relevant controls:

• secret_key must be 32+ random bytes, stored in a secrets manager, and never randomised between restarts. Rotating it silently invalidates every session and breaks SSO token decryption.
• admin_user and admin_password must be set from environment variables, not literals, on first start; afterwards the admin_password line in grafana.ini is ineffective. Reset through the API or the grafana-cli admin reset-admin-password command.
• Authentication must be brought forward of any internet ingress — auth.proxy or an OIDC provider in front of Grafana is the correct control; basic_auth against the public listener is not.
• Data source secureJsonData must not contain long-lived credentials for sources that support better auth. For Prometheus, basic auth with a read-only token is appropriate. For Postgres, a read-only data-source user is appropriate. For Loki with multi-tenant mode, the X-Scope-OrgID header must be set in the data source configuration rather than carried in the URL.
• disable_gravatar = true and cookie_samesite = lax are small defaults worth setting.

The full security posture of Grafana is out of scope for this lesson and lives in Part XXII.

Performance implications

Three limits dominate in production:

• Render cache memory. Each rendered PNG for a heavy dashboard can be a few hundred KiB. Bound the cache size; rotate on schedule. The default cache_size of 100 is a starting point, not a guarantee.
• Alert scheduler CPU. Every rule evaluates every interval. A thousand rules at evaluation_interval = 60s is roughly one evaluation every 60 ms across the population. This is cheap on modern hardware but the rule count is the lever an operator holds.
• Provision polling. Each provisioning path polls its directory at a configured interval. The default is 60 seconds; this is harmless on disk-resident YAML but can become the bottleneck if the path is a slow remote mount.

Scale out before scale up. A second Grafana node looking at the same Postgres handles three times the load for a quarter of the hardware cost.

Production guidance

• Pin the Grafana version and the plugin versions in the same release pipeline. Plugin and Grafana version mismatches are the second-most-common reason dashboards fail to load.
• Set secret_key from a secrets manager and read it on every start, not at provisioning time. Helm and Compose both have the right primitives for this.
• Treat /api/health as a load-balancer health check only when database: ok is the response. Don’t fall back to the login page as a health check; it lies.
• Provision data sources and dashboards from the filesystem, not the UI, as soon as the team has more than one operator. File-based provisioning is the topic of Part XXIII lesson 6.
• Set the image_renderer_url to a sibling service and not the in-process queue once a single Grafana host handles more than ~20 dashboards in a PDF / PNG cycle.

Verification

You should now be able to answer:

• Where is the Grafana database on a default container install, and what does it own?
• What does a healthy /api/health return and how do you read it?
• Why is “the panel shows no data” almost never a Grafana outage?
• Which configuration knob gates whether a second Grafana node can join the first as a peer?
• Where in the file system does file-based provisioning live?

Quiz