Panels and Queries

A panel shows a flat line at zero. The on-call engineer suspects the service is silent. They open the query inspector and find that the backend returned a single series with a five-minute step. The flat line is not “no work” — it is “stepped so coarsely the trade-shaped curve looks flat”. A different panel would have made that visible.

This lesson is about the two halves of a panel: the query that fetches a frame, and the visualisation that turns the frame into pixels. The two can fail independently. Knowing which is failing is the diagnostic difference between a five-minute fix and a fifty-minute detour.

What a panel is

A panel is a single unit of visualisation on a dashboard. It owns a title, an optional description with dynamic fields, an array of target queries (each with a refId such as A, B, C …), a panel-type-specific configuration, a list of transform steps, and a list of alert conditions.

The five panel types in Grafana 11 that an operator needs on day one:

+----------------+---------------------------------------------------+
| Panel type     |  When to use                                       |
+----------------+---------------------------------------------------+
|  Timeseries    |  How X has changed over time. Default for         |
|                |  Prometheus, Loki rate, Tempo service-map.        |
+----------------+---------------------------------------------------+
|  Stat          |  X "right now" plus a sparkline of recent          |
|                |  values. Default for SLI panels and KPIs.         |
+----------------+---------------------------------------------------+
|  Bar gauge     |  X for each of N buckets compared against a       |
|                |  threshold. Default for SLO error-budget gauges.   |
+----------------+---------------------------------------------------+
|  Table         |  X per row, ranked or sorted. Default for         |
|                |  log streams, top-N panels, and audit lists.      |
+----------------+---------------------------------------------------+
|  Logs (panel)  |  The Loki-native log list view, with extracted    |
|                |  fields, level filter, and a wrap link to the      |
|                |  trace whose span the log line came from.          |
+----------------+---------------------------------------------------+
|  Text          |  Static text or markdown; no query. Use it for     |
|                |  runbook excerpts and SLO definitions.             |
+----------------+---------------------------------------------------+

The list is wider than this — heatmaps, candlesticks, gauges, histograms, geomaps — but those six cover roughly 95% of the panels in a typical production-stack dashboard.

Why a sysadmin cares

Panels fail in three distinct shapes. Each shape has a different diagnostic.

1. Empty panel, no error. The query returned zero series. The most common cause is a label mismatch between the panel query and the actual data — usually a recent rename of the instance or job label.
2. Slow panel. The query returned the right shape but the panel takes ten seconds to render. The cause is usually a transformation that is fine for ten series and untenable for ten thousand.
3. Wrong-looking panel. The query returns data; the visualisation renders it; the chart looks wrong. The cause is a panel-options choice that does not match the metric’s units (an axis in seconds when the values are in milliseconds) or a transform that hides the dimension the operator wanted.

Knowing the panel type and the visualisation options is what turns “the chart is weird” into “set the unit to milliseconds and the panel is correct”.

How it works

A panel’s life cycle on a refresh runs through five stages:

+----------+    +--------------+    +----------------+    +-------------+
| refresh  |--> |  queries     |--> |  transform     |--> |  render     |
| trigger  |    |  (one per    |    |  (rename,      |    |  pixels     |
| (timer,  |    |   refId)     |    |   join,        |    |             |
|  full    |    |              |    |   filter,      |    |             |
|  reload) |    +--------------+    |   calculate)   |    +-------------+
+----------+          |             +----------------+           |
                       |                   |                     |
                       v                   v                     v
                data source         pipeline of frames      SVG / canvas
                plugin call        (each step typed)       into DOM

The queries stage sends each refId’s target to the relevant data-source plugin, which returns a frame. The transform stage is a pipeline where each step rewrites the frames in place — Rename by regex, Filter by value, Merge, Group by, and the more expensive Join and Reduce operations. The render stage is type-specific: a timeseries panel builds an SVG / canvas of line paths; a table panel renders DOM rows; a logs panel streams lines in as Loki returns them.

The query inspector exposes the boundary between stages. It shows the raw frame returned by the data source, the frame after each transform, and the rendered configuration options. Use it every time a panel looks wrong.

How to configure it

Below is a portable, idempotent JSON for a single timeseries panel that asks Prometheus for up per scrape job and renders a rate over 5 minutes. The targets array is the query; the fieldConfig and options blocks are the visualisation.

{
  "type":       "timeseries",
  "title":      "Scrape job up",
  "description": "up == 1 means Prometheus is scraping it",
  "datasource": { "type": "prometheus", "uid": "prom-prod" },
  "gridPos":    { "x": 0, "y": 0, "w": 12, "h": 8 },
  "targets": [
    {
      "refId":      "A",
      "datasource": { "type": "prometheus", "uid": "prom-prod" },
      "expr":       "up{job=~\"$job\"}",
      "legendFormat": "{{instance}}",
      "interval":   "",
      "maxDataPoints": 800
    }
  ],
  "fieldConfig": {
    "defaults": {
      "unit":      "short",
      "min":       0,
      "max":       1,
      "decimals":  0,
      "custom": {
        "drawStyle":     "line",
        "lineWidth":     1,
        "fillOpacity":   10,
        "showPoints":    "never",
        "spanNulls":     false,
        "gradientMode":  "none"
      }
    }
  },
  "options": {
    "tooltip":  { "mode": "multi", "sort": "none" },
    "legend":   { "showLegend": true, "displayMode": "list",
                   "placement": "bottom", "calcs": ["mean", "last"] }
  }
}

A few shapes worth noting in this JSON:

• targets is an array. Adding another target with refId: "B" produces a second query per refresh. The visualisation overlays the frames.
• fieldConfig.defaults.unit is the unit string. short is dimensionless; s, ms, bytes, percent, and reqps render correctly. Setting this to none defeats axis labels and is the source of “the axis is wrong” tickets.
• maxDataPoints is the cap on points returned; the data source may downsample to fit. This is the single lever an operator holds to make a slow panel fast.
• legendFormat may use $variable for template variables and the usual Prometheus label-placeholder syntax.

For a transformation example (a panel-side rename that turns value_bytes into value):

{
  "type": "table",
  "title": "Top 10 instances by write bytes",
  "datasource": { "type": "prometheus", "uid": "prom-prod" },
  "targets": [
    { "refId": "A",
      "expr":  "topk(10, sum by (instance) (rate(node_disk_bytes_written[5m])))" }
  ],
  "transformations": [
    { "id": "organize",
      "options": {
        "indexByName": { "instance": 0, "Value": 1 },
        "renameByName": { "Value": "Bytes / sec" }
      }
    }
  ]
}

How to validate it

Two checks: one on the wire, one inside the panel.

Severity: READ-ONLY.

# 1. Run the same query Grafana would run, from the panel
#    inspector's "Query" tab. Copy it from the inspector's
#    "Show query in Prometheus" link or run:
curl -G -sf http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=up{job=~"node"}' \
  --data-urlencode 'time='$(date +%s) | jq '.data.result | length'
# 8

# 2. Inside the panel: open the inspector (Inspect > Data)
#    and confirm:
#    - Data source: prom-prod (the UID)
#    - Status:     Success
#    - Time:       <single-digit ms typical for local Prometheus>
#    - Frames:     one or more rows with typed columns
#    - Transforms: each step produced the expected rows and
#      the expected column renames

The Time value inside the inspector is the round trip to the backend plus the data-source plugin’s frame conversion. It does not include transform or render. A Time over one second is usually the backend; a fast Time but a slow panel is the transforms or the render.

How it can fail

Six high-frequency failure shapes; each maps to a recognisable symptom.

1. Empty panel, no error. Query runs, zero series match. Most often a label-mismatch: the instance or job selector was renamed upstream. Open the inspector to confirm Series returned: 0 and Status: Success.
2. Loud “Data source not found”. The datasource.uid on the panel (or on the target) does not exist in /api/datasources. Often caused by a UID rename during a provisioning migration.
3. Stepped line that hides changes. The query step is too large. Open the inspector’s “Stats” tab and check Total datapoints against maxDataPoints — if equal, the data source was capped and downsampled.
4. Wrong axis unit. A duration plotted as bytes, or a rate plotted as none. Fix is to set fieldConfig.defaults .unit. The inspector cannot diagnose this; only the human eyeball catches it.
5. Missing log fields. A Logs panel showing only the message line with no levels / labels. The data source query returned parsed fields but the panel’s displayedFields was not extended to include them.
6. Render-blocking transform. A Join or Extract fields transform applied across a hundred thousand rows. Sympton: a five-second Time and an additional ten-second browser hang. Split the data source query to do the heavy work in Prometheus / Loki, not in the panel.

How to troubleshoot it

The order is the same every time:

1. Inspect the panel. Inspect > Data. Read the Status field. Success means the panel can talk; the work is below this layer.
2. Check Series returned. Zero means the query is well-formed but the labels do not match. Open the data source’s Explore tab, run the same expression without the {} selector, and confirm the series names. Match them.
3. Read Time. Slow time is the data source. Inspect the underlying Prometheus / Loki / Tempo backend and follow its own diagnostics (slow queries, query budget exhaustion).
4. Look at the transforms. Each transform’s row count is in the inspector. A transform that drops to zero rows is the culprit. Disable transforms one at a time to find it.
5. Switch to a Stat panel as a smoke test. If the panel works as a Stat with refId: A only, the visualisation options are the problem. If the Stat panel also shows nothing, the query is the problem.
6. Try the same query in Explore. Explore uses the same data source and the same frame format but bypasses the transform and render stages. A panel that fails but Explore succeeds has a transform or visualisation problem, not a data source problem.

Security implications

• Expressions are evaluated by the backend. A malicious dashboard author can write a PromQL / LogQL expression that consumes backend resources without consuming panel resources. The mitigations are panel-options caps (maxDataPoints, maxLines, queryTimeout) and data-source quotas. Per-user quotas are not in Grafana itself; they live in the backends.
• Templating variables are interpolated server-side by the plugin. A user-controlled value in a template variable is safely escaped before being placed into PromQL or LogQL. This is not the case for __value or value strings in some downstream scripts; verify before pasting an expression.
• Sharing a panel URL exposes nothing private. The panel URL embeds the dashboard UID and time range but not the data source credentials or the raw data.

Performance implications

• A panel refresh makes N queries where N is the number of refId targets plus the number of multi-value template variable combinations. A dashboard with five variables of ten values each and three panels is 150 backend requests per refresh.
• maxDataPoints is the cap. Default null on the panel request is “use the panel pixel width as the cap”, which is almost always what you want.
• Heavy transforms (Join, Reduce by labels, partition by label) belong on the panel’s primary refId, not on a secondary result; some transforms duplicate frames and produce O(N^2) work on long ranges.
• The image rendering service helps in PDF / PNG exports, not in interactive refreshes. Interactive refreshes are bound by the browser’s main thread and the SVG paths produced.

Production guidance

• Adopt a labelling convention for refIds across the team: A for the primary query, B for overlays, C for the calculation that produced a derived metric. Document the convention in your team’s dashboard style guide.
• Set fieldConfig.defaults.unit on every timeseries panel. The shortest path to a confusing dashboard is a mix of axes with and without units.
• Use Transformations > Filter by value to cut empty series off the legend. Render-time overhead stays bounded even with many series; the legend is the bottleneck.
• Keep templating variables at the dashboard level, not the panel level. A panel-scoped variable repeats the same query for every panel that uses it.
• For high-cardinality panels, prefer a query-side topk / bottomk to a panel-side “limit 20” — the data source does the work it is good at, and the panel renders faster.

Verification

You should now be able to answer:

• What does each of timeseries, stat, table, bar gauge, logs, and text panels answer that the others do not?
• Where in a panel’s life cycle does a transform step run?
• What does the inspector’s Time field measure, and what does it deliberately exclude?
• Why is maxDataPoints the right knob for a slow but correct panel?
• How do you tell a query bug from a visualisation bug?

Quiz