Label Passing Through the Pivot

A metric panel shows a series labelled service="checkout-svc", route="/v2/cart", status="500". The data link’s URL template renders {service="${__series.labels.service}", route="${__series.labels.route}", status="${__series.labels.status}"}. The operator clicks the line and lands in Loki with a query that matches three labels. The stream is small, scoped to the route that spiked, and returns the lines that explain the metric.

The same panel, the same click, a different team. Their URL template renders {job="${__series.labels.service}", level="${__series.labels.route}"}. The substitution happens, but the resulting LogQL is wrong in two ways: the metric label is service while the Loki stream label is job, and the metric’s route is being substituted into a Loki filter for log level. The click returns “no results”. The on-call concludes the service is silent.

The pivot lives or dies by which labels pass and how they land. Pass the right labels, and the pivot is fast and focused. Pass the wrong labels, and the pivot is a UI that silently misdirects.

What it is

Label passing is the act of substituting a value from the metric panel’s data model into the destination query’s parameter set. In the dashboard-to-logs workflow, the substitution happens in the data link’s URL template at click time.

The Grafana 11.x template syntax for label substitution is the dollar-brace form, with a discriminator prefix that identifies the source:

The five __-prefixed forms are data-link variables; they are resolved by Grafana from the panel’s data model. The last form is a dashboard template variable; it is resolved from the dashboard’s variable state. The two namespaces do not collide because data-link variables begin with __ and dashboard variables conventionally do not.

For a metric-panel-to-Loki pivot, the right choice is almost always ${__series.labels.NAME} (Grafana syntax: ${__series.labels.<labelname>}). The metric series carries the label; the Loki stream selector filters on the matching label value; the substitution is the bridge.

Why a sysadmin cares

The pivot is the bridge. The bridge either carries the right labels or carries wrong ones or carries nothing. Four production shapes appear when the label passing is sloppy:

• Empty substitution. The metric panel exposes service_name after a label rename; the URL template references ${__series.labels.service}. The substitution is empty; the LogQL becomes {service="", route="..."}; the query returns no lines. The engineer concludes the service is silent.
• Wrong label class. The metric series carries request_id (high-cardinality per-request identifier); the URL template passes it through to the Loki query. The resulting stream selector \{service="x", request_id="abc123"\} is correct in shape but produces a stream of one line — the line for that one request — at the cost of a Loki query that scans every chunk for that service. The pivot is correct but the cardinality kills Loki.
• Label-name drift. The metric legend uses http_route; the Loki stream label is route. The substitution is right; the LogQL is {http_route="${service}"} against a stream that does not have that label. The query returns nothing.
• PII leakage. The metric series carries customer_email because a developer added a label for debugging six months ago. The pivot passes the value through to the Loki query. The URL lands in the access log; the destination data source sees a customer identifier; the security team opens a ticket.

How it works

The mental model is “the metric label and the Loki stream label are two names for the same thing”. The pivot’s job is to bridge the two. The bridge has three parts: the metric label name, the substitution template, and the Loki stream label name.

   metric panel                URL template               Loki stream selector
   -------------                ------------               ---------------------
   series:                       ${__series.labels.X}
     service: "checkout-svc"  ---                      -->  {service="checkout-svc"}
     route:   "/v2/cart"       ---                      -->  {..., route="/v2/cart"}
     status:  "500"            ---                      -->  {..., status="500"}

Three observations:

1. The metric label name and the Loki stream label name must agree. If the metric exposes handler and the Loki stream has route, the substitution is empty or the LogQL references a label that does not exist on the log side. The right discipline is to align the names at the source — the Prometheus relabel config and the Alloy pipeline both — so the bridge has nothing to map.
2. The substitution happens at click time, not at render time. The URL template is rendered per-click, not per-panel-render. The same panel can render ten thousand labels; the URL is computed when the operator clicks, against the specific series they clicked.
3. The substitution is string-substitution, not type-checked. Grafana does not validate that the substituted value is a valid LogQL label value. A label that contains a literal " produces a LogQL that fails to parse. The right discipline is to constrain metric label values to a known-safe character set.

How to configure it

The configuration is two-fold: align the metric label name and the Loki stream label name at the source, and author the URL template to substitute the right labels.

# /etc/prometheus/prometheus.yml
# The relabel config that drops the original `handler` label
# and adds the canonical `route` label that Loki uses.
metric_relabel_configs:
  - source_labels: [handler]
    target_label: route
    action: labelmap
  - source_labels: [route]
    regex: '(.+)'
    target_label: route
    action: replace

# /etc/alloy/config.alloy
# The Alloy pipeline that extracts the same `route` label
# from the structured log and stamps it on the Loki stream.
loki.source.kubernetes "pods" {
  // ... kubernetes discovery ...
}

loki.process "pods" {
  stage.json {
    expressions = {
      route = "route",
    }
  }
  stage.labels {
    values = {
      route = "",
    }
  }
}

{
  "type": "timeseries",
  "title": "5xx rate by service and route",
  "datasource": { "type": "prometheus", "uid": "prom-prod-us" },
  "targets": [
    {
      "expr": "sum by(service, route) (rate(http_requests_total{status=~\"5..\"}[5m]))",
      "refId": "A"
    }
  ],
  "fieldConfig": {
    "defaults": {
      "links": [
        {
          "title": "Logs: ${__series.labels.service} ${__series.labels.route}",
          "url": "/explore?schemaVersion=1&panes=%7B%22logs%22%3A%7B%22datasource%22%3A%22loki-prod-us%22%2C%22queries%22%3A%5B%7B%22expr%22%3A%22%7Bservice%3D%5C%22%24%7B__series.labels.service%7D%5C%22%2Croute%3D%5C%22%24%7B__series.labels.route%7D%5C%22%2Cstatus%3D%5C%225%5C%22%7D%22%7D%5D%7D%7D&from=${__value.time:date-seconds}-120&to=${__value.time:date-seconds}+120",
          "targetBlank": true,
          "includeVars": true
        }
      ]
    }
  }
}

The relevant choices, walked through:

• Aligning label names at the source. The Prometheus metric_relabel_configs block rewrites the handler label to route. The Alloy pipeline extracts the route field from the structured log and stamps it on the Loki stream. Both sides now agree on the name route. The URL template can reference ${__series.labels.route} and the substitution lands on a Loki stream label that exists.
• Passing the service label. ${__series.labels.service} resolves to the value of the metric label service. The Loki stream selector {service="..."} filters on the matching Loki stream label. The bridge carries one value.
• Passing the route label. Same pattern; same discipline. The route label is low-cardinality (a few hundred distinct values across the catalogue), safe to pass, and the right filter for “this specific endpoint”.
• Passing the status label. ${__series.labels.status} resolves to 500 (or 502, 503, 504). The Loki stream selector adds status="5" (the regex match for 5xx) so the pivot returns only error lines. The alternative is to pass the exact status code, which is also valid.
• Not passing the request ID. The metric series has a request_id label that the SDK adds per request. The URL template does not reference it; high-cardinality labels belong in exemplars, not in pivots.

The dashboard-level equivalent uses the same __field.labels.<name> syntax, which Grafana resolves the same way against the same data model.

How to validate it

# READ-ONLY: confirm the labels on the metric series.
curl -fsS -u grafana-admin:$GRAFANA_ADMIN \
  --data-urlencode 'query=sum by(service, route, status) (rate(http_requests_total[5m]))' \
  http://grafana.internal:3000/api/datasources/proxy/uid/prom-prod-us/api/v1/query \
  | jq '.data.result[0].metric'
# {
#   "service": "checkout-svc",
#   "route": "/v2/cart",
#   "status": "500"
# }

# READ-ONLY: confirm the labels on the Loki stream.
curl -fsS -u grafana-admin:$GRAFANA_ADMIN \
  --data-urlencode 'query={service="checkout-svc",route="/v2/cart"}' \
  --data-urlencode 'limit=1' \
  http://grafana.internal:3000/api/datasources/proxy/uid/loki-prod-us/loki/api/v1/query \
  | jq '.data.result[0].stream'
# {
#   "service": "checkout-svc",
#   "route": "/v2/cart"
# }

# READ-ONLY: substitute the labels into the URL template
# by hand and confirm the resulting URL renders to a 200.
SVC=checkout-svc
ROUTE=/v2/cart
T=1755100320
URL="http://grafana.internal:3000/explore?schemaVersion=1\
&panes=%7B%22logs%22%3A%7B%22datasource%22%3A%22loki-prod-us%22%2C%22queries%22%3A%5B%7B%22expr%22%3A%22%7Bservice%3D%5C%22${SVC}%5C%22%2Croute%3D%5C%22${ROUTE}%5C%22%2Cstatus%3D%5C%225%5C%22%7D%22%7D%5D%7D%7D\
&from=$((T-120))&to=$((T+120))"
curl -fsS -u grafana-admin:$GRAFANA_ADMIN "$URL" \
  -o /dev/null -w '%{http_code}\n'
# 200

# CONFIGURATION: reload Prometheus and Alloy after a label
# rename.
sudo systemctl reload prometheus
sudo systemctl reload alloy

A clean validation: the metric label set and the Loki stream label set agree on the names; the substitution into the URL produces a query that returns lines; the link behaves the same in a fresh browser session.

How it can fail

The most expensive label-passing failure modes from real production incidents.

1. Empty substitution from a label rename. The metric panel’s legend exposes route after a refactor; the URL template references ${__series.labels.handler} (the old name). The substitution is empty; the LogQL becomes {service="x", handler="", status="5"}; Loki returns no lines. The symptom is a link that opens to “no results” with no error.
2. Cardinality explosion from passing a per-request label. The metric series carries trace_id because the SDK adds it per request. The URL template passes it through. The resulting LogQL {service="checkout-svc", trace_id="abc123"} is correct in shape; the stream is one line; the cost is a Loki query that scans every chunk for that service for the time window. The symptom is a slow pivot that returns one line, repeated for every click.
3. Label-name drift between metric and Loki. The metric legend uses route; the Loki stream label is path. The substitution is right; the LogQL references a label Loki does not have. The query returns no lines. The symptom is identical to the empty-substitution case from the operator’s perspective.
4. Special characters in a label value. A customer_id label value that contains a literal " produces a LogQL like {service="x", customer_id="abc"def"} that fails to parse. The symptom is “Loki rejected the query” with the parse error visible in the Explore view.
5. PII labels passed without review. A metric series has customer_email because a developer added it for debugging. The pivot passes the value through; the URL lands in browser history and the access log; the destination data source sees a customer identifier. The symptom is a privacy ticket from the security team.
6. Mismatch between the metric’s by clause and the URL template. The metric query groups by service only. The URL template references ${__series.labels.route}. The substitution is empty because the metric series does not have a route label. The symptom is a link that always renders an empty route.

How to troubleshoot it

The diagnostic order is “what labels does the metric expose?”, “what labels does the Loki stream have?”, “do the names agree?”, “does the substitution produce a valid LogQL?”.

1. Inspect the metric series labels. Run the metric query through the Prometheus proxy and confirm the metric block on the returned series. A series that exposes service only cannot pass route to a LogQL.
2. Inspect the Loki stream labels. Run a sample query against Loki and confirm the stream block on the returned streams. A stream that has path and not route cannot match a LogQL that references route.
3. Compare the two label sets. A label that exists on one side and not the other is the drift. Fix at the source: Prometheus metric_relabel_configs for the metric, Alloy stage.labels for the log.
4. Render the URL template by hand. Substitute the label values into the template manually. A route="" in the rendered URL is the empty-substitution failure shape; the template references the wrong label name.
5. Validate the rendered LogQL. Run the rendered LogQL through the Loki proxy directly. A working direct query that fails through the pivot isolates the problem to the URL template.
6. Audit the labels a pivot passes. For each ${__series.labels.<name>} in the URL template, confirm the label is low-cardinality, non-PII, and present on both sides. Remove the high-cardinality and PII labels.

Security implications

• The pivot passes label values to a URL. A label like customer_id or email is a privacy incident the moment the pivot renders. Audit the labels a pivot passes; remove any PII or high-cardinality labels before the pivot ships.
• The pivot’s destination has its own permissions. A pivot that passes a label to a Loki data source with broader read permissions than the source Prometheus is a privilege escalation. Review the pair as a pair.
• The label values land in browser history. A pivot that passes customer_id puts the value in the URL bar; the URL bar is in browser history. Treat pivot URLs as audit-able URLs.

Performance implications

• Cardinality is the cost. A pivot that passes a high-cardinality label is a slow Loki query. The high-cardinality labels do not belong in the pivot; they belong in exemplars.
• The substitution is cheap. Grafana resolves the template at click time; the cost is microseconds. The cost is paid in the destination query’s response time.
• The LogQL selectivity matters. A pivot that passes a narrow set of labels (service, route, status) asks Loki for a small stream. A pivot that passes no labels asks Loki for every line. The narrow pivot is fast; the wide pivot is slow.

Production guidance

• Align label names at the source. Prometheus metric_relabel_configs and Alloy stage.labels are the right places to fix a drift.
• Maintain an allow-list of labels a pivot passes. The list is version-controlled alongside the dashboard JSON. Remove any label that is high-cardinality, PII, or per-request.
• Validate the metric’s by clause against the pivot’s URL template. The pivot can only pass labels the metric series actually exposes.
• Audit pivot URLs for PII before the pivot ships. Treat the URL template as audit-able code.
• Exercise the pivot against a representative set of label values. A pivot that works for service="checkout-svc" may fail for service="checkout-svc-staging" because the staging service has a different route label.

Verification

You should now be able to answer:

• Which Grafana 11.x template syntax substitutes a metric series label into a Loki stream selector, and what is the difference between ${__series.labels.NAME} (data link variable) and ${NAME} (dashboard variable)?
• Why is aligning label names at the source (Prometheus relabel config and Alloy pipeline) better than remapping them at the pivot?
• Which four classes of labels pass through a pivot cleanly, and which two classes (high-cardinality, PII) do not?
• What is the failure shape when a metric’s by clause exposes a label set that does not match the labels the pivot’s URL template references?

Quiz