Metric to Log Workflow · Observability

The error rate for the checkout service is eighteen percent. The on-call engineer opens the dashboard and clicks the spike. They need to know what the application was logging at the moment the errors happened. The dashboard has no log panel. The engineer has to open a new tab, open Explore, type the service name into the Loki query bar, and guess the time range. Three minutes later they have the log lines. The investigation is now three minutes old and the on-call engineer has lost focus.

The metric-to-log pivot removes the dead time. The panellist clicks the spike. Grafana opens Loki with the same labels as the metric, the same time range as the dashboard, and the relevant log lines in front of them. The drill is two seconds, not three minutes.

What it is

The metric-to-log pivot is a Grafana data link that, on click, opens a Loki Explore page using the labels of the clicked metric series as the LogQL stream selector. The pivot is configured per panel, not per data source. The mechanism is the Grafana data link template syntax and the Loki stream label contract.

The shared identifier is the label set. The metric series is labelled with job, instance, status, and so on. The Loki stream is labelled with the same values. The pivot substitutes the labels of the clicked metric into the Loki stream selector and opens the query.

  Prometheus series
  +--------------------------------------+
  | http_server_requests_total           |
  |   job="checkout"                     |
  |   instance="10.0.1.5:8080"           |
  |   status="500"                       |
  +----------------------+---------------+
                         |
                         | click
                         v
  Loki query (auto-built)
  +--------------------------------------+
  | {job="checkout",                    |
  |  instance="10.0.1.5:8080",          |
  |  status="500"}                      |
  +--------------------------------------+

Why a sysadmin cares

The metric-to-log pivot is the most common pivot in an on-call runbook. It is the first drill an analyst runs when the metric panel flags an anomaly. The drill is the difference between “I have an alert” and “I have a hypothesis with evidence”.

Three operational payoffs.

Speed. The pivot removes the label-copy step. The on-call engineer does not need to know which label names match between Prometheus and Loki. The link does the matching.
Correctness. The pivot preserves the time range of the dashboard. The engineer is looking at the same ten-minute window as the metric panel, not the dashboard’s default range.
Breadth. The pivot can be configured with extra label filters (for example, level="error") so the engineer lands on the relevant severity, not on every log line.

The cost is the discipline of keeping the Prometheus and Loki label sets aligned. The convention is the same set of labels on both sides: job, instance, status, method, path.

How it works — the data link template

The Grafana data link is a URL template. The template is substituted at click time using the values of the clicked metric datapoint. The key substitutions:

${__series.labels} — the full label set of the series, rendered as {job="x", instance="y"}.
${__series.name} — the metric name.
${__value.time} — the timestamp of the datapoint.
${__url_time_range} — the dashboard’s current time range.
${__data.fields.label} — a specific field value from a table panel.

The standard metric-to-log pivot URL opens the Loki Explore page with the same labels:

/explore?schemaVersion=1&panes=%7B%22logs%22%3A%7B%22datasource%22%3A%22loki%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22expr%22%3A%22%7B${__series.labels}%7D%22%7D%5D%7D%7D

The ${__series.labels} is replaced at click time with the label set of the clicked series. The datasource field selects the Loki data source by UID. The result is a Loki query pane with the stream selector pre-filled.

The substitution that survives every Grafana upgrade

The single most resilient pattern is the bare {__series.labels} substitution. The substitution is replaced with the entire label set, formatted as a LogQL stream selector. The expansion is:

${__series.labels}  ->  {job="checkout", instance="10.0.1.5:8080", status="500"}

The expansion is what Loki expects inside a stream selector. The engineer can append a line filter (| json | level="error") and a parser (| json | latency > 1) without breaking the substitution.

The substitution family that does not work is the field substitution (${__field.labels}). The field substitution is replaced with the labels of the click target, but the field is only populated in table panels. On a time series panel, the field is empty and the substitution produces an empty stream selector.

How to configure it

The data link is configured on the panel. The cleanest path in a managed Grafana is the provisioning YAML; the dashboard JSON is acceptable for one-off panels.

# grafana/provisioning/dashboards/checkout.yaml
apiVersion: 1
providers:
  - name: checkout
    folder: Observability
    type: file
    options:
      path: /var/lib/grafana/dashboards

The dashboard JSON carries the data link on the relevant panels. Two patterns are common.

Pattern A — link on the metric panel that opens Explore with the same labels:

{
  "targets": [
    {
      "expr": "sum by(job, instance, status) (rate(http_server_requests_total{job=\"checkout\"}[1m]))",
      "legendFormat": "{{status}} on {{instance}}"
    }
  ],
  "options": {
    "dataLinks": [
      {
        "title": "Logs for {{job}} {{instance}} {{status}}",
        "url": "/explore?schemaVersion=1&panes=%7B%22logs%22%3A%7B%22datasource%22%3A%22loki%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22expr%22%3A%22%7B${__series.labels}%7D%20%7C%3D%20%22error%22%22%7D%5D%7D%7D&orgId=1"
      }
    ]
  }
}

Pattern B — link as a side panel that opens with the same time range, for a deeper drill:

{
  "type": "logs",
  "datasource": { "type": "loki", "uid": "loki" },
  "targets": [
    {
      "expr": "{${__series.labels}}",
      "refId": "A"
    }
  ],
  "options": {
    "dataLinks": [
      {
        "title": "Trace for this error",
        "url": "/explore?schemaVersion=1&panes=%7B%22traces%22%3A%7B%22datasource%22%3A%22tempo%22%2C%22queries%22%3A%5B%7B%22query%22%3A%22$${__value.raw}%22%2C%22queryType%22%3A%22traceql%22%7D%5D%7D%7D&orgId=1"
      }
    ]
  }
}

The Loki side — the data source is configured with the same label set as Prometheus. The Prometheus OTel receiver sets the job and instance automatically. The Loki pipeline sets the same labels via the relabel pipeline.

# grafana/provisioning/datasources/loki.yaml
apiVersion: 1
datasources:
  - name: Loki
    type: loki
    uid: loki
    url: http://loki:3100
    jsonData:
      maxLines: 1000

The application side — the Prometheus exporter and the Loki agent must agree on the labels. The Alloy pipeline that scrapes the application and ships to Loki:

# /etc/alloy/config.alloy
loki.relabel "checkout" {
  rule {
    action   = "labelmap"
    regex    = "job"
    target_label = "job"
  }
  rule {
    action   = "labelmap"
    regex    = "instance"
    target_label = "instance"
  }
  rule {
    source_labels = ["level"]
    target_label  = "level"
  }
}

loki.write "checkout" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
  }
}

How to validate it

# 1. The metric has the labels we expect.
curl -s 'http://prometheus:9090/api/v1/series?match[]=http_server_requests_total{job="checkout"}' \
  | jq '.data[0]'
# {"job":"checkout","instance":"10.0.1.5:8080","status":"500","method":"POST","path":"/api/v1/orders"}

# 2. The Loki stream has the same labels.
logcli query --since=10m \
  '{job="checkout",instance="10.0.1.5:8080",status="500"}'
# 2026-08-13T03:00:12Z {job="checkout" instance="10.0.1.5:8080" status="500"} msg="payment failed"

# 3. The pivot URL renders correctly. The ${__series.labels}
#    substitution expands to the same label set.
echo '/explore?schemaVersion=1&panes=%7B%22logs%22%3A%7B%22datasource%22%3A%22loki%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22expr%22%3A%22%7B${__series.labels}%7D%22%7D%5D%7D%7D&orgId=1' \
  | sed 's|${__series.labels}|{job="checkout", instance="10.0.1.5:8080", status="500"}|'
# /explore?schemaVersion=1&panes=...{job="checkout", instance="10.0.1.5:8080", status="500"}...

# 4. The dashboard data link is loaded.
curl -s -u admin:admin http://grafana:3000/api/dashboards/uid/checkout \
  | jq '.dashboard.panels[0].options.dataLinks'
# [{"title":"Logs for ...","url":"/explore?...{__series.labels}..."}]

How it can fail

Six recurring failure shapes.

The label set is misaligned. The Prometheus series is labelled with instance="10.0.1.5:8080". The Loki stream is labelled with instance="checkout-7d4b8". The substitution produces a Loki query that returns zero rows. Symptom: the pivot opens Explore, the Loki query returns no results, the operator assumes the logs are missing.
The data link uses ${__field.labels}. The field substitution only works on table panels. On a time series panel, the field is empty. Symptom: the pivot opens Explore with {job=""}, the query errors out.
The dashboard JSON is not re-imported. The data link is added in the panel editor but the provisioning file is not updated. The next time the dashboard is loaded from provisioning, the link is gone. Symptom: the link works briefly, then disappears after a Grafana restart.
The Loki data source UID is wrong. The link uses datasource=loki but the data source UID is loki-prod. Symptom: the pivot opens Explore with the wrong data source, or with the default data source picker.
The time range is empty. The link uses ${__url_time_range} but the dashboard has no time range set. Symptom: the pivot opens Explore with the browser’s default time range (last six hours), not the dashboard’s range.
The label values are wrapped in extra quotes. The Loki pipeline adds a quote pair around every label value. The substitution produces {job=""checkout""}. Symptom: the Loki query is a parse error.

How to troubleshoot it

The diagnostic order is “is the label set aligned?”, “does the link render?”, “does the substitution expand?”, “does the resulting Loki query return rows?”.

Is the label set aligned? curl -s 'http://prometheus:9090/api/v1/series?match[]=...' and logcli query --since=10m '{...}'. The label sets must match by name and value.
Does the link render? Open the panel editor, hover over the data link field. The link text is shown. The presence of a ${__series.labels} substitution is the sanity check.
Does the substitution expand? Open the dashboard, click the panel, observe the URL. The ${__series.labels} should be replaced with the full label set. If the URL still contains ${__series.labels}, the template parser is not running.
Does the resulting Loki query return rows? Run the expanded query in logcli or in the Explore query bar. The query should return the log lines for the label set. If it returns zero rows, the label set is misaligned (failure mode 1).

What if the pivot opens a Loki query with no rows?

The on-call engineer has twenty seconds to decide whether the pivot is broken or the system is silent. The default assumption is that the logs are missing. The engineer pages the logging team. The logging team confirms Loki is healthy. The engineer pages the platform team. The platform team realises the label set was misaligned after a recent relabel pipeline change.

The cost is twenty minutes of useless paging. The fix is to make the failure mode cheap to detect: the pivot should land on a query that returns at least one row, or the provisioning should fail at apply time if the label set does not match.

The discipline is to add a panel-level sanity check that asserts “the metric-to-log pivot for this panel returns at least one row when the metric is non-zero”. The check catches the misalignment before the on-call engineer does.

Security implications

The metric-to-log pivot does not introduce new attack surface. The URL is internal to the Grafana instance and does not leave the cluster. The substitution variables are derived from the clicked datapoint, not from user input.

The risk is around the Loki query expansion. A label value that contains a Loki parser keyword (for example, job="__name__") can produce a query that errors out. The mitigation is to sanitise the label values at the Loki pipeline boundary and to reject label values that match the parser keywords.

The second-order risk is around the Explore page being open in a context that has broader permissions than the dashboard. The default is “Editor” or “Viewer” depending on the user’s role. A panel that links to an Explore page with a query that returns credentials is a misconfiguration. The mitigation is to limit the Explore data sources to the same set as the dashboard.

Performance implications

The pivot is a single Explore page load. The Loki query that runs is the same shape as a regular dashboard query. The performance cost is the cost of the Loki query, not the cost of the pivot.

The cost to watch is the cardinality of the stream selector. A panel that pivots on a high-cardinality label (for example, user_id) produces a stream selector that scans every stream in the Loki index. The discipline is to pivot on labels with low cardinality (job, instance, status) and to filter on the high-cardinality labels (user_id) inside the query, not in the stream selector.

Production guidance

Align the label set. The Prometheus exporter and the Loki pipeline must agree on the label names and the label values. The audit is a single shell command that compares the two series.
Pivot on lowercase labels. Loki stream labels are case-sensitive. The convention is lowercase labels everywhere, with no underscores converted to camelCase.
Use the bare ${__series.labels} substitution. The field substitution is fragile. The field set is populated only on table panels.
Limit the pivot to a row in the dashboard. The data link is a panel-level setting. The pivot only works from the panel that declares it. Operators who want to pivot from a different panel need to declare the link on that panel too.

Verification

You should now be able to answer:

Which shared labels does the metric-to-log pivot rely on?
What is the correct Grafana data link template variable for a metric-panel pivot?
What is the failure shape of a label-set mismatch between Prometheus and Loki?
How do you validate the pivot end to end with a synthetic request?