Filtered Stream Pivot

A metric panel shows the 5xx rate climbing at 14:32 for checkout-svc / POST /v2/cart. The operator clicks the line. They land in Loki with the query {service="checkout-svc", route="/v2/cart"} |= "status=500" against a four-minute window centred on the clicked point. The result is fourteen log lines. Twelve are the actual error events; two are warnings from the same code path. The operator reads the lines, identifies the cause, files the fix. Time to answer: ninety seconds.

The same panel, a different team. The URL template references ${service} (a dashboard template variable set to “all”). The substitution is service="all". Loki returns nothing. The team concludes the service is silent. Time to answer: ten minutes, plus the on-call for the log pipeline getting a “logs are missing” ticket.

The pivot lands on a Loki query. The shape of that query is the answer. A query that returns fourteen lines of the right shape is a pivot that works. A query that returns nothing, or returns forty thousand lines of mixed severity, is a pivot that does not work. The difference is the four filter positions in LogQL and which one is used for what.

What it is

A filtered stream pivot is the dashboard-to-logs correlation where the resulting Loki query is a stream selector with a narrow time window and one or two additional filter stages. The shape is:

{ <stream selector> } [ <pipeline stages> ]

The stream selector ({...}) is the mandatory first part; it filters by Loki stream labels — typically the service identity, the route, and sometimes the status code. The pipeline stages that follow are an optional chain that filters or transforms the lines: |= (line contains), != (line does not contain), |~ (line matches regex), | json (parse as JSON and extract a field), | level=... (filter by extracted level).

For a metric-panel-to-logs pivot, the right shape has three filters: a narrow time window (two minutes before the clicked point, two minutes after), a status or severity filter that maps the metric’s failure mode to log severity, and a label set that scopes the stream to the service and route the metric shows.

Why a sysadmin cares

The pivot’s value is the line count. Four production shapes appear when the filter shape is wrong:

• No time filter. The pivot opens Loki with the dashboard’s default six-hour window. The engineer lands on thousands of lines, most of which are unrelated to the anomaly. They scroll, give up, and conclude the pivot is useless.
• No status filter. The metric is a 5xx rate; the pivot has no status="5" filter. Loki returns every log line for the service and route, including info and debug. The engineer has to scroll past 90% of the lines to find the errors.
• Stream selector on the wrong label. The pivot’s stream selector is {job="checkout-svc"} but the metric’s service label is the right one. Loki returns the wrong service’s logs. The engineer lands on someone else’s outage.
• Pipeline filter in the wrong position. The author puts the |= "status=500" filter after a | json stage but the log line is plain text. The filter matches nothing. The author puts the |= "status=500" filter before the stream selector in the URL template; Loki treats it as part of the stream selector and returns an error.

How it works

The mental model is “the LogQL is four positions, each with a job”. The stream selector narrows by stream label; the time window narrows by clock; the line filter narrows by content; the parser+label filter narrows by extracted field.

   position 1              position 2       position 3          position 4
   stream selector         line filter      parser              label filter
   {service="...",         |= "..."        | json              | level="error"
    route="...",                           | status_code="..."
    status="..."}
   ---by Loki stream----   ---by content---  ---by structure---  ---by extracted---

Three observations on the shape:

1. The stream selector is mandatory. Without a stream selector Loki refuses the query. The selector names the Loki stream labels the pivot targets. The right choice is the labels the metric exposes, with the values substituted from the clicked series.
2. The time window is separate. The LogQL does not carry the time range; the Grafana Explore URL carries from and to as separate parameters. The pivot passes both the query and the time range; Loki applies both.
3. Filters compose left-to-right. A |= filter after the stream selector narrows the lines that match the stream. A | json stage parses the lines; a | level=... after the parser narrows by extracted field. Filters that depend on a parser must come after the parser.

How to configure it

The configuration is the URL template that produces the right LogQL plus the right time range. The two are passed to Loki as separate parameters in the Explore URL.

{
  "type": "timeseries",
  "title": "5xx rate by service and route",
  "datasource": { "type": "prometheus", "uid": "prom-prod-us" },
  "targets": [
    {
      "expr": "sum by(service, route) (rate(http_requests_total{status=~\"5..\"}[5m]))",
      "refId": "A"
    }
  ],
  "fieldConfig": {
    "defaults": {
      "links": [
        {
          "title": "Logs: ${__series.labels.service} ${__series.labels.route}",
          "url": "/explore?schemaVersion=1&panes=%7B%22logs%22%3A%7B%22datasource%22%3A%22loki-prod-us%22%2C%22queries%22%3A%5B%7B%22expr%22%3A%22%7Bservice%3D%5C%22%24%7B__series.labels.service%7D%5C%22%2Croute%3D%5C%22%24%7B__series.labels.route%7D%5C%22%2Cstatus%3D%5C%225%5C%22%7D%20%7C%3D%20%5C%22level%3Derror%5C%22%22%7D%5D%7D%7D%7D&from=${__value.time:date-seconds}-120&to=${__value.time:date-seconds}+120",
          "targetBlank": true,
          "includeVars": true
        }
      ]
    }
  }
}

The relevant choices, walked through:

• Stream selector with the right labels. The selector is {service="${service}", route="${route}", status="5"}. The service and route values are substituted from the clicked series. The status="5" is a regex match against the status stream label, picking up 500, 502, 503 and 504 with one filter. The selector narrows by Loki stream before any line is read.
• Line filter by content. The pipeline adds |= "level=error" after the stream selector. The line-filter stage narrows by content. The right choice for “errors only” is a level=error line filter when the log format is plain text, or a | json | level="error" pipeline when the log format is JSON.
• Two-minute time window. The from and to parameters are ${__value.time:date-seconds}-120 and ${__value.time:date-seconds}+120. The window is centred on the clicked point; the operator sees the anomaly’s window, not the dashboard’s window.
• Right data source. The panes JSON names loki-prod-us as the data source UID. The substitution targets the right cluster; the engineer lands in the logs that match the metric.

The JSON-encoded URL above encodes {service=\"...\",route=\"...\",status=\"5\"} |= \"level=error\" as the LogQL string. The full URL, when rendered against a service of checkout-svc and a route of /v2/cart, is:

http://grafana.internal:3000/explore?schemaVersion=1
  &panes={"logs":{"datasource":"loki-prod-us",
    "queries":[{"expr":
      "{service=\"checkout-svc\",route=\"/v2/cart\",status=\"5\"} |= \"level=error\""
    }]}}
  &from=1755100200
  &to=1755100440

That URL is what the operator sees in the address bar after the click.

How to validate it

# READ-ONLY: run the LogQL the pivot produces, directly
# against the Loki proxy, and confirm the line count is
# small (target: tens, not thousands).
curl -fsS -u grafana-admin:$GRAFANA_ADMIN \
  --data-urlencode 'query={service="checkout-svc",route="/v2/cart",status="5"} |= "level=error"' \
  --data-urlencode 'start=1755100200000000000' \
  --data-urlencode 'end=1755100440000000000' \
  --data-urlencode 'limit=50' \
  http://grafana.internal:3000/api/datasources/proxy/uid/loki-prod-us/loki/api/v1/query_range \
  | jq '.data.result | map(.values | length) | add'
# 14

# READ-ONLY: render the URL the pivot produces by hand
# and confirm the result is a 200.
URL="http://grafana.internal:3000/explore?schemaVersion=1\
&panes=%7B%22logs%22%3A%7B%22datasource%22%3A%22loki-prod-us%22%2C%22queries%22%3A%5B%7B%22expr%22%3A%22%7Bservice%3D%5C%22checkout-svc%5C%22%2Croute%3D%5C%22%2Fv2%2Fcart%5C%22%2Cstatus%3D%5C%225%5C%22%7D%20%7C%3D%20%5C%22level%3Derror%5C%22%22%7D%5D%7D%7D\
&from=1755100200&to=1755100440"
curl -fsS -u grafana-admin:$GRAFANA_ADMIN "$URL" \
  -o /dev/null -w '%{http_code}\n'
# 200

# READ-ONLY: confirm the time range the pivot produces
# is the expected two-minute window, not the dashboard's
# six-hour default.
curl -fsS -u grafana-admin:$GRAFANA_ADMIN "$URL" \
  | grep -oE 'from=[0-9]+&to=[0-9]+'
# from=1755100200&to=1755100440

A clean validation: the line count is in the tens, the URL returns 200, the time range is the expected window. A failure mode below maps to one of these signals failing.

How it can fail

The most expensive filtered-stream failure modes from real production incidents.

1. No time filter in the URL. The pivot opens Loki with the dashboard’s default time range. The operator lands on a six-hour window of every line for the service. The symptom is the scroll-of-shame pivot.
2. No status filter in the stream selector. The metric is a 5xx rate; the selector is {service="x", route="y"} with no status="5" filter. Loki returns every line for the route. The symptom is “the pivot returns tens of thousands of info lines”.
3. Stream selector references the wrong labels. The metric exposes service and route; the selector is {job="x", handler="y"}. The substitution succeeds; the LogQL references labels Loki does not have; the query returns no lines. The symptom is “the pivot opened to no results”.
4. Line filter inside the stream selector. The author wrote {service="x" |= "level=error"}. Loki returns a parse error. The symptom is “the pivot opened to an error in the Explore view”.
5. Parser stage missing before a label filter. The pipeline is | level="error" with no | json before it. Loki treats level as a stream label; the stream label does not exist; the filter is a silent no-op; the query returns every line. The symptom is “the level filter does not narrow anything”.
6. Time window reversed. The author wrote from=${__value.time:date-seconds}+120&to=${__value.time:date-seconds}-120. Loki treats the query as zero-duration and returns no lines. The symptom is “the pivot always returns empty”.

How to troubleshoot it

The diagnostic order is “does the rendered LogQL parse?”, “does it return lines?”, “is the line count in the right ballpark?”, “is the time range the right window?”.

1. Render the URL template by hand. Substitute the labels and the time into the URL manually. Open the rendered URL in a private browser tab. If the page errors out, the LogQL is malformed.
2. Run the rendered LogQL directly. Substitute the same labels into the LogQL and run it through the Loki proxy with the same time range. A working direct query that fails through the pivot isolates the problem to the URL encoding or the time range.
3. Count the lines. The target for a pivot’s LogQL is tens of lines for a narrow anomaly, low hundreds for a sustained incident. A pivot that returns thousands is too wide. Add a filter; tighten the time range.
4. Confirm the time window. The pivot’s from and to parameters should be the expected two-minute window. A pivot that opens with from=now-6h&to=now is opening against the data source’s default range; the URL template is missing the time range.
5. Validate the parser pipeline. If the LogQL has | level="error", confirm the pipeline has | json (or | logfmt) before it. A label filter on an extracted field without a parser is a silent no-op.
6. Inspect Loki’s query stats. Loki exposes per-query statistics through the proxy; the stats show the bytes processed, the lines scanned, and the lines returned. A pivot that scans gigabytes to return fourteen lines is too wide.

Security implications

• The pivot’s URL is in the access log. A LogQL that passes a customer_id filter puts the identifier in the URL; the URL lands in browser history and in the server access log. Audit the filters a pivot passes.
• The destination data source has its own permissions. A pivot that targets a Loki data source with broader read permissions than the source Prometheus is a privilege escalation. Review the pair.
• The parser pipeline may extract sensitive fields. A pipeline that parses JSON and exposes a customer_email field makes the field visible in the Explore view. Strip sensitive fields at the parser stage if necessary.

Performance implications

• The stream selector is the cheapest filter. A narrow stream selector asks Loki for a small set of streams; the chunk fetch is small. The first filter a pivot adds should always be in the stream selector, not in the pipeline.
• The line filter scans every line. A pipeline that applies |= "..." reads every line in the stream. A wide stream selector means a wide line scan.
• The parser is expensive. A | json stage parses every line. Use it only when the line filter is not enough.
• A correct filter is fast. A pivot with a narrow stream selector, a status filter, a level=error line filter, and a two-minute window returns fourteen lines in milliseconds.

Production guidance

• Always pass the time window in the URL. The window is what makes the pivot focused; without it the pivot is wallpaper.
• Always pass a status or severity filter that matches the metric’s failure mode. The metric is the filter; the pivot should not be wider than the metric.
• Use the stream selector for label filters; use the pipeline for content and extracted-field filters. Mixing the two is the common mistake.
• Validate the line count for every pivot. A pivot that returns thousands is too wide; tighten the filters.
• Encode the LogQL in the dashboard JSON with care. The panes JSON is hand-fragile; copy the URL the Explore UI generates.

Verification

You should now be able to answer:

• What are the four filter positions in LogQL and what is each one for?
• What is the right time window for a metric-to-logs pivot, and what parameter carries it in the Explore URL?
• Why is a status or severity filter the right addition to a stream selector for a 5xx rate pivot?
• What is the failure shape when a parser stage is missing before a label filter on an extracted field?

Quiz