LogQL Stream Selectors

At 02:30 support pages you. The checkout service has been returning HTTP 500 for roughly one in three requests since 14:00. They want every log line from the service between 14:00 and 14:10. You paste {service="checkout"} into Loki and get an answer in under a second. A colleague opens the same Grafana, types {} to “see everything”, and gives up after eight minutes. Same data. Same backend. Different first three characters.

Those three characters are the stream selector, and they are the single biggest dial on query cost in Loki.

What a stream selector is

A stream selector is the {...} block at the head of every LogQL query. It is a conjunction of label matchers — each one a comparison between a label name and a value — that Loki uses to identify which streams are eligible for the rest of the query.

The four matchers are the canonical Prometheus-style operators applied to labels:

• label="value" — exact equality. The label is present and the value matches byte for byte. This is the cheapest matcher because Loki’s inverted index stores it as a hash lookup.
• label!="value" — negated equality. The label is either absent or has any value other than value. Equality negation is faster than regex negation because the index can do set subtraction directly.
• label=~"regex" — regex match. Loki uses RE2, so the engine is linear in input size and immune to catastrophic backtracking. The matcher still costs more than equality because every candidate stream has its label value re-evaluated against the expression.
• label!~"regex" — negated regex. The most expensive matcher in production because every stream that does not match has to be enumerated.

A selector may combine several matchers. They are ANDed. The selector is the conjunction; Loki returns the intersection of every matcher’s matched set.

{service="checkout", env="prod"}
{service=~"checkout|inventory", env!="staging"}
{service!~"debug-.*", level=~"error|warn"}

Why a sysadmin cares

The selector is the dial that decides whether a query returns in 800 milliseconds or whether it ties up a querier for a minute.

• Cost. Loki’s per-tenant limits are enforced in stream-hours and bytes scanned. A query with no selector touches every stream the tenant has. A query with a bounded selector touches a handful. The difference is what makes an alert page stay green versus what makes the whole tenant fall over.
• Latency. Selectors that hit the inverted index are sub- millisecond. Selectors that fall back to chunk scanning are proportional to bytes scanned. The first complaint of a poorly designed selector is a slow dashboard.
• Multi-tenancy. Loki is multi-tenant. Every query is implicitly scoped to a tenant via the X-Scope-OrgID header (or the configured equivalent). The selector does not cross tenants — a selector that “does not match” often means the request reached Loki with the wrong tenant header rather than the labels being wrong.
• Investigation ergonomics. A well-chosen selector is also a well-chosen question. \{service="checkout", env="prod"\} | level= "error" is the question “what errors did production checkout emit?” The structure of the selector is part of how the on-call engineer thinks.

How it works — the mental model

LogQL query
   |
   v
{ service="checkout", env="prod" }   <- stream selector (label matchers)
   |
   v
Loki index lookup
   |- inverted index for label "service" -> hash("checkout") -> {streamA, streamB, ...}
   |- inverted index for label "env"     -> hash("prod")     -> {streamX, streamY, ...}
   |- intersect
   v
candidate streams: { streamA, streamD }      <- handful, not thousands
   |
   v
for each stream, for each chunk in [t0, t1]:
   open the chunk, apply line filter / parser / aggregation

The crucial point is the third arrow. The selector narrows the candidate set using the inverted index before any chunk is opened. Everything downstream — line filters, parsers, aggregations — runs over the survivors. A selector that selects everything is a selector that does nothing.

How to configure it

The four matcher forms, with the canonical use case for each.

# Equality. The default. Use when the label value is known.
{service="checkout"}

# Negated equality. Use to exclude a single known value.
{level!="debug"}

# Regex alternation. Use when several values are equivalent.
{service=~"checkout|inventory|pricing"}

# Anchored regex. RE2 is linear, but the prefix anchor lets the
# index skip candidates faster.
{service=~"^checkout.*"}

# Combined matchers. ANDed together.
{service=~"checkout|inventory", env="prod", level!="debug"}

Three rules of thumb:

• Prefer equality. It is the cheapest matcher and the easiest to reason about. Reach for regex only when the set of values is known and small.
• Anchor regex with ^ when the pattern has a literal prefix. {service=~"^checkout.*"} is faster than {service=~"checkout.*"} because the index can prune non-matching buckets on the prefix before evaluating the suffix.
• Never write {} in production. The empty selector is “every stream I have”. It is correct for one purpose only: discovering what labels exist, via the logcli labels command.

How to validate it

The validation ladder. Start at the index, never at the chunks.

# 1. What label values exist for service?
logcli --addr=http://loki:3100 labels service
# {service="auth"}
# {service="checkout"}
# {service="inventory"}
# {service="pricing"}

# 2. How many streams match a bounded selector?
logcli --addr=http://loki:3100 series --since=15m '{service="checkout"}' | wc -l
# 12      (bounded: 3 replicas x 4 instances)

# 3. How many streams match an empty selector?
logcli --addr=http://loki:3100 series --since=15m '{}' | wc -l
# 2187    (every stream in the tenant)

# 4. What does the selector actually return?
logcli --addr=http://loki:3100 query --since=15m \
  '{service="checkout", env="prod"} | level="error"'
# {service="checkout", env="prod"} 2026-08-14T14:02:11Z error   ...
# {service="checkout", env="prod"} 2026-08-14T14:02:14Z error   ...

# 5. How long did the query take?
logcli --addr=http://loki:3100 instant --since=15m \
  '{service="checkout", env="prod"} | level="error"' --stats
# status: success
# summary: executed in 47.213ms
# bytesProcessed: 1.2MB

The series and labels commands talk to the index, not the chunks. They are the right tool for “did the selector match what I expected?”. The query and instant commands read chunks; they are the right tool for “what is the answer?”.

How it can fail

Six recurring failure modes. Each maps to an observable symptom.

1. The empty selector. A new on-call engineer writes {} to “see what’s there” and the dashboard times out. Symptom: loki_request_duration_seconds for /loki/api/v1/query spikes into the seconds; the query returns limit reached or a 500.
2. The unbounded regex. {service=~".+"} is functionally identical to {}. A well-meaning engineer wraps the selector in a regex “to be safe”. Symptom: query latency rises to the same level as the empty selector.
3. Catastrophic label cardinality. A pipeline accidentally promotes a per-request field (request_id, trace_id) to a stream label. Every request is a new stream. Symptom: loki_index_streams grows by an order of magnitude after a deployment; the index gateway runs out of memory.
4. Wrong tenant header. A multi-tenant Loki receives a query with the operator’s personal tenant header instead of the team’s. The selector is correct but matches zero streams. Symptom: empty result set; the same selector from another client returns data.
5. Label not extracted by the pipeline. The selector asks for {region="eu-west-1"} but the pipeline never extracted the region field. The label is not present on any stream. Symptom: logcli labels region returns no rows; the selector returns zero streams.
6. Negation used where equality would do. {service!="debug-tools"} matches every stream whose label set does not include service="debug-tools", including streams where the label is absent. Symptom: query “returns too much” and the on-call engineer cannot tell whether the noise is the negation or a different problem.

How to troubleshoot it

The diagnostic order for “the selector returns nothing” or “the selector returns everything”:

1. Confirm the tenant. Inspect the HTTP header on the request. curl -H "X-Scope-OrgID: team-checkout" .... The tenant id has to match what the pipeline wrote under. Most “empty selector” bugs are tenant bugs.
2. Confirm the label exists. logcli labels <labelname>. If the label is absent from the output, the pipeline never extracted it; fix the extraction stage, not the selector.
3. Confirm the label values. logcli labels <labelname> lists the values seen in the last retention window. If the value you expected is not there, the source is mis-tagging.
4. Count streams with series. logcli series --since=15m '\{label="value"\}' | wc -l. The count tells you whether the selector is narrow or wide. A selector returning thousands of streams is usually a misconfigured label or a missing filter.
5. Inspect one stream’s label set. logcli series --since=15m '\{...\}' | head -1. The first stream’s full label set tells you what other labels you have to filter on.
6. Time-box the search. A 24-hour selector that returns nothing is more useful than a 5-minute one that returns nothing — the 24-hour version tells you whether the label is missing entirely or just absent right now.

Security implications

Three concerns, in order of severity.

• Cross-tenant data exposure. Loki is multi-tenant by design. A misconfigured reverse proxy that strips or rewrites X-Scope-OrgID allows a tenant to see another tenant’s data. Stream selectors do not protect you here; the tenant boundary is enforced above the query engine. See the chapter on Loki tenant isolation.
• Regex denial of service. RE2 is linear, but a regex with a pathological shape still costs CPU per stream. A user-controlled label value passed into a regex matcher is a small DoS vector. Treat labels as data, not as code.
• Sensitive label values. Labels are visible in logs and in the index. A label like customer_id or user_email is a PII surface. The selector cannot protect you; the label extraction stage is the right place to drop or hash such fields.

Performance implications

The selector is the cheapest part of the query path. That is why spending more matchers to make it tighter is almost always a win.

• Equality matchers. Hash lookup in the inverted index. O(1) per matcher per stream candidate. This is what the index is built for.
• Regex matchers. Each candidate stream’s label value is evaluated against the RE2 expression. The cost is linear in the number of candidates and in the expression’s complexity. An anchored expression with a literal prefix is the cheapest possible regex.
• Negation matchers. Enumerate the candidates that match every other matcher and subtract. Cost grows with the number of candidates. {env!="prod"} against a fleet of two environments is cheap; {env!="prod"} against a fleet of two hundred is not.
• Empty selector. Cost is bounded by the tenant’s total stream count. This is the maximum-cost selector and the only one that is never the right answer in production.

Production guidance

• A selector belongs on every Grafana panel, every alert, every recording rule. The only legitimate exception is a one-shot exploration query run interactively by a human, not a panel.
• The selector must include env or an equivalent. A selector without an environment label accidentally returns development logs alongside production ones. The on-call engineer does not know which until the post-mortem.
• Promote the fields you want to filter on to labels at the pipeline boundary, not in the query. A regex over a parsed field is doing the index’s job, badly.
• Audit selectors on dashboards with logcli series against realistic time windows. A selector that returns thousands of streams in production is a bug.

Verification

You should now be able to answer:

• Why is {service="checkout"} faster than {service=~".+"} when both return the same streams?
• When is a regex matcher the right tool, and what anchoring rule makes it cheapest?
• What is the first thing to check when a selector returns zero streams — the selector, the labels, or the tenant?
• What is the operational cost of an empty selector, in concrete terms of bytes scanned and streams touched?

Quiz