Label Matching Operators

A new exporter rolls out across the fleet. The team writes a dashboard for it and the dashboard reads “No data”. The same metric returns values when the selector is dropped entirely. The selector is the suspect. Half the time the answer is “the label value has a typo”; the other half the answer is “the regex is not anchored and matches in an unexpected way”. This lesson is the four label-matching operators, when to use each, and the trap that catches every operator eventually.

What they are

A label matcher has the form { name op value } where name is a label name identifier, op is one of the four operators, and value is a quoted string. The four operators:

A selector may carry zero or more matchers. A selector with no matchers selects every series of that metric name. A selector with one or more matchers selects the series whose labels match every matcher. Matching is conjunctive: all matchers must match.

The matcher syntax is:

node_cpu_seconds_total{mode="idle"}                       # exact
node_cpu_seconds_total{mode!="idle"}                      # not-equal
node_cpu_seconds_total{instance=~"node-(a|b):.*"}         # regex
node_cpu_seconds_total{instance!~"node-c:.*"}             # regex-not
node_cpu_seconds_total{mode="idle", job="node"}           # AND
node_cpu_seconds_total{mode=~"idle|iowait"}               # alternation

The values inside the quotes are strings. There is no implicit type coercion; mode="0" matches a string label whose value is the character 0, not the number zero.

Label name vs label value

There are two distinct kinds of matching in PromQL:

• Label name matching — selecting which label names appear in the result. Done by label_replace(...), label_join(...) and the sum by (...) / sum without (...) aggregation modifiers. The label names appear as bare identifiers; they are not values.
• Label value matching — selecting which series are in the result based on the value of a specific label. Done by the matchers inside { }. The values are quoted strings.

The two are easy to conflate. The grammar makes the distinction clear: inside { }, the left-hand side is always a label name identifier and the right-hand side is always a string. Outside { }, label names appear in by(...), without(...) and as arguments to label_replace.

# Label-name selection: aggregate keeping only "instance"
sum by (instance) (rate(http_requests_total[5m]))

# Label-value selection: filter to a specific instance label
sum by (instance) (rate(http_requests_total{instance="api-1"}[5m]))

Why a sysadmin cares

The silent failure of =~ without anchors is the single most common PromQL mistake. The expression {job=~"prometheus"} matches any series whose job label contains the substring prometheus. That is the regex default. Anchors are explicit:

{job="prometheus"}       # exact equality, no regex
{job=~"^prometheus$"}    # regex with start and end anchors
{job=~"^prometheus.*"}   # "starts with prometheus"

A team that writes {job=~"api"} expecting to match job="api" will match job="api", job="api-gateway", job="my-api" and anything else that contains the substring. The dashboard renders “correctly” with the wrong series.

How it works

The label matcher is a filter applied during series selection. The selector layer of the evaluation pipeline calls into the TSDB index with a list of (name, op, value) tuples. The index returns the series whose labels match every matcher.

The index lookup uses two different paths depending on the operator:

• = and != — the index performs an exact match on the label name’s inverted index. instance="node-a" looks up the postlist for (instance, node-a) directly. This is fast.
• =~ and !~ — the index does not store a regex postlist. For each candidate label value, the engine runs the RE2 regular expression. The postlist lookup yields the candidate set; the regex narrows it.

The RE2 engine is fully anchored-on-the-left by default; the expression =~"prometheus" matches any value starting with prometheus. To match the entire value, add ^ and $ explicitly. RE2 does not support backreferences; complex extraction belongs in label_replace or in the exporter, not in a matcher.

How to configure it

The label matchers are part of the expression, not the configuration. The relevant operational configuration is what controls label cardinality, which determines how expensive each matcher is:

# prometheus.yml — drop high-cardinality labels at scrape time
scrape_configs:
  - job_name: api
    static_configs:
      - targets: ['api-1:9100', 'api-2:9100']
    metric_relabel_configs:
      # Drop the request-id label. A single request would otherwise
      # create a new series; a regex on it would scan every value.
      - action: labeldrop
        regex: 'request_id|correlation_id'
      # Cap label values to a bounded set.
      - action: labelmap
        regex: 'env_(.*)'
        replacement: 'env'

The operational discipline is: drop unbounded labels before they reach the TSDB. A regex matcher on a label that has 10 million distinct values is a guaranteed query timeout.

How to validate it

The /api/v1/series endpoint returns every series matching a selector, with the full label set. It is the right tool for sanity-checking a matcher before it goes into a dashboard:

# What does the engine see for this regex?
curl -s --data-urlencode 'match[]=node_cpu_seconds_total{instance=~"node-.*"}' \
  http://localhost:9090/api/v1/series \
  | jq '.data | length'

The response is a list of series descriptors, each with the full label set. Inspect the first few to confirm the regex anchored the way you intended:

curl -s --data-urlencode 'match[]=node_cpu_seconds_total{instance=~"node-.*"}' \
  http://localhost:9090/api/v1/series \
  | jq '.data[0:3] | map(.labels.instance)'

A regex that should match only node-a and node-b returning my-node-1 and prod-node-7 is the substring-anchor bug. The fix is to add ^ and $:

{instance=~"^node-(a|b)$"}

The expression browser has a small additional affordance: the “Graph” tab shows the number of series in the legend. If the count is wildly different from expectation, the matcher is wrong.

How it can fail

Five failure modes specific to label matchers:

1. Unanchored regex. {job=~"prometheus"} matches job="prometheus-extras". The fix is anchors: ^prometheus$ for exact, ^prometheus for “starts with”.
2. Quoting mistake. {job=prometheus} is a syntax error — the value must be a quoted string. The error reads parse error: unexpected identifier "prometheus".
3. Label name vs label value. {sum by (instance) (rate(...))} is a parse error; the label name goes outside { }.
4. Cardinality explosion. =~ on a label with millions of values (request IDs, trace IDs, user IDs) returns a vector with millions of series. The engine either returns it (and the panel times out) or times out mid-evaluation. The fix is upstream: drop or hash the label before scraping.
5. Unicode / case mismatch. RE2 is byte-oriented and case-sensitive by default. env=~"prod" does not match env="PROD". The fix is (?i) for case-insensitive matching or, better, canonicalisation at the exporter.

Security implications

Label matchers have one security-relevant surface: they read arbitrary series from the TSDB. An attacker who can submit arbitrary PromQL to the HTTP API can read any series that is not otherwise restricted. The mitigation is the same as the previous lesson: authentication on /api/v1/*, restrictive proxy rules, no anonymous admin access.

A second surface is regex cost. A regex matcher against a label that has high cardinality is a denial-of-service against the engine; treat the API as a privileged surface.

Performance implications

The cost of a selector is, in order of magnitude:

1. = and != — O(log n) postlist lookup per matcher. Cheap.
2. =~ and !~ against a low-cardinality label — O(k) where k is the number of distinct values. Cheap.
3. =~ and !~ against a high-cardinality label — O(n) where n is the total series count. Catastrophic.

The operational rule: regex matchers are for low-cardinality labels (job, instance, region, env). High-cardinality labels (request_id, trace_id, user_id) must be dropped or hashed before they enter the TSDB.

Verification

You should now be able to answer:

• What is the difference between = and =~ in a label matcher?
• Why is =~"prometheus" different from =~"^prometheus$"?
• Where do label names appear outside { }?
• Which matcher operator is the most expensive on a high-cardinality label, and why?

Quiz