Lab: PromQL Selectors and Matchers

Objective

A selector is the cheapest thing in PromQL to write and the most expensive thing to get subtly wrong, because a wrong selector does not error. It returns series — just not the ones you meant — and every function, aggregation and alert threshold downstream operates faithfully on the wrong set.

This lab removes the guessing. You build the label space yourself, so you know exactly which four targets exist and exactly which labels each one carries. Then you run each matcher against it and count. By the end you will have measured, not assumed, the answer to the two questions that cause most selector bugs in production: is a regex matcher anchored, and what does a matcher do to a series that does not carry the label at all.

Architecture

One Prometheus and four node_exporter containers. The exporters are interchangeable — nothing in this lab depends on what they measure. They exist to be targets, and the whole point is the label set each one is given by the scrape configuration.

                       +---------------------------+
                       |  prometheus 2.55  :9090   |
                       |  4 scrape_configs         |
                       +-------------+-------------+
                                     | scrape every 15s
        +--------------+-------------+-------------+--------------+
        v              v                           v              v
  +-----------+  +--------------+          +---------------+  +-----------+
  | exp-api   |  | exp-gateway  |          | exp-payments  |  | exp-batch |
  +-----------+  +--------------+          +---------------+  +-----------+
  job=api        job=api-gateway           job=payments-api   job=batch
  env=prod       env=prod                  env=production     (no env label)
  team=payments  team=Edge                 team=payments      team=Data

Three deliberate collisions are built into that table, and each one is a bug that has shipped to production somewhere:

1. Three of the four job names contain the substring api. This is what makes the anchoring question answerable by counting.
2. env is spelled prod on two targets and production on a third. Two teams, two conventions, one label name.
3. exp-batch has no env label at all. This is the case every selector author forgets, and it is the one that pages the wrong team.

Requirements

• A Linux host with Docker Engine 28.x and Docker Compose v2, and network access on first run to pull two images (roughly 300 MiB total).
• curl and jq on the host. Every measurement in this lab is a Prometheus HTTP API call parsed with jq; nothing is read off a dashboard.
• Free TCP ports 9090 and 9101 on the host. The other three exporters are reachable only inside the compose network, which is all this lab needs.
• Roughly 200 MiB of memory and a few hundred MiB of disk for the TSDB.
• No out-of-band access requirement. The lab touches nothing outside its own directory and compose project, and changes no host networking or firewall state.

Scenario

A platform team has one dashboard variable and one alert route, both filtered by job. The dashboard was written a year ago with the selector {job=~"api"} and has shown a single service ever since.

Last week a reviewer flagged it. They had read — correctly, about regular expressions in general — that a pattern with no anchors matches a substring, and concluded that the existing filter was a latent bug that would start matching api-gateway and payments-api as soon as anyone looked at it. They changed it to what they believed was the equivalent-but-explicit form, {job=~".*api.*"}, and shipped it.

The next morning the panel showed three services summed into one line, the error-ratio number dropped by a third because batch traffic diluted it, and an alert that had never fired in a year fired against the wrong owner.

Your job is to establish, by counting series against a label space you control, which of the two selectors was correct and why — and then to work out what the reviewer should have written if they had wanted a substring match.

Tasks

Task 1: Build the label space

LABDIR="$HOME/rb-obs-selectors"
mkdir -p "$LABDIR"
cd "$LABDIR"

prometheus.yml. Read this file as the answer key: it is the complete, authoritative statement of which labels exist in this TSDB.

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  # Three job names that all contain the substring "api". The whole
  # anchoring question in Task 3 is answerable because of this.
  - job_name: api
    static_configs:
      - targets: ['exp-api:9100']
        labels:
          env: prod
          team: payments

  - job_name: api-gateway
    static_configs:
      - targets: ['exp-gateway:9100']
        labels:
          env: prod
          team: Edge          # capital E — Task 5 depends on it

  # Same environment as the two above, spelled by a different team.
  - job_name: payments-api
    static_configs:
      - targets: ['exp-payments:9100']
        labels:
          env: production
          team: payments

  # No env label at all. This is the target every selector forgets.
  - job_name: batch
    static_configs:
      - targets: ['exp-batch:9100']
        labels:
          team: Data

compose.yaml:

name: rb-obs-selectors

x-exporter: &exporter
  image: prom/node-exporter:v1.8.2
  restart: unless-stopped

services:
  exp-api:
    <<: *exporter
    container_name: rb-sel-exp-api
    ports:
      - '9101:9100'      # the only exporter published to the host

  exp-gateway:
    <<: *exporter
    container_name: rb-sel-exp-gateway

  exp-payments:
    <<: *exporter
    container_name: rb-sel-exp-payments

  exp-batch:
    <<: *exporter
    container_name: rb-sel-exp-batch

  prometheus:
    image: prom/prometheus:v2.55.1
    container_name: rb-sel-prom
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention.time=2h'
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prom-data:/prometheus
    ports:
      - '9090:9090'
    depends_on:
      - exp-api
      - exp-gateway
      - exp-payments
      - exp-batch

volumes:
  prom-data:

$ docker compose up -d

Give it about a minute, then confirm all four targets are being scraped successfully. A selector experiment run against a half-populated TSDB produces counts you will spend twenty minutes trying to explain:

curl -sf http://localhost:9090/-/ready && echo PROMETHEUS-READY

curl -sG http://localhost:9090/api/v1/query --data-urlencode 'query=up' \
| jq -r '.data.result[] | "\(.metric.job)\t\(.metric.instance)\tup=\(.value[1])"' | sort

Four rows, every one ending up=1. If a row is missing, the exporter has not started; if a row reads up=0, Prometheus can see the config but not the container.

Task 2: Read the label space before you query it

Before any matcher, ask the TSDB what it holds. These two endpoints are the cheapest tools in the entire toolkit and almost nobody reaches for them, because the expression browser is right there:

# Every label name in the TSDB.
curl -s http://localhost:9090/api/v1/labels | jq -r '.data[]' | head -20

# Every distinct value of one label name.
for L in job env team; do
  printf '%s: ' "$L"
  curl -s "http://localhost:9090/api/v1/label/$L/values" | jq -c '.data'
done

job has four values, env has two, team has three. Notice what the env list does not tell you: that one target has no env label at all. The label-values endpoint enumerates values that exist, and an absent label has no value to enumerate. That blind spot is the whole of Task 4.

Now the endpoint that answers “what would this selector match” without evaluating anything:

curl -sG http://localhost:9090/api/v1/series \
  --data-urlencode 'match[]=up' \
| jq -r '.data[] | "\(.job)\tenv=\(.env // "(none)")\tteam=\(.team)"' | sort

Four rows, and exp-batch reports its env as absent because jq was told to say so. This is the shape of the target population every later count is drawn from. Keep it on screen.

Task 3: Count what each job matcher selects

Here is the experiment. Six selectors over the same four targets. Write down your prediction for each count before you run it — the value of this task is in the predictions you get wrong.

for SEL in \
  'up{job="api"}' \
  'up{job=~"api"}' \
  'up{job=~".*api.*"}' \
  'up{job=~"api.*"}' \
  'up{job=~".*api"}' \
  'up{job!~".*api.*"}' ; do
  N=$(curl -sG http://localhost:9090/api/v1/query \
        --data-urlencode "query=count($SEL)" \
      | jq -r '.data.result[0].value[1] // "0"')
  printf '%-26s -> %s series\n' "$SEL" "$N"
done

The counts:

Row two is the whole lab. job=~"api" selected one series, not three. Prometheus wraps every regex matcher in start and end anchors before compiling it, so =~"api" is evaluated as =~"^api$" and behaves identically to ="api" on this data. The reviewer in the scenario was applying a true fact about regular expressions in general to a matcher that is not general.

Confirm the equivalence directly rather than taking the counts on trust:

curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=count(up{job=~"api"}) - count(up{job=~"^api$"})' \
| jq -r '.data.result[0].value[1]'

Zero: writing the anchors changes nothing, because they were already there.

Task 4: The target with no label

Now the second question. exp-batch has no env label. Run the four env matchers and see where it lands:

for SEL in \
  'up{env="prod"}' \
  'up{env!="prod"}' \
  'up{env=""}' \
  'up{env!=""}' \
  'up{env=~"prod"}' \
  'up{env=~"prod.*"}' ; do
  printf '%-24s -> ' "$SEL"
  curl -sG http://localhost:9090/api/v1/query --data-urlencode "query=$SEL" \
  | jq -r '[.data.result[].metric.job] | sort | join(", ")'
done

The rule underneath every row: a label that is absent is treated as the empty string. batch has no env, so its env is "", and "" != "prod" is true — which is why env!="prod" returns it. env="" selects it explicitly, and env!="" is the matcher that excludes it.

Row five is worth a second look after Task 3: env=~"prod" does not match env="production", because the anchors make it exact. If you want both spellings you have to say so, either with env=~"prod.*" or, better, by fixing the label at its source.

Task 5: Case, and the one flag worth knowing

RE2 is case-sensitive. team was deliberately given mixed-case values, so the consequence is countable:

for SEL in 'up{team="Edge"}' 'up{team=~"edge"}' 'up{team=~"(?i)edge"}'; do
  printf '%-26s -> ' "$SEL"
  curl -sG http://localhost:9090/api/v1/query --data-urlencode "query=count($SEL)" \
  | jq -r '.data.result[0].value[1] // "0"'
done

One, zero, one. (?i) at the front of the pattern turns off case sensitivity for the rest of it, and it is the only RE2 flag you will reach for often.

Treat it as a diagnostic, not a fix. A (?i) in a production selector is a label-hygiene problem wearing a costume: somewhere upstream, two producers disagree about capitalisation, and every consumer of that label now has to know. Canonicalise at the exporter or in metric_relabel_configs, and let the selectors stay literal.

Task 6: The selector that is not allowed

A selector has to select something before the engine will accept it. Try the one that does not:

curl -sG http://localhost:9090/api/v1/query --data-urlencode 'query={env=""}' \
| jq -r '{status, errorType, error}'

Read the error text. The rule it is enforcing: a vector selector must contain either a metric name or at least one matcher that cannot match the empty string. {env=""} matches the empty string by definition, so on its own it would select every series in the database that lacks an env label — a query whose cost is the size of the TSDB and whose intent is almost never what the author meant.

Adding a metric name satisfies the rule, because a name is itself a matcher on __name__ that cannot be empty:

curl -sG http://localhost:9090/api/v1/query --data-urlencode 'query=up{env=""}' \
| jq -r '.data.result[] | .metric.job'

And __name__ can be matched like any other label, which is how you select across metric families:

curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=count by (__name__) ({__name__=~"node_filesystem_.+", job="api"})' \
| jq -r '.data.result[] | "\(.metric.__name__)\t\(.value[1])"' | sort

Task 7: Measure what an unbounded selector costs

The Prometheus HTTP API will report the cost of any query you ask it to run. Look at the whole stats object once so you know what is on offer:

curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=up' \
  --data-urlencode 'stats=all' \
| jq '.data.stats'

Then compare a bounded selector against the unbounded one. {__name__=~".+"} is legal — .+ cannot match the empty string, so it satisfies Task 6’s rule — and it selects every series in the database:

for Q in 'up' 'up{job=~".*"}' '{__name__=~".+"}'; do
  printf '%-20s ' "$Q"
  curl -sG http://localhost:9090/api/v1/query \
    --data-urlencode "query=$Q" --data-urlencode 'stats=all' \
  | jq -r '"series=\(.data.result | length)  eval=\(.data.stats.timings.evalTotalTime)s"'
done

Two things to take from the numbers, and only two — at four targets the rest is noise:

• up and up{job=~".*"} cost about the same. The regex is anchored and applied to a label with four distinct values, so the work it adds is four regex evaluations. The received wisdom that “regex matchers are slow” is not a property of =~; it is a property of =~ on a label with a large value set, where the engine must test the pattern against every distinct value.
• {__name__=~".+"} costs orders of magnitude more, because the selector’s bound is the whole database rather than one metric family. Selector cost tracks how much of the TSDB the matcher lets through, not which operator produced it.

Task 8: The same selector, two shapes

The last shape question, and the one behind a large share of “the expression browser shows data and the panel says No data” tickets:

for Q in 'up{job="api"}' 'up{job="api"}[5m]'; do
  printf '%-22s -> ' "$Q"
  curl -sG http://localhost:9090/api/v1/query --data-urlencode "query=$Q" \
  | jq -r '"resultType=\(.data.resultType)  keys=\(.data.result[0] | keys | join(","))"'
done

The first is a vector whose single result carries value — one sample. The second is a matrix whose result carries values — every sample in the last five minutes, about twenty of them at a 15-second scrape interval. Same selector, same series, two different objects.

A Grafana time-series panel plots an instant vector, evaluated once per pixel column. Hand it a matrix and there is nothing to plot, so the panel says “No data” while the API is returning a healthy response. The [5m] suffix belongs inside a function that consumes it, and nowhere else:

curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=up{job="api"}[5m] / 2' \
| jq -r '{status, errorType, error}'

Read that error too. Arithmetic operators take instant vectors and scalars; there is no rule for dividing a matrix by two, so the engine refuses rather than guessing.

Validation

Four checks. Each proves a claim from the tasks rather than repeating a step.

1. The label space is the one you designed. Four targets, all up, with exactly one carrying no env label:

curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=count(up) - count(up{env!=""})' \
| jq -r '"targets with no env label: \(.data.result[0].value[1])"'

Must print 1.

2. Regex matchers are anchored. The bare pattern and the explicitly anchored pattern select an identical set, and the substring form does not:

curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=count(up{job=~"api"}) == count(up{job=~"^api$"})' \
| jq -r '"anchored equivalence holds: \(.data.result | length == 1)"'

curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=count(up{job=~".*api.*"}) - count(up{job=~"api"})' \
| jq -r '"extra series a substring match would add: \(.data.result[0].value[1])"'

true, then 2. That second number is the size of the incident in the scenario.

3. The three selectors from the scenario are written and validated. Use promtool inside the Prometheus container, which is the same check without the curl/jq scaffolding and the form that belongs in CI:

docker compose exec prometheus promtool query series \
  http://localhost:9090 --match='up{job=~"api"}'

docker compose exec prometheus promtool query series \
  http://localhost:9090 --match='up{job=~".*api.*"}'

docker compose exec prometheus promtool query series \
  http://localhost:9090 --match='up{env!="prod", env!=""}'

One series, three series, one series. The third selector is the corrected form of “everything outside prod”: without the env!="" clause it also returns batch, which has no env label and no opinion about the question. Record all three selectors and what each one is for.

4. An unbounded selector is measurably more expensive. Compare the two queries from Task 7 and confirm the direction of the gap, not its size:

for Q in 'up' '{__name__=~".+"}'; do
  curl -sG http://localhost:9090/api/v1/query \
    --data-urlencode "query=$Q" --data-urlencode 'stats=all' \
  | jq -r --arg q "$Q" '"\($q): \(.data.result | length) series"'
done

Expected Outcome

• Five containers running: one Prometheus with four healthy targets whose labels you can read straight off prometheus.yml.
• A recorded table of six job matchers and their series counts, establishing that =~ is anchored and that a substring match must be written .*x.*.
• A recorded table of six env matchers, establishing that an absent label behaves as the empty string and that env!="" is the matcher that excludes it.
• A stats measurement showing that selector cost tracks how much of the TSDB the matcher admits, not which operator was used.
• Three validated selectors you would be willing to put in a dashboard variable, an alert rule, and a routing tree respectively.

Troubleshooting

• A target reads up=0. Prometheus parsed the config but could not reach the exporter. Check the container is running with docker compose ps; the target address in prometheus.yml must be the compose service name and port 9100, not a host port.
• All counts come back as 0. The TSDB has no samples yet. Wait a full minute after up -d and re-run; count() over an empty selector legitimately returns no result at all, which the // "0" in the loop renders as zero.
• jq prints null for a label. The series does not carry that label. That is the Task 4 case, not an error — the // "(none)" fallback exists to make it visible.
• Prometheus exits immediately after up -d. A malformed prometheus.yml is fatal at startup. docker compose logs prometheus names the file and line. YAML indentation under static_configs is the usual cause: labels: is a sibling of targets:, both nested under the list item.
• The exporter logs collector errors. node_exporter in a container cannot read some host subsystems and says so at startup. It is expected here and irrelevant: this lab uses the exporter as a labelled target, not as a source of host truth.
• A count disagrees with the table by exactly one. You are probably counting a metric other than up. Every exporter emits thousands of series; up emits exactly one per target, which is why it is the right metric for a counting experiment.

Cleanup

Everything the lab created is one directory, one compose project and one named volume.

$ cd ~/rb-obs-selectors && docker compose down -v

docker volume ls | grep rb-obs-selectors || echo "volumes gone"
rm -rf "$HOME/rb-obs-selectors"

Nothing else on the host was modified: no host networking, no firewall rules, no files outside the lab directory. The two pulled images remain in the local image cache and can be left there.

Production notes

Default to = and !=. They are exact, they read as exact, and they use the index directly. Reach for =~ when you need alternation or a genuine pattern, and remember that alternation is the common case: env=~"prod|dr" is clearer and cheaper than two rules.

Write the anchors even though they are implied. job=~"^api$" and job=~"api" behave identically, but the first one tells the next reader that you knew. The reviewer in the scenario was not careless; they were applying a correct general rule in a place where it does not hold, and an explicit anchor is the cheapest way to say “this was considered”.

Every != on a label needs a decision about the unlabelled case. Before merging a selector containing !=, ask what it does to a target that lacks the label. If the answer is “includes it, and that is wrong”, add the ,label!="" clause. This is a code-review question, not a testing question — the unlabelled target usually does not exist yet when the selector is written.

Validate selectors in CI against a real TSDB, not by eye. The check in Validation step 3 is one command:

# Substitute your own values before running:
PROM=http://prometheus.example.com:9090
SELECTOR='up{job=~"api"}'
EXPECTED=1

FOUND=$(promtool query series "$PROM" --match="$SELECTOR" | grep -c .)
test "$FOUND" -eq "$EXPECTED" \
  || echo "selector drift: expected $EXPECTED series, found $FOUND"

Run it against staging on every dashboard or rule change. It catches the class of bug where a selector is still syntactically valid after a label is renamed upstream and quietly matches nothing.

Fix label spelling at the source, not in the selector. env=~"prod.*" and (?i) are both signs that two producers disagree. Every consumer downstream now has to carry that knowledge, and one of them eventually will not. Normalise in metric_relabel_configs at the scrape, where the fix is written once.

Keep the matcher’s bound small. Selector cost tracks how much of the TSDB the matcher lets through. A regex on job is cheap because job has tens of values; the same regex on a label with a million values is a different operation with the same syntax. Drop unbounded labels at scrape time so no selector can be written against them.

What You Learned

• Prometheus wraps every regex matcher in anchors, so =~"api" is exact and a substring match must be spelled .*api.*. You measured it: one series versus three, over a label space you built.
• An absent label is the empty string. env!="prod" returned the target that has no env at all, env="" selected it alone, and env!="" was the matcher that kept it out.
• /api/v1/series answers the selector question; /api/v1/query answers the value question. An empty series response means the matcher is wrong, with none of the “maybe there is just no sample right now” ambiguity.
• A selector must contain one matcher that cannot be empty, which is why a metric name is not merely conventional, and why {__name__=~".+"} is legal and dangerous at the same time.
• Selector cost is about the bound, not the operator. An anchored regex over a four-value label costs what an equality match costs; an unbounded selector costs the database.