Vector Matching Mistakes

A dashboard panel shows orders_failed / orders_total. The operator inspects the result and the rate is correct on most services. For one service — the one that has had zero failures in the last hour — the panel does not show a line. The operator assumes the service is “perfectly healthy” (no failures). The failure that occurred was simply not visible: the division operator applied 0 / 0 because the orders_ failed series had no samples for that instance, and the empty vector on one side of an arithmetic operator on the other side removes the entry from the output entirely.

The lesson is that vector matching is silent. Every binary expression in PromQL has a matching clause. The default matching clause is the one-to-one match on all labels. If the two vectors do not agree on labels, the result drops series silently. There is no error, no warning, no log line.

What it is

A vector matching mistake is a query pattern that uses a binary operator between two vectors and either:

• Chooses the wrong matching clause (on, ignoring, group_left, group_right).
• Omits a clause where one is required.
• Relies on the default one-to-one match when the data shape is many-to-many.
• Collides two labels with the same name on either side of the join.
• Propagates an empty vector across a comparison or arithmetic operator and silently drops series from the result.

The compounding factor is that the result compiles and evaluates. The shape of the failure is “wrong number of series” or “wrong labels on surviving series.” Both are discovered only when the operator inspects a panel.

Why a sysadmin cares

Vector matching is at the heart of every rate-over-error and every annotation-by-label query. The dashboard pattern that makes Grafana useful — “show me this metric, split by another metric” — is group_left and group_right. Get the matching clause wrong and the dashboard panel silently drops the dimension the operator cared about. The lesson covers the shapes that dominate production incidents.

How it works

A binary operator between two vectors has four shapes:

one-to-one (default)    A + B
   Match every series on the left with exactly one series
   on the right that shares every label. Series on either
   side without a match are dropped.

one-to-one              A + on(foo, bar) B
   Same as default but only the listed labels must match;
   other labels are independent.

many-to-one             A + group_left B
   For every series on the right, attach ALL matching
   series on the left. Drop series on the left without a
   right-side partner.

many-to-one (other way) A + group_right B
   Same idea but the left side is the "many" side.

many-to-many            (not directly supported)
   Either A or B must be on the "one" side of the join via
   group_left or group_right; otherwise the engine returns
   an empty vector for the whole expression.

The matching clauses:

# Default: one-to-one, all labels match
A / B

# Explicit on() — only the listed labels must match
A / on(job, instance) B

# Explicit ignoring() — all labels match except the listed
A / ignoring(version) B

# group_left — every series on the left keeps all matching
# series from the right, joined on the listed labels
A / on(job) group_left() B

# group_left with extra labels carried through to the output
A / on(job) group_left(version, cluster) B
# ^ the left-hand labels stay; the listed extra labels from
# the right are added to the left-hand output, joining on
# the listed on() labels

Three rules:

• The default A + B requires every pair of labels to agree. Two series with overlapping but not identical label sets silently produce a smaller output.
• group_left carries all matching series from the right side; if two series on the right have the same labels as one on the left, the output explodes (one entry per right match).
• group_left(label1, label2) adds the listed labels from the matched right-hand series to the output. If a label name exists on both sides, the right-hand value wins.

The boolean trap and the AND masking failure:

• Boolean trap. (orders_failed / orders_total) > 0.05 returns an empty vector when the right side has no match. The operator looking at the count of the result sees a smaller number than expected and concludes “failures dropped” when actually “failures never made it into the output.”
• AND masking. A == 1 and B == 1 returns an empty vector when either side is empty. The operator looking at count(...) to verify count ≥ 2 sees count = 0 and concludes “no hosts are healthy” when actually “the selector returned no series.”

Both shapes are the same root cause: vector matching drops silently; the count metric and the visual count diverge.

How to configure it

The configuration surface is the rules file and the dashboard JSON. Both expose the matching clause.

# /etc/prometheus/rules/sli.yml
groups:
  - name: sli.per-service
    interval: 30s
    rules:
      # Correct: arithmetic between two vectors with
      # matching on the join key. Both sides have explicit
      # group_left() to add the listed right-hand labels.
      - record: sli:checkout:error_ratio:5m
        expr: |
          sum by (job, instance) (
            rate(checkout_errors_total[5m])
          )
          / on (job, instance)
            sum by (job, instance) (
              rate(checkout_total[5m])
            )

      # Counterpart: percent-correct SLI for the
      # latency-side. Note the matching clause includes
      # the partition dimension.
      - record: sli:checkout:availability:5m
        expr: |
          1 - (
            sum by (job, instance) (
              rate(checkout_errors_total[5m])
            )
            / on (job, instance)
              sum by (job, instance) (
                rate(checkout_total[5m])
              )
          )

The dashboard panel that consumes the recording rule:

# Grafana dashboard JSON excerpt
panels:
  - type: timeseries
    title: 'Checkout availability per instance'
    datasource: prometheus
    targets:
      - expr: sli:checkout:availability:5m
        legendFormat: '{{instance}}'
        # No further group_left; the recording rule already
        # carries the per-instance label.
    options:
      legend:
        showLegend: true
        displayMode: 'table'
        calcs: ['lastNotNull']

For label collisions, the discipline is to re-aggregate with sum by (...) after the join so that the join only adds labels it explicitly carries:

# Right: explicit re-aggregation after the join
sum by (job, instance, status) (
  rate(http_requests_total[5m])
) / on (job, instance)
  count by (job, instance) (up) == 1

The fix for many-to-many without group_left:

# Wrong: many-to-many join, returns empty result
sum without (instance) (rate(http_requests_total[5m]))
+ sum without (instance) (rate(checkout_total[5m]))

# Correct: pick a side for the "group" position
sum without (instance) (rate(http_requests_total[5m]))
+ on(job) group_left() sum without (instance) (
  rate(checkout_total[5m])
)

How to validate it

Five checks confirm the matching pipeline is correct.

# 1. Inspect the LHS and RHS independently. Two queries;
#    the joining labels must agree and the row counts must
#    be what the operator expects.
curl -sf 'http://prometheus:9090/api/v1/query' \
  --data-urlencode 'query=count by (job, instance) (rate(checkout_errors_total[5m]))'
curl -sf 'http://prometheus:9090/api/v1/query' \
  --data-urlencode 'query=count by (job, instance) (rate(checkout_total[5m]))'
# Equal row counts across both queries means the join will
# have a match for every series; unequal counts means some
# series will be silently dropped.

# 2. Run the join expression and inspect the result count.
curl -sf 'http://prometheus:9090/api/v1/query' \
  --data-urlencode 'query=count(sli:checkout:error_ratio:5m)'
# A count lower than the smallest of the two inputs means
# matching dropped series.

# 3. Inspect the resulting labels. For the join to be
# useful, the labels on the output must include the
# partition the operator wanted.
curl -sf 'http://prometheus:9090/api/v1/query' \
  --data-urlencode 'query=sli:checkout:availability:5m'
# The metric block in the response carries the labels;
# inspect them for instance and job.

# 4. Validate fixture-level with promtool test rules.
promtool test rules /etc/prometheus/tests/sli_test.yml

# 5. Run the boolean-trap fixture: empty right side, see
# the resulting count.
# (Done in unit tests below.)

The boolean-trap fixture:

# /etc/prometheus/tests/sli_test.yml
rule_files:
  - /etc/prometheus/rules/sli.yml
evaluation_interval: 1m
tests:
  # Both sides populated; the join produces one series
  # per (job, instance) pair.
  - interval: 1m
    input:
      - series: 'checkout_errors_total{job="x",instance="a"}'
        values: '1+0x10'
      - series: 'checkout_total{job="x",instance="a"}'
        values: '100+0x10'
    promql_expr_test:
      - expr: 'sli:checkout:error_ratio:5m'
        exp_samples:
          - labels: 'sli_checkout_error_ratio_5m{job="x",instance="a"}'
            value: 0.01

How it can fail

1. The default match (A / B) silently drops series. A metric with an extra_label that the other side does not carry is dropped. Symptom: a panel that should show four services shows three. Detected by inspecting the input selectors and the output series count together.
2. Many-to-many without group_left. A A + B where both sides have more than one series per match-set. The result is the empty vector. Symptom: an alert with for: 5m for: A + B > 0 does not fire because the expression is always empty.
3. Wrong side of the join. A / on(job) group_left() B when the multiplicities are reversed; the join keys on the wrong side. Symptom: the result has the labels of the wrong side. Detected by inspecting the metric labels in the response.
4. Label collision in group_left. group_left(env) when the left side already has an env label. The right-hand value overwrites the left-hand value in the output. Symptom: the dashboard legend reads “env=prod” for services that are clearly dev. Detected by inspecting the labels in the response and comparing to the input labels.
5. Boolean trap. (numerator / denominator) > threshold when denominator is missing on a series. The division is removed, the threshold applies to the survivors. Symptom: the result is smaller than the input; count() of the result is smaller than the input count. Detected by comparing input and output counts.
6. AND masking. A == 1 and B == 1 returns an empty vector when either side is empty. Symptom: alerts that should fire do not; recorded alert state is no_data.

How to troubleshoot it

Diagnostic order:

1. Inspect both sides independently. Run each side of the join as its own query; compare row counts and label sets. If row counts differ, matching has dropped series.
2. Inspect the matching clause. Confirm on(...) lists exactly the labels that must match. Confirm group_left(...) lists exactly the labels that should be carried.
3. Inspect the output. Run the expression as an instant query. Inspect the response JSON for labels and values. The labels are what survives matching; the values are what the operator sees.
4. Replay against a fixture. promtool test rules evaluates the join against synthetic input and reports the output. The fixture’s labels are the contract that the join will produce the right shape.
5. Compare LHS row count to result count. count(A) and count(result). If count(result) < min(count(A), count(B)), matching dropped series.
6. Look for the boolean trap. When the result has fewer rows than expected and one side has zero rows, the trap has fired.

Security implications

The label collision shape is the only matching-clause failure with a security implication: a label collision can overwrite a security-relevant label on the output. The platform security part of this course returns to this surface under the “label collisions in alert annotations” section. The shorter version: do not put a label in group_left(...) that is going to clash with a label the operator already trusts in the alert.

Performance implications

The performance cost of binary operators is bounded by the cost of the matching step. The matching step is O(NM) where N is the left-hand series count and M the right-hand series count. For group_left, the cost is O(M(number of left- hand matches)) per matched pair. The shapes that explode:

• group_left without a selector on the right — many-to-many matches each return one series per combination.
• Many-to-many by accident — both sides have many series per match-set.

A left-hand of 100k series joined to a right-hand of 100k series with no further selection is 10B comparisons per query. The --query.max-samples cap catches most of these; the production default of 50M samples is the line where the engine refuses.

Production guidance

• Always write the matching clause explicitly. Never rely on the default A + B in production code.
• Re-aggregate both sides to a known label set before the join. sum by (job, instance) (rate(x[5m])) is the conventional shape.
• Use on(...) with a specific list. The list documents the join’s contract and catches accidental label drift.
• Add a unit test against a fixture that exercises both the empty-vector case and the label-collision case.
• Use sum of right-hand output in group_left to collapse before re-emission; the output vector should be bounded by the left-hand side, not the right.

Verification

You should now be able to answer:

• What is the default matching clause for a binary operator between two vectors, and why is it the silentest failure shape in PromQL?
• When is group_left required, and when is on(...) sufficient?
• What is the boolean trap, and what alert shape catches it?
• What is the production discipline for label collisions in group_left(...)?
• Why must every binary operator be unit-tested against fixtures on both sides of the empty-vector boundary?

Quiz