Vector Matching

A panel that should show CPU usage per pod shows CPU usage per instance with the right number. The team wrote the query as sum(rate(node_cpu_seconds_total[5m])) / on(instance) kube_pod_info. The “instance” label on node_cpu_seconds_total is the host; the “instance” label on kube_pod_info is the node name. Sometimes they match; sometimes they don’t. The panel is intermittently empty.

Vector matching is the layer that decides which series on the left pair with which series on the right. The previous lesson introduced the operators; this lesson is the matching rules they share.

What it is

Every binary operator in PromQL has a matching policy:

• The default — pair series that have identical values on every label they share. Series whose labels differ on any shared label do not pair.
• on(label_list) — pair only on the listed labels.
• ignoring(label_list) — pair on every shared label except the listed ones.

Set operators (and, or, unless) follow the same rules for matching; the difference is what they emit when a match is found. The full grammar is:

<vector_expr> <operator> <vector_expr> [ on | ignoring ( <label_list> ) ]
<vector_expr> <set_op> <vector_expr> [ on | ignoring ( <label_list> ) ]
[ * on | ignoring ( <label_list> ) group_left | group_right [( <labels> ) ] ]

The matching modifier is between the two operands. The grouping modifier (group_left / group_right) appears after the matching modifier and is paired with a set operator (and, or, unless); it cannot appear after +, -, /, etc.

The three matching shapes

One-to-one
  Left:   {a=1}, {a=2}, {a=3}
  Right:                {a=3}, {a=4}, {a=5}
  Match:                                {a=3}
  Result:  one output series per matched pair

One-to-many (group_left)
  Left:   {a=1}, {a=2}
  Right:        {a=2, b=10}, {a=2, b=20}, {a=2, b=30}
  Match:                a=2 pairs with all three right series
  Result:  three output series on the left, each with b=X appended

Many-to-many (set operators, default matching)
  Left:   {a=1}, {a=2}, {a=3}
  Right:                       {a=2}, {a=3}, {a=4}
  Match:                a=2, a=3
  Result:  a=2 and a=3 emitted by and/or/unless as the rule dictates

The one-to-one case is the default for arithmetic and comparison. The one-to-many case is what group_left/group_right produce. The many-to-many case is what set operators do without the group modifier: each side is treated as a set of series and the set-theoretic operation applies to the matched series.

Why a sysadmin cares

Vector matching is the silent failure of binary operators. The most expensive form of “the panel reads no data” is a join that produces zero output because the labels do not match the way the operator assumed.

Three common production incidents and their causes:

1. Pod-to-node ratio is zero. node_cpu_seconds_total has labels {instance, cpu, mode, job}. kube_pod_info has {namespace, pod, instance, ...}. Joining on the default matches on instance and job — kube_pod_info does not have job. Result: zero matches. The fix is on(instance).
2. CPU usage per service is missing the namespace. Two metrics with service labels but different namespace conventions. The default matches on every common label; namespace differs and the join fails. The fix is ignoring(namespace).
3. Group-left cardinality explosion. kube_service_info is “one”; kube_pod_info is “many”. The wrong grouping direction produces N pods per service in the output and a legend with thousands of series. The fix is to use group_left with the smaller side on the left.

How it works

The evaluator’s vectorBinop (and vectorBinopSet for set operators) computes a matching key per series. The default matching key is the tuple of every label both sides share. With on(label_list), the key is restricted to that list. With ignoring(label_list), the key is the shared labels minus the listed ones.

The implementation is a hash join. The right side is hashed by key; the left side is iterated; each left series probes the hash table for matching right series. The arithmetic, comparison or set operation runs per matched pair.

For group_left / group_right, the implementation walks the left side (or right side, respectively) and emits one output series per matched right side (or left side). The optional (labels) argument names the right-side labels that should be appended to the left side; without it, every right-side label not in the matching key is appended.

How to configure it

Vector matching is in the expression. The relevant operational configuration is the label set at scrape time. Two levers:

# prometheus.yml — make the labels consistent across the fleet
scrape_configs:
  - job_name: kubernetes-pods
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      # Every pod scrape gets a cluster label so joins across
      # metrics from different jobs can match on (cluster, namespace, pod).
      - source_labels: [__meta_kubernetes_cluster_name]
        target_label: cluster
        action: replace
      # Standardise instance: use the pod IP, not the kubelet URL
      - source_labels: [__meta_kubernetes_pod_ip]
        target_label: instance
        action: replace

A team that joins across jobs without a common cluster label will see different label sets per scrape job and the join will fail intermittently. The label set at scrape time is the foundation of vector matching at query time.

How to validate it

The /api/v1/series endpoint returns the full label set of matching series. To validate that two metrics share the label keys you intend to join on:

# Show the label set of metric_a's first series
curl -s --data-urlencode 'match[]=metric_a' \
  http://localhost:9090/api/v1/series \
  | jq '.data[0] | keys'

# Show the label set of metric_b's first series
curl -s --data-urlencode 'match[]=metric_b' \
  http://localhost:9090/api/v1/series \
  | jq '.data[0] | keys'

The intersection of those two key lists is the default matching key. If the intersection is empty, the default join returns an empty vector; you need on(...) with labels that actually match.

To validate a join before shipping:

# Confirm the join produces the expected number of series
curl -s --data-urlencode 'query=metric_a / on(instance) metric_b' \
  http://localhost:9090/api/v1/query \
  | jq '.data.result | length'

# Confirm group_left produces the expected enrichment
curl -s --data-urlencode 'query=metric_a * on(instance) group_left(service) metric_c' \
  http://localhost:9090/api/v1/query \
  | jq '.data.result[0]'

The first call’s count should match the expected cardinality of the result. The second call’s first row should have the new service label populated.

How it can fail

Five failure modes:

1. Empty default match. Two metrics share only __name__ (which is never part of a matching key) and one other label that means different things on each side. The default join is empty. The fix is on(instance), on(namespace, pod), or ignoring(dimension) — and a label-standards audit upstream.
2. Grouping direction reversed. A many-to-one join written with the many side on the left and group_right produces N series per “one” instead of 1. The fix is to flip the sides or change to group_left.
3. ignoring over-broad. ignoring(__name__) matches on every label except __name__. __name__ is not in the matching key anyway; the modifier is a no-op. The trap is that ignoring quietly broadens the match, which can pair series that should not be paired. Prefer on(...) for precision.
4. Group-left label explosion. group_left() with no argument appends every right-side label to the left. A right side with 50 labels produces a result series with 50 extra labels. The fix is group_left(specific_label) with the labels you actually need.
5. Stale series during a deploy. A restart drops the old series and creates a new one with a fresh start-time timestamp; the join between the two halves of a rolling restart misses samples at the boundary. The fix is the for: clause on the alert, or a recording rule that smooths the transition.

Security implications

Vector matching has no security surface of its own. The denial-of-service concern is unchanged: a join that multiplies two large vectors exhausts engine resources. Validate cardinality before shipping; pre-aggregate where you can.

Performance implications

The cost of a join is roughly (left + right) * (matching cost). The matching cost is a hash table probe per series on the left. The dominant cost is usually the output vector size: a many-to-many join that produces millions of series is the expensive case. Aggregate the larger side first; the join then runs on smaller vectors and produces a smaller result.

For dashboards that join two metrics on every refresh, the recording-rule discipline is: pre-aggregate each side into its own rule, then join the rules. The dashboard reads from the joined rule.

Verification

You should now be able to answer:

• What is the difference between on(labels) and ignoring(labels)?
• When does the default matching produce an empty join, and how do you fix it?
• Why does group_left() (with no argument) sometimes produce a result with many more labels than expected?
• How do you decide which side of a group_left/group_right is the “one”?

Quiz