Binary Operators

A panel that should answer “what fraction of requests are 5xx” shows zero or shows a nonsense ratio. The query is http_requests_total{status="500"} / http_requests_total. The arithmetic looks right; the operator silently dropped half the series because the labels did not match. The next lesson covers matching in depth; this lesson is the operators themselves and the shapes they accept.

PromQL has three families of binary operator: arithmetic, comparison, and set. Each behaves differently on vectors. The trap that catches most operators is forgetting which one they are.

What they are

The three families:

The shape constraints are strict. A range vector on either side of an arithmetic operator is a parse error. A scalar on one side and a vector on the other is allowed and broadcasts the scalar to every series.

Arithmetic works element-wise between matching series:

# Rate of 5xx as a fraction of total
sum(rate(http_requests_total{status="500"}[5m]))
  /
sum(rate(http_requests_total[5m]))

The two sides are both instant vectors. The left has one series; the right has one series (after the inner sum). The division yields one series.

Comparison operators, by default, emit 0 (dropped from the result) or 1 (kept with the original sample value):

# Boolean comparison: 1 where condition holds, 0 otherwise
up == 1
node_cpu_seconds_total > 100

The bool modifier changes the semantics: instead of emitting 0/1 with the sample value, the operator drops series where the comparison is false:

# Filter to series with value greater than 100
node_cpu_seconds_total > bool 100

# Useful for alerting: only alert when the metric exists and is high
up == bool 0

The bool form is the right tool for “give me the series that match this condition”. The non-bool form is the right tool for “give me a 0/1 indicator”.

Set operators combine series sets without arithmetic:

• and — series that appear in both sides.
• or — series that appear in either side (union).
• unless — series in the left side that do not appear in the right.

# Services that are up AND have CPU above 80%
(up == 1) and (rate(node_cpu_seconds_total{mode!="idle"}[5m]) > 0.8)

# Services that are up OR have CPU above 80%
(up == 1) or (rate(node_cpu_seconds_total{mode!="idle"}[5m]) > 0.8)

# Services that are NOT in the maintenance window
up == 1 unless on(instance) maintenance_window

and is the most common operator in alert rules; or and unless are less common but useful when an alert should fire for either of two conditions or suppress during a window.

group_left and group_right

group_left and group_right extend the set operators. They say “for each series on the ‘many’ side, attach all matching metadata from the ‘one’ side”. The classic example is attaching a human-readable service name from a kube_service_info metric to a per-pod metric:

# For each pod, attach the service it belongs to
rate(container_cpu_usage_seconds_total[5m])
  * on(namespace, pod) group_left(service)
  kube_pod_labels{label_service=~".+"}

The * (multiplication by 1) is a no-op arithmetic; the work is the on(namespace, pod) group_left(service) modifier. The modifier says:

• on(namespace, pod) — match on these two labels.
• group_left(service) — keep all matching service values from the right side as new labels on the left side.

group_right is the same thing with the sides reversed: keep the right side’s series and pull metadata from the left.

The “many-to-one” naming comes from the typical relationship: many pods per service, so the right side is the “many” set and the left side is the “one” set that gets enriched.

Why a sysadmin cares

The bool modifier is the difference between “alert when this is true” and “alert when the metric exists and is high”. Without bool, a comparison like cpu_usage > 80 returns one series per CPU with a value of 1 when the comparison is true and 0 when false; the alert fires because the value is 1. With bool, the series with the comparison false are dropped; the alert only sees the series that breach the threshold, which is usually what you want.

The and operator is the difference between “alert when both conditions hold on the same series” and “alert when both conditions hold on the platform”. The default matching of and is on every label; if the two conditions produce series with different label sets, the matching is empty and the alert is silent. on(...) is the fix.

The group_left operator is the difference between “see the pod metric” and “see the pod metric with the service name attached”. Service-level dashboards typically want the latter; the metric exporter does not always provide it natively, so the join happens in PromQL.

How it works

The evaluator walks the binary operator node. For arithmetic and comparison, the right side is evaluated first (post-order), then the left, then the operator applies per-series.

For set operators, the evaluator materialises both sides as sets of series and applies the set-theoretic operation. The default matching uses the intersection of the label sets; on(...) and without(...) narrow or widen that intersection. The next lesson covers matching in depth; this lesson treats the operators.

For group_left / group_right, the evaluator:

1. Evaluates the right side (the “many” set, by convention).
2. For each series on the left (the “one” set), finds every series on the right that matches on the on(...) labels.
3. Emits one result series per matching left/right pair, with the right side’s listed labels merged into the left.

If the left side is “one-to-many” with the right, this can multiply the output series count. The discipline is to make the “one” side the one with fewer series.

How to configure it

Binary operators are part of the expression; the relevant operational configuration is what makes the inputs safe to combine. Two pieces:

# prometheus.yml — make labels uniform across the fleet
scrape_configs:
  - job_name: api
    static_configs:
      - targets: ['api-1:9100', 'api-2:9100']
    relabel_configs:
      # Add a consistent service label to every series
      - source_labels: [__meta_kubernetes_service_name]
        target_label: service
        action: replace

A common production mistake is to combine two metrics whose label sets diverge — service on one side, app on the other. The join is empty; the panel reads zero. The fix is at the scrape boundary, not in the query.

How to validate it

The HTTP API exposes the operator behaviour directly. To validate a comparison, query both sides and the combined expression:

# Confirm the left side returns the expected series
curl -s --data-urlencode 'query=up == 0' \
  http://localhost:9090/api/v1/query | jq '.data.result'

# Result is a vector with the value 1 for down targets, 0 for up
# targets — a one/zero indicator. The panel will plot 0 and 1.

# Same query with bool modifier
curl -s --data-urlencode 'query=up == bool 0' \
  http://localhost:9090/api/v1/query | jq '.data.result | length'

The bool form drops the zero entries; the result is the down-targets-only vector.

For a group_left join:

# Confirm the left side has the series you expect
curl -s --data-urlencode 'query=count by(namespace, pod) (kube_pod_info)' \
  http://localhost:9090/api/v1/query | jq '.data.result | length'

# Confirm the join produces the expected enrichment
curl -s --data-urlencode 'query=count by(namespace, pod, service) (
  kube_pod_info * on(namespace, pod) group_left(service) kube_pod_labels
)' \
  http://localhost:9090/api/v1/query | jq '.data.result[0:2]'

The result is one series per pod with service populated from the matching labels.

For set operators, validate the cardinality directly:

# Series that are up AND scraping
curl -s --data-urlencode 'query=(up == 1) and (scrape_samples_scraped > 0)' \
  http://localhost:9090/api/v1/query | jq '.data.result | length'

# Series that are up OR scraping (probably most of the fleet)
curl -s --data-urlencode 'query=(up == 1) or (scrape_samples_scraped > 0)' \
  http://localhost:9090/api/v1/query | jq '.data.result | length'

A query whose and result is empty and whose or result is the size of one side alone is the label-mismatch failure.

How it can fail

Five failure modes:

1. Range vector on either side. metric[5m] / 100 is a parse error: expected instant vector. Wrap the range vector in a function first.
2. Silent label mismatch. metric_a / metric_b with no on(...) matches on every label both sides share. If the two metrics share only {instance}, the join is series-wise per instance. If they share nothing, the join is empty.
3. Bool vs non-bool confusion. A comparison without bool emits 0/1 with the sample value; the panel plots both. With bool, the false series are dropped. For alerts, bool is almost always the right form.
4. group_left cardinality explosion. A left side with one series per service joined to a right side with 10 pods per service produces 10 series per service in the result. The panel may render all 10; the dashboard legend explodes. The fix is to aggregate the right side first.
5. Set operator with stale series. and on two series where one is stale at t returns nothing for that series at t. The alert fires intermittently as series come and go. The fix is the staleness marker (- suffix) or a recording rule with for:.

Security implications

Binary operators do not introduce a new surface. The denial-of-service concern remains: a query that combines two high-cardinality metrics with no aggregation produces a result vector too large for the engine to evaluate in the timeout. The discipline is the same — pre-aggregate into recording rules.

Performance implications

The cost of a binary operator is proportional to the size of the left and right vectors. metric_a / metric_b with metric_a having 100 series and metric_b having 100 series is up to 10,000 comparisons. The group_left form with 100 left and 1000 right is up to 100,000 comparisons. The lever is aggregation:

# Cheap: aggregate first, then divide
sum without(cpu) (rate(cpu_seconds_total[5m]))
  / on(instance) group_left
sum without(cpu) (rate(cpu_seconds_count[5m]))

Verification

You should now be able to answer:

• What is the difference between cpu > 80 and cpu > bool 80?
• What does and do when the two sides have no common labels?
• Why does group_left often produce more series than the left side had?
• Which operator emits 0/1 with the sample value and which one drops the false series?

Quiz