by() and without()

A Grafana dashboard says “per-job request rate.” The query is sum(rate(http_requests_total[5m])). The dashboard renders one number. The on-call engineer clicks through to a per-host view, which uses sum by (instance) (rate(http_requests_total[5m])). Now there are twelve rows but the job label is gone. The operator hovers over a row and asks: which service is web05 running? The dashboard does not know — the job label was dropped by the by clause, which keeps only what it names.

The by and without clauses are how the operator decides which labels survive an aggregation. They look interchangeable. They are not. The difference is which labels survive, and the mistake is almost always a wrong assumption about what survived.

What by() and without() are

Both are aggregation modifiers — clauses attached to an aggregation operator that name the labels the reducer should respect. The clause is a parenthesised list of label names.

sum by (job) (rate(http_requests_total[5m]))
sum without (instance) (rate(http_requests_total[5m]))
avg by (job, instance) (node_load1)
max without (route) (rate(http_requests_total[5m]))

by (L1, L2, ...) partitions the input vector by the listed labels and emits output series whose only labels are the listed ones. Every other label is dropped. If you list two labels, the output has at most N1 × N2 distinct series (one per unique combination).

without (L1, L2, ...) partitions the input vector by every label except the listed ones. The output series retain every label that was not in the list. If the input has labels \{job, instance, route, status\} and you say without (instance, route), the output keeps job and status.

The two clauses are equivalent under complement: sum by (a, b) (x) produces the same output as sum without (every-other-label) (x). The difference is which labels you have to enumerate. In a query whose input has many labels, by is usually shorter and more readable; without is usually safer because it does not break when a new label is added to the input.

Why a sysadmin cares

Two production patterns drive the by / without choice:

• Per-cluster vs per-host view. “What is the fleet doing?” is sum by (job) (...). “What is each host doing?” is sum by (job, instance) (...). The clauses change which labels the panel renders.
• Rolling up at recording-rule time. A recording rule that pre-aggregates per-host data into a per-job view uses sum without (instance) (...) to keep every other label except the one being collapsed. A rule that adds an aggregated_by dimension uses by (job, aggregated_by) (...).

The clauses are also where silent bugs live. A by clause that forgets a label drops that dimension. A without clause that omits a label keeps it. Both directions have cost.

How it works

input vector
   |
   v
group by the kept-label set (the complement of the without set)
   |
   v
run the reducer per group
   |
   v
emit one series per group, label set = kept labels

The grouping set is what matters. Each unique combination of kept labels defines one output series. The reducer runs over every input series that shares that combination.

sum by (job) (rate(http_requests_total[5m]))

input:
  {job="api", instance="web01", route="/checkout", status="200"}   120.4
  {job="api", instance="web02", route="/checkout", status="200"}   118.2
  {job="api", instance="web01", route="/cart",    status="200"}    82.1
  {job="api", instance="web02", route="/cart",    status="200"}    79.5
  {job="worker", instance="wq01", status="200"}                   44.7

groups:
  {job="api"}:    120.4 + 118.2 + 82.1 + 79.5  = 400.2
  {job="worker"}:  44.7

output:
  {job="api"}     400.2
  {job="worker"}   44.7

Notice that instance, route, and status were dropped from the output. They participated in the grouping only insofar as they defined which series to sum. The output has only the labels in the by clause.

The same input with sum without (instance, route) produces:

output:
  {job="api", status="200"}     400.2
  {job="worker", status="200"}   44.7

job and status survived because they were not in the without list. instance and route were dropped. The output has two distinct (job, status) combinations.

How to configure it

The clauses are not configured; they are written. The recording- rule pattern shows the production convention.

# /etc/prometheus/rules/grouping.yml
groups:
  - name: aggregation-hierarchy
    interval: 30s
    rules:
      # Per-instance rate. Keep both job and instance.
      - record: instance:http_requests:rate5m
        expr: sum by (job, instance) (rate(http_requests_total[5m]))

      # Per-job rate. Drop instance, keep everything else (job, status, route).
      - record: job:http_requests:rate5m
        expr: |
          sum without (instance) (
            instance:http_requests:rate5m
          )

      # Fleet total. Drop instance, drop job, collapse to one series.
      - record: cluster:http_requests:rate5m
        expr: |
          sum without (instance, job) (
            instance:http_requests:rate5m
          )

The second rule uses without (instance) because the input recording rule carries both job and instance, and the operator wants the job label to survive. The third rule drops both instance and job to collapse the fleet into a single series.

The complementary style — by (job) instead of without (instance) — is equally correct and produces the same output. The choice is about readability: by (job) says “I want the job label” in three characters; without (instance) says “I want everything except instance,” which is a longer enumeration when the input has many labels.

How to validate it

# 1. Static check.
promtool check rules /etc/prometheus/rules/grouping.yml
# SUCCESS: /etc/prometheus/rules/grouping.yml

# 2. Confirm the label set of the output series matches the by clause.
curl -s 'http://prometheus:9090/api/v1/query?query=job:http_requests:rate5m' \
  | jq '.data.result[].metric | keys'
# [ "job" ]

# 3. Confirm the per-instance rule has both job and instance.
curl -s 'http://prometheus:9090/api/v1/query?query=instance:http_requests:rate5m' \
  | jq '.data.result[].metric | keys'
# [ "instance", "job" ]

# 4. Confirm the fleet total collapsed to one series.
curl -s 'http://prometheus:9090/api/v1/query?query=cluster:http_requests:rate5m' \
  | jq '.data.result | length'
# 1

The second step is the bug check. If the output has more labels than the by clause, the inner expression is carrying labels the clause did not drop. Investigate the input series; the clause is correct but the input has changed.

How it can fail

The high-frequency failure modes for by and without:

1. by (instance) forgets job. A per-instance recording rule that drops job makes it impossible to tell which service the instance belongs to. The dashboard shows “web05 is at 80% CPU” without saying whether web05 is an API host, a worker, or a database node. Always include the service-identifying labels in the by clause.
2. without (instance) on a metric that has more labels than expected. A recording rule intended to collapse per-instance data that uses without (instance) keeps every other label (route, status, region, pod_template_hash, …). The “per-job” rule becomes “per-(job, route, status, region, …)” and the series count explodes. Use by (job) instead, or list every label you want to drop in the without clause.
3. Clause silently added to the wrong aggregator. A query topk(10, rate(http_requests_total[5m]) > 100) has no aggregation to attach by to — the rate function does not accept grouping clauses. The clause must follow an aggregation operator (sum by, avg by, max without, …), not a range-vector function.
4. Clause order matters in some operators. sum by (job) (rate(...)) and sum(rate(...)) by (job) are both valid PromQL but mean the same thing. The convention is to attach the clause to the aggregator, not the inner expression. A query with the clause inside the inner expression has been miscoded.
5. New labels on the source metric. A recording rule written with by (job) is correct today. Tomorrow the source metric gains a region label. The by (job) clause still drops region. The rule is still correct, but the operator who wants “per-region” view now has to update every recording rule. The safer pattern in a fast-moving data model is without (instance) so the new label survives by default.
6. Empty grouping set. sum by () (metric) is legal and produces one series with no labels at all. It is also the same as sum(metric). The two forms are equivalent; the former is occasionally useful when a templating system needs a clause placeholder.

How to troubleshoot it

When a panel “drill-down” fails or a recording rule emits the wrong label set:

1. Inspect the label set of the input. Run the inner expression without the aggregation and inspect the labels each series carries. The by / without clause is a function of this set.
2. Inspect the label set of the output. Run the full aggregation and inspect the output series’s label set. It must match the by clause exactly (for by) or the complement of the without clause exactly (for without).
3. Inspect the cardinality of the output. A recording rule that emits 1,000 series when the team expected 12 is using the wrong clause — most likely a without that did not drop enough labels, or a by that did not keep enough.
4. For drill-down problems, ensure the rule kept the labels you need. The recording rule is the boundary. If a label is not on the rule’s output series, no downstream query can recover it.

Security implications

• Recording rules that use by (job, instance, ...) keep every label that may include customer identifiers, request IDs, or session IDs. The rule persists those labels in the TSDB. Audit the rules on the same schedule as the source metrics.
• without clauses can hide labels the operator did not intend to drop. A without (instance) rule on a metric with a customer_id label keeps customer_id. If the operator thought the rule collapsed “everything,” they may have been wrong. Verify with curl /api/v1/query and inspect the label set.
• A by clause that omits a label is irreversible — once the recording rule drops the label, no downstream query can recover it. Do not omit labels you may need for an investigation.

Performance implications

• The cost of an aggregation is dominated by the cardinality of the kept-label set, not the cardinality of the input. A sum by (job) (rate(...)) over 100,000 input series that all share the same 12 jobs is a 12-series output. The grouping is cheap.
• A without clause on a metric with many labels keeps more output series than a by clause on the same input. Choose the shorter clause when the input has many labels and the aggregation is rolled up further by a higher-level rule.
• Recording rules with by clauses that produce high-cardinality output (per-pod, per-route, per-status) cost more in the TSDB than rules with coarser output. The performance part of the course returns to cardinality budgets.

Verification

You should now be able to answer:

• What does sum by (job) (x) keep in the output?
• What does sum without (instance) (x) keep in the output?
• Why is by (instance) often the wrong clause for a per-host recording rule?
• When should you prefer without to by?

Quiz