LogQL Aggregations

The on-call engineer wants to know three things: “what is the error rate per service”, “what is the p99 latency per endpoint”, and “how many requests did we serve last hour”. The log stream has every line needed to answer each question. The aggregation is what converts the stream into the answer.

What a LogQL aggregation is

A LogQL aggregation is the suffix of a log query that converts the filtered stream into a numeric series. Two shapes:

• Range-vector aggregation. The query ends with one of the range functions — count_over_time, sum_over_time, avg_over_time, max_over_time, min_over_time, quantile_over_time, bytes_rate, bytes_over_time, rate, sum (over unwrap). The query becomes a metric query; the result is a time series.
• Instant aggregation. The query ends with a plain aggregation operator — sum, count, avg, max, min, topk, bottomk, stddev, stdvar. The query becomes a single-point query; the result is one number per grouping.

The aggregations are the same operators that PromQL exposes. LogQL adopted the Prometheus vector semantics because the on-call engineer already knows them.

# Range vector: error rate over time, grouped by service.
count_over_time({service=~"checkout|inventory"} | json | level="error" [5m])

# Unwrap: total latency summed across requests, per endpoint.
sum_over_time(
  {service="checkout"} | json | unwrap latency_ms [5m]
) by (endpoint)

# Quantile: p99 latency per endpoint over the last five minutes.
quantile_over_time(0.99,
  {service="checkout"} | json | unwrap latency_ms [5m]
) by (endpoint)

# Instant: count of distinct request_ids in the last five minutes.
count(
  count_over_time(
    {service="checkout"} | json | unwrap request_id [5m]
  ) by (request_id)
) by (service)

Why a sysadmin cares

The aggregation is what turns Loki from a log viewer into a metrics source. Without it, the operator is staring at lines. With it, the operator is looking at a chart that the alert manager can act on.

• Alerting. A Loki recording rule that aggregates count_over_time(...) produces a metric series; the alert manager evaluates the metric. The on-call engineer pages on a number, not on a wall of green.
• Capacity planning. A sum_over_time(... | unwrap bytes) on the access log produces the egress-per-tenant series. The finance team reads it; the platform team budgets against it.
• SLOs. A quantile_over_time(0.99, ...) produces the p99 latency. The SLO is “p99 below 500 ms for 99% of the month”; the recording rule is what produces the input.
• Cross-signal correlation. A Loki aggregation that uses the same label set as a Prometheus metric lets the on-call engineer pivot from “the error count spiked” to “the /checkout endpoint failed 412 times” without translating.

How it works — the mental model

{service="checkout"}        <- selector (index lookup)
| json                       <- parser (extracts fields)
| level="error"              <- filter on parsed field
| unwrap latency_ms          <- convert field to numeric value
[5m]                         <- range vector: 5-minute window
sum by (endpoint)            <- reduce within the window

The mental model has four moves:

1. Selector. Choose the streams. This is the index lookup.
2. Filter and parse. Choose the lines and extract the fields. This is the chunk scan.
3. Range vector. Choose the time window per data point. [5m] means “the entries that arrived in the last five minutes”.
4. Aggregate. Reduce the entries in the window to a single number per grouping. The grouping is set by by (...) or without (...).

The result is a metric series that can be plotted, alerted on, or recorded.

How to configure it

The five aggregations every on-call engineer uses.

# count_over_time: how many matching lines in the window.
count_over_time({service="checkout"} | json | level="error" [5m])

# sum_over_time on an unwrapped field: total of a numeric field.
sum_over_time({service="checkout"} | json | unwrap bytes [5m])

# avg_over_time: average of a numeric field.
avg_over_time({service="checkout"} | json | unwrap latency_ms [5m])

# max_over_time: peak of a numeric field in the window.
max_over_time({service="checkout"} | json | unwrap latency_ms [5m])

# quantile_over_time: percentile of a numeric field.
quantile_over_time(0.99,
  {service="checkout"} | json | unwrap latency_ms [5m]
) by (endpoint)

# rate: events per second.
rate({service="checkout"} | json | level="error" [5m])

The grouping clauses:

# by: keep only these labels in the result.
sum_over_time({service="checkout"} | json | unwrap latency_ms [5m]) by (endpoint)

# without: drop these labels from the result.
sum_over_time({service="checkout"} | json | unwrap latency_ms [5m]) without (instance)

# topk: keep only the k largest series.
topk(3, sum_over_time({service=~".+"} | json | unwrap latency_ms [5m]) by (endpoint))

A common production pattern: aggregate at ingest via a recording rule, then re-aggregate cheaply in dashboards.

# recording rule example
groups:
  - name: checkout-error-rate
    rules:
      - record: service:checkout_errors:rate5m
        expr: |
          sum by (service, env) (
            count_over_time({service="checkout"} | json | level="error" [5m])
          )

How to validate it

# 1. The aggregation runs against the expected window.
logcli --addr=http://loki:3100 instant --since=5m \
  'sum(count_over_time({service="checkout"} | json | level="error" [5m])) by (service)'
# {service="checkout"}  412

# 2. The unwrap path produces a numeric series.
logcli --addr=http://loki:3100 instant --since=5m --output=stats \
  'sum_over_time({service="checkout"} | json | unwrap latency_ms [5m]) by (endpoint)'
# {endpoint="/checkout"} 1240
# {endpoint="/cart"} 312

# 3. The quantile path returns the requested percentile.
logcli --addr=http://loki:3100 instant --since=5m \
  'quantile_over_time(0.99,
    {service="checkout"} | json | unwrap latency_ms [5m]
  ) by (endpoint)'
# {endpoint="/checkout"} 1240
# {endpoint="/cart"} 318

# 4. The rate path produces events per second.
logcli --addr=http://loki:3100 instant --since=5m \
  'rate({service="checkout"} | json | level="error" [5m])'
# {service="checkout"} 0.0136

# 5. The grouping is what you expected.
logcli --addr=http://loki:3100 instant --since=5m --output=stats \
  'sum(count_over_time({service=~".+"} | json | level="error" [5m])) by (service)'
# {service="checkout"} 412
# {service="inventory"} 17
# {service="pricing"} 8

Two practical checks:

• Compare the aggregation against a Prometheus metric that should agree. If the application’s Prometheus counter reports 412 errors and the Loki aggregation reports 412 errors, both are right.
• Compare the window against a longer window. A count_over_time [5m] that says “10 errors” should look like a fifth of count_over_time [25m] over a steady-state period.

How it can fail

Six recurring failure modes. Each maps to an observable symptom.

1. No by clause on a high-cardinality aggregation. sum by () (...) aggregates everything into one series. The result is correct but uninformative. Symptom: the panel shows one number for the entire fleet.
2. Aggregating the wrong field. sum_over_time(... | unwrap latency) when the field is latency_ms. The numeric value is wrong by a factor of 1000. Symptom: the chart drops off a cliff or spikes into the stratosphere.
3. unwrap on a non-numeric field. A field that is "1240ms" or "1.24s" cannot be unwrapped. Symptom: every contribution is zero or the query returns an error.
4. A 1-minute window on a noisy series. A count_over_time [1m] on a service with bursty traffic produces a chart that looks like a heart monitor. The 5-minute window is the right default. Symptom: the alert fires for noise, the on-call team fatigues.
5. rate confused with count_over_time. rate(... [5m]) is events per second; count_over_time(... [5m]) is events in the window. The two are not interchangeable; mixing them produces charts whose units are unclear.
6. Aggregating before parsing. count_over_time(\{...\} [5m]) by (level) tries to group by a field the parser has not seen. Symptom: Loki returns “level is not a label” or zero series.

How to troubleshoot it

The diagnostic order for “the aggregation returns the wrong number” or “the panel is flat”:

1. Confirm the parsed field exists. logcli query --limit=1 '\{...\} | json' | jq . and check the field is present.
2. Inspect the unwrap path. logcli instant '... | unwrap latency_ms' without an aggregation. The result should be numeric.
3. Compare units. ms vs seconds, bytes vs MB. A unit mismatch is the single commonest cause of “the chart is off by 1000”.
4. Compare the window. Try [1m], [5m], [15m], [1h]. The right window is the one whose data point is stable.
5. Confirm the grouping. sum by (x) (...) and sum without (x) (...) are different aggregations. Make sure the grouping matches the question.
6. Time-box the search. A 24-hour aggregation is more useful than a 5-minute one when the field has just been introduced.

Security implications

• Cardinality explosion. A by clause on an unbounded label (request_id, user_id, trace_id) is a denial-of-service vector against the recording-rule evaluator or the querier. Limit the labels in by to the bounded set.
• Sensitive numeric fields. An unwrap on a field that contains a credit-card-shaped pattern returns the value in the series. The aggregation is doing its job; the field is the problem. Scrub at the pipeline.
• Cross-tenant aggregation. The aggregation runs in the querier’s tenant context. A multi-tenant Loki does not let one tenant aggregate another tenant’s streams. RBAC is the authoritative control.

Performance implications

• Cost. An aggregation runs against every line the preceding filter accepted. A count_over_time [5m] over the whole tenant reads every chunk the selector opened. A sum_over_time(... | unwrap x [5m]) reads the same chunks and converts every field to a number.
• Window length. A 5-minute window is the production default. A 1-minute window is noisier; a 1-hour window is more expensive. Pick the shortest window that produces a stable data point.
• Step alignment. A quantile_over_time(0.99, [5m]) at step=1m produces five quantiles per 5-minute period. A step=30s produces ten. The step is the unit of work for the querier.
• Recording rules. A Loki recording rule evaluates the aggregation once per step and stores the result as a metric. A dashboard that reads the recording rule is cheap. A dashboard that re-runs the aggregation on every refresh is expensive. The recording rule is the right place for any aggregation the team always wants to read.

Production guidance

• Default to count_over_time [5m] for event counts and sum_over_time(... | unwrap x [5m]) for numeric totals. The 5-minute window is the standard cadence.
• Use quantile_over_time for percentiles. avg_over_time is not the right tool for a question about the tail.
• Put the aggregation in a recording rule when the dashboard or alert reads it on every refresh. The rule evaluates once per step; the panel reads the result.
• Limit the by clause to bounded, operationally meaningful labels. The default is service and env; add endpoint, method, or status when the question demands it.

Verification

You should now be able to answer:

• What is the difference between count_over_time and rate, and when is each the right tool?
• Why must a parsed field come before unwrap and sum by in the chain?
• What is the production default window length, and why?
• When does an aggregation belong in a recording rule rather than a dashboard query?

Quiz