Error Rates From Logs · Observability

The on-call engineer opens Grafana at 03:00. The HTTP 5xx panel for the checkout service is at 1.2%; the alert that should have fired at 1% did not. The reason, after twenty minutes of digging, is that the alert is on loki_ingester_* series, not on the service’s actual error count. The service exposes no native metric for failed requests; only its logs. The five-x count that would have woken someone up is sitting in three gigabytes of log lines that no alert is reading.

This is the moment log-derived error rates earn their place. A query that counts error-level log lines per instance per second is the same signal as a 5xx counter; the cost is paid in ruler CPU rather than in scrape cycles, but the alert fires correctly on the right service, at the right threshold, attributed to the right host.

What it is

An error rate from logs is a time series whose samples are the rate of log lines classified as errors, typically derived from a structured level field emitted by the application. The shape is the canonical RED-method error rate: a count of error events per second, attributed to the producing service or instance, suitable for both dashboarding and alerting.

The Loki 3.x query form combines a stream selector, a line filter (or a structured parser), and a metrics aggregation:

sum(rate({job="checkout"}
         | json
         | level="error" [5m])) by (instance)

The query reads as: “give me the rate, per second over five minutes, of log lines from job=checkout whose parsed JSON has level=error, grouped by instance.” The result is one series per instance with values in errors per second.

Why a sysadmin cares

Native metrics are still the right answer when they exist. Log error rates are the right answer in three specific situations:

Vendor-owned binaries without instrumentation. Mainframe adapters, third-party SaaS bridges, anything that emits structured logs to a file and has no exporter contract.
New services before native metrics land. The first production deploy has logs but no Prometheus counters.
Cross-language frameworks that emit the same log format. A polyglot stack where some services expose metrics and others do not, but every service emits JSON with a level field.

Outside those cases the cost is paid for no benefit. A native counter is one scrape; a log error rate is a query per evaluation across every matching chunk.

How it works

The pipeline has three pieces: the stream selector narrows to the producing service, the parser lifts the level field out of the line, and the metrics aggregation turns the line stream into a per-instance rate.

  log stream                       Loki query engine                result
  ----------                       -----------------                ------
 {"ts":..., "level":"error",     parser: json                     sum(rate(...))
  "msg":"..."}  -->  line filter  -->  level=error  -->  reduce  --> vector
 {"ts":..., "level":"info",       drop
  "msg":"..."}                    keep
 {"ts":..., "level":"error",     |     bucket-by-instance
  "msg":"..."}

The line filter | json | level="error" does three things in sequence:

Parse. | json parses each line as JSON; the parse failure rate is itself a signal — too many parse failures means the log format has drifted.
Filter. level="error" keeps only lines whose parsed level field equals exactly "error". The match is case-sensitive; "ERROR" is different.
Reduce. The aggregation downstream (sum(rate(...)) by (instance)) computes the per-instance rate.

Two pieces of discipline to keep in mind:

The level filter is on the parsed field, not on the line. level="error" only works after | json (or | logfmt) has run. A line filter |= "error" is text-level; it cannot distinguish "level":"error" from "error_count": 0 without a parser.
rate requires a window. [5m] is the rate window; shorter windows are noisier, longer windows are slower to react. The trade-off is real and worth calibrating against traffic.

Under the hood: how rate() handles log streams

Loki’s rate() over a log stream is the same primitive as Prometheus’s rate() over a counter, but applied to a stream of discrete events rather than to a counter sample. Internally the query engine:

Selects the matching streams (those whose labels match the selector and whose chunks intersect the query window).
Streams each chunk through the parser and the line filter; the filter emits one “tick” per surviving line.
Buckets the ticks into the configured step interval.
Computes the per-bucket rate as ticks_in_bucket / bucket_duration_seconds.
sum by (instance) aggregates the per-stream series across streams sharing the same instance label.

The cost is paid per chunk fetched and per line parsed; a rule that runs every minute against a service emitting 100 lines/s processes 6,000 lines per evaluation, which is trivial. A rule that runs every minute against a service emitting 100,000 lines/s processes 6 million lines per evaluation, which is not. The cost scales with throughput, not with traffic patterns; a noisy service at 02:00 costs the same as the same service at 14:00.

How to configure it

The Loki ruler recording rule that produces the metric:

# /etc/loki/rules/prod-eu/errors.yaml
groups:
  - name: app_error_rate
    interval: 1m
    rules:
      # Per-instance error rate, per second, averaged over 5m.
      - record: app:log_errors:rate5m
        expr: |
          sum(rate({job="checkout", cluster="prod-eu"}
                   | json
                   | level=~"error|fatal" [5m])) by (instance)

      # Per-instance warning rate (for comparison / correlation).
      - record: app:log_warnings:rate5m
        expr: |
          sum(rate({job="checkout", cluster="prod-eu"}
                   | json
                   | level="warn" [5m])) by (instance)

      # Total log throughput per instance (denominator for SLO math).
      - record: app:log_lines:rate5m
        expr: |
          sum(rate({job="checkout", cluster="prod-eu"} [5m])) by (instance)

A few notes on the shape:

level=~"error|fatal" matches both uppercase and lowercase severity. If the application only ever emits lowercase, use level=~"error|fatal" exactly; if the format is mixed, document which is in scope.
by (instance) is the per-host attribution. Without instance, the result is one series for the whole service and per-host paging becomes guesswork.
The denominator series (app:log_lines:rate5m) lets a dashboard compute errors / lines as a percentage, which is more useful than absolute rate when traffic varies.
interval: 1m matches the rate window. An interval shorter than the window produces overlapping evaluations; an interval longer produces stale series.

The same expression, run ad-hoc against the query API:

# READ-ONLY: validate the query returns a metric, not a stream.
curl -s -u "$LOKI_USER:$LOKI_PASS" \
  --data-urlencode 'query=sum(rate({job="checkout", cluster="prod-eu"} | json | level=~"error|fatal" [5m])) by (instance)' \
  --data-urlencode 'start=2026-08-13T14:00:00Z' \
  --data-urlencode 'end=2026-08-13T14:05:00Z' \
  --data-urlencode 'step=60s' \
  -G http://loki.internal:3100/loki/api/v1/query_range \
  | jq '.data.resultType, (.data.result | length)'
# "matrix"
# 6

resultType: matrix and a non-zero series count confirm the query is a metric query and the data is present.

Pick the level filter to match the format

Three filter shapes appear repeatedly:

Filter	What it matches	Risk
`	= “error”`	Any line containing the substring “error”
`	~ ”(?i)level\s=\s“?error”?”`	Regex on raw line
`	json	level=~“error\|fatal”`

The third shape is the right answer when the application emits structured logs. The second is the fallback when the logs are free-form text. The first is rarely correct and is a smell.

How to validate it

The validation chain has three signals: the Loki ruler loaded the rule, the rule produces a metric, and the metric matches a known shape.

# READ-ONLY: the rule is loaded and evaluated recently.
curl -s -u "$LOKI_USER:$LOKI_PASS" \
  http://loki.internal:3100/loki/api/v1/rules \
  | jq '.data.groups[] | select(.name=="app_error_rate") | .rules[] | {name, lastEvaluation}'

# READ-ONLY: the produced metric has the expected cardinality.
curl -s -u "$MIMIR_USER:$MIMIR_PASS" \
  --data-urlencode 'query=count(app_log_errors_rate5m)' \
  -G http://mimir.internal:9009/prometheus/api/v1/query \
  | jq '.data.result[0].value[1]'
# "6"

# READ-ONLY: the metric is sane (errors/s should be small for healthy traffic).
curl -s -u "$MIMIR_USER:$MIMIR_PASS" \
  --data-urlencode 'query=app_log_errors_rate5m' \
  -G http://mimir.internal:9009/prometheus/api/v1/query \
  | jq '.data.result[] | {instance: .metric.instance, eps: .value[1]}'
# {"instance":"checkout-7f9c","eps":"0.12"}
# {"instance":"checkout-8k3d","eps":"0.08"}

A working rule returns a recent lastEvaluation, a series count that matches the number of running instances, and per-instance values that match operational expectations (typically a few per second or less for healthy services).

How it can fail

The level key is not stable. A library upgrade renames level to severity; the parser stops matching; the metric drops to zero. Symptom: error rate panel goes flat at zero with no alert; the on-call notices only when a customer does.
|= "error" matches non-error lines. A health-check response logged at INFO contains the substring "error_count = 0"; the line filter counts it. Symptom: the panel reads 5 errors/s during a quiet period, falsifies the alert threshold, and pages on background noise.
Parser failures are not errors but no-ops. A log line that fails | json is silently dropped; a service that emits a mix of structured and unstructured logs loses half its error count. Symptom: error count is suspiciously low compared to native metrics for the same service.
Case mismatch. The application emits "level":"ERROR" (uppercase); the query matches level="error" (lowercase). Symptom: the metric is zero during an active incident. Fix: use level=~"error|ERROR|fatal|FATAL" or normalise at ingestion.
Per-instance attribution breaks after a rollout. The instance label is a pod_ip:port pair; after a redeploy the ports change. Symptom: the new instance has no error series for the first few minutes; the alert fires on a “missing” instance. Use a stable label (pod_name, hostname).
Multiple error counters in the same line. A line that carries level="error" and level="info" after the parser keeps the parsed field but a pre-existing parser misconfig can produce two fields. Symptom: the rate is roughly 2x what it should be; investigation finds the log format carries both levels (rare; almost always a parser bug).

How to troubleshoot it

The diagnostic order: does the rule evaluate, does the parser match, is the level filter correct, is the cardinality sane.

Rule evaluates? GET /loki/api/v1/rules. A missing lastEvaluation or a stale one (older than interval + 1m) means the ruler is stuck; check the ruler logs.
Parser matches? Run the same expr against /loki/api/v1/query_range and inspect one of the streams that the parser should be producing. If the result is empty, the parser is failing; loosen the filter to confirm the underlying stream is present.
Level filter correct? Run {job="checkout"} | json | __error__="" to confirm the parser sees clean JSON; then ... | level=~".+" to see what distinct level values the application emits.
Cardinality sane? count by (__name__) (app_log_errors_\ rate5m) in the target. The number should match the number of running instances; a sudden jump means an unbounded label has crept in.
Cross-check with native metrics. If the service exposes a native 5xx counter, compare the two over a known-good window. A consistent ratio (errors-on-logs ≈ 5xx-count) confirms the metric; a discrepancy points to the level filter or the parser.
Inspect the ruler logs. /var/log/loki/ruler.log records parser failures, evaluation errors, and remote-write rejections.

Security implications

The level field can be attacker-influenced. A request that smuggles "level":"error" into a log message (e.g. through a templated log statement) can drive the metric artificially. The fix is to treat the level field as a server-emitted value, not as user input; libraries that emit level themselves are safer than libraries that trust request context.
The metric replicates PII. An error-level line that carries a user_email or request_body is replicated into Prometheus / Mimir, into remote-write backups, into Grafana query caches. Audit the log format; redact sensitive fields at the agent before they reach Loki.
The alert path inherits the query budget. An alert that fires on the derived metric executes the LogQL query on every evaluation; an attacker who can drive log volume can drive ruler CPU. Rate-limit at ingestion.

Performance implications

Parser cost dominates. A query against JSON logs parses every line in the matching streams. A service emitting 1 MB/s of JSON is 1 MB of parses per evaluation; a service emitting 100 MB/s is 100x worse. The cost is paid whether the line is an error or not; the parser runs before the filter.
Stream count dominates. A rule whose selector opens 1,000 streams (one per pod) is more expensive than a rule whose selector opens 10 streams (one per cluster). Coarsen the selector when the cardinality is not needed.
Index lookup cost is bounded by label matcher selectivity. The {job="checkout"} matcher uses the index; the level=~"error|fatal" matcher does not (level is a parsed field, not a Loki stream label). The first narrows; the second filters after the chunks are open.
Ruler CPU scales with rule count and query cost. A services-team that adopts log-derived metrics for twenty services is paying for twenty rules; budget for it.

Production guidance

Write the level filter against the parsed field, not the raw line. The parsed filter is exact; the raw filter is a guessing game.
Pin a stable instance label. Pod IPs change on every redeploy; service names do not. Use pod_name or hostname, not instance derived from the pod IP.
Include a parser-failure signal. A separate metric for parse failures (sum(rate(\{job="checkout"\} | json | __error__!="" [5m])) by (instance)) catches log-format drift before the error rate drops to zero.
Calibrate the rate window against traffic. A 5-minute window averages over five minutes of traffic; for a service at 1 req/s, that is 300 events, which is too few for a meaningful rate. Use a longer window for low-traffic services.
Document the decommission path. Each log-derived error rate should have an owning team, a target native metric, and a written expiry. The metric is a temporary expedient; the expiration is part of the design.

Verification

You should now be able to answer:

Why is |= "error" usually the wrong filter for an error-rate query, and what should replace it?
What is the difference between rate of error lines and rate of failed requests, and when is each the right signal?
How does by (instance) enable per-host attribution, and what label should the instance selector use for stability across rollouts?
Which Loki ruler signal confirms a rule has evaluated successfully in the last interval?
What is the failure shape when the application renames the level field, and how do you detect it early?