Log-Derived Metrics

A legacy line-of-business application does not expose a Prometheus endpoint. It logs structured JSON to stdout, twice per second per request, and the only operational question anyone asks of it is “how many errors in the last five minutes, by host?”. No native metric exists. Adding one would require shipping a new build and negotiating with a vendor. The answer that is already in the platform: derive the metric from the log stream.

The Loki ruler, when configured with a metrics-generation query, runs the LogQL expression on a schedule, reduces the log streams into time series, and writes them as Prometheus samples to the configured target. The same Grafana panel that consumes a native counter consumes the derived one. No rebuild, no vendor negotiation, no instrumentation gap.

The trade-off is real and worth pricing before adoption. Each derived metric is paid for in ruler CPU, in stored samples, in remote-write bytes, and in the operational cost of every extra label you permit through.

What it is

A log-derived metric is a time series whose samples are produced by evaluating a LogQL expression on the Loki log stream rather than by scraping an instrumented target. The Loki ruler evaluates a LogQL query whose result is a metric (“metrics query”), and writes the resulting series through a remote-write-compatible sink. The downstream store can be Prometheus, Mimir, or any other receiver that speaks the remote-write protocol.

   log stream                 Loki ruler               remote-write target
  -----------                ------------             ---------------------
 {job="x"} | json  -- query --> MetricsQuery  -- series --> Prometheus / Mimir
 {job="y"} |= "err" -- query --> MetricsQuery  -- series --> Prometheus / Mimir
 ...

The Loki ruler is the same component that evaluates recording and alerting rules for log streams; the metrics-generation pattern is just a rule whose expr returns a metric rather than a log stream. The “metrics” query form replaces the usual log extraction with a metric reduction, and the output is series in the Prometheus exposition format.

Why a sysadmin cares

Native metrics are the right answer whenever they exist. Log metrics exist for two production situations:

• Legacy or vendor-owned binaries. No way to add an exporter, no way to convince the vendor. The signal is in stdout; the only honest way to get it into Prometheus is to read stdout.
• Rapid prototyping. A new service has logs before it has metrics. A log-derived metric is the path from “we have logs” to “we have a dashboard” without waiting for the instrumentation rollout to land.

Outside those two cases, log-derived metrics are the wrong tool. A native counter is one scrape and one sample per interval. A log-derived counter is one LogQL evaluation per interval, a chunk-scan per matching stream, and a label-set recomputation on every write. The cost ratio is roughly 100x to 1000x per series per minute, before considering storage.

How it works

The Loki ruler runs on a fixed evaluation interval (60 seconds by default). For each rule, the ruler submits the LogQL expression to the Loki query engine with the current evaluation window. The engine streams matching chunks, applies the line filters and parsers, executes the unwrap or aggregation, and returns a Prometheus-shaped result. The ruler then writes the result as remote-write samples to the configured backend.

+---------------+        +---------------------+        +-------------------+
| rule_files    |        | ruler               |        | remote_write       |
|   *.yaml      | --->   |   evaluate @ t      | --->   |   Prometheus      |
|   groups:     |        |     submit LogQL    |        |   Mimir           |
|     rules:    |        |     receive series  |        |   cortex          |
|       expr:   |        |     enforce limits  |        |                   |
+---------------+        +---------------------+        +-------------------+
                                  |
                                  | (also)
                                  v
                          /loki/api/v1/rules
                          Grafana / amtool

Two mental-model points worth holding:

• The ruler is a query client, not a different storage engine. It uses the same query path that Grafana’s Explore uses; rule evaluation is just an automated call on a fixed cadence. When query latency climbs, rule evaluation climbs with it.
• The output is Prometheus series, not Loki series. The metric lives in whatever target receives the remote-write. A rule whose remote-write target is unreachable produces no samples, but the rule still evaluates and still spends Loki query budget.

The query form is what distinguishes a “metrics query” from an ordinary log query:

# Metric query: produces a time series.
# The "metrics" form applies an aggregation that yields a value
# per series per step (sum, count, rate, quantile, unwrap+...).
sum(rate({job="app"} |= "level=error" [5m])) by (instance)

# Ordinary log query: produces log lines.
{job="app"} |= "level=error"

In Loki 3.x the metrics form is implicit in the rule type. A rule with a record action whose expr returns a metric reduces to a series; a rule with an alert action whose expr returns a metric evaluates to a vector that the alert compares against a threshold.

How to configure it

The rule file format is Prometheus-compatible. The expr contains the LogQL query, and the ruler infers whether the result is a metric or a log query from the query shape.

# /etc/loki/rules/fake/rules.yaml
groups:
  - name: app_error_rate
    interval: 1m
    rules:
      # Count of ERROR-level log lines per instance, per minute.
      - record: app:log_errors:rate5m
        expr: |
          sum(rate({job="app", cluster="prod-eu"}
                   | json
                   | level=~"error|fatal" [5m])) by (instance)
        labels:
          severity: error
          source: log

      # Total log volume per service per minute.
      - record: app:log_lines:rate5m
        expr: |
          sum(rate({cluster="prod-eu", job=~"app|api|worker"}
                   [5m])) by (job, level)

A few notes that matter in production:

• record: names follow Prometheus conventions. Use level:metric:operation (app:log_errors:rate5m); the second segment names the metric, the third names the operation. The ruler does not enforce the convention, but Grafana dashboards and downstream tooling will.
• labels: overrides merge into every series. A rule that stamps source: log onto every produced series makes it possible to grep source="log" in Grafana and see exactly which panels are derived versus native.
• interval: overrides the group evaluation interval. The default is 1m; lower it (30s) only when the metric drives a short-window alert and you have the ruler CPU budget.
• expr must terminate in a metric. A LogQL expression that ends in a stream (e.g. {job="x"}) is not a metrics query; the ruler evaluates it but produces nothing useful, and emits a warning.

The Loki config that loads the file:

# /etc/loki/config.yaml (microservices mode excerpt)
ruler:
  enabled: true
  storage:
    type: local
    local:
      directory: /etc/loki/rules
  rule_path: /etc/loki/rules/fake
  alertmanager_url: http://alertmanager.internal:9090
  remote_write:
      enabled: true
      client:
        url: http://mimir.internal:9009/api/v1/push
        tenant_id: prod-eu
        batch_send_deadline: 1m
        min_backoff: 100ms
        max_backoff: 10s
  evaluation_interval: 1m
  poll_interval: 1m
  query_timeout: 30s

The rule_path is scanned on every poll_interval. Adding a new file is a drop-in; the ruler picks it up at the next poll. Removing a file removes the rule; existing series go stale and stop receiving samples, but they are not deleted from the target until the target’s staleness handling kicks in.

Validate before applying (the ruler has no native dry-run; the canonical sanity check is the Loki query API):

# READ-ONLY: confirm the LogQL expression is valid and returns a metric.
curl -s -u "$LOKI_USER:$LOKI_PASS" \
  --data-urlencode 'query=sum(rate({job="app", cluster="prod-eu"} | json | level=~"error|fatal" [5m])) by (instance)' \
  --data-urlencode 'start=2026-08-13T14:00:00Z' \
  --data-urlencode 'end=2026-08-13T14:05:00Z' \
  --data-urlencode 'step=60s' \
  --data-urlencode 'limit=1000' \
  -G http://loki.internal:3100/loki/api/v1/query_range \
  | jq '.data.result | length'
# 4   (one series per instance in the cluster)

# READ-ONLY: syntax-check the rule file with promtool's ruler check.
# promtool 2.55 understands the Loki ruler format when --syntax-only is used.
promtool check rules /etc/loki/rules/fake/rules.yaml

How to validate it

A rule that loads but produces nothing is the most common failure. The diagnostic checklist is the same at every step.

# READ-ONLY: the rule is loaded by the ruler.
curl -s -u "$LOKI_USER:$LOKI_PASS" \
  http://loki.internal:3100/loki/api/v1/rules \
  | jq '.data.groups[].rules[] | {name, query, lastEvaluation}'

# {
#   "name": "app:log_errors:rate5m",
#   "query": "sum(rate({job=\"app\"} ...)) by (instance)",
#   "lastEvaluation": "2026-08-13T14:30:00.000Z"
# }

# READ-ONLY: confirm the ruler pushed samples to the target.
curl -s -u "$MIMIR_USER:$MIMIR_PASS" \
  --data-urlencode 'query=app_log_errors_rate5m{source="log"}' \
  -G http://mimir.internal:9009/prometheus/api/v1/query \
  | jq '.data.result | length'
# 4

# READ-ONLY: confirm the ruler is healthy on its own metrics.
curl -s http://loki.internal:3100/metrics \
  | grep -E '^loki_ruler_.*_total|.*_last_success.*' \
  | head
# loki_ruler_evaluation_seconds_bucket{...}
# loki_ruler_rules_last_eval_success{rule_name="app:log_errors:rate5m"} 1

A working rule appears in the rules API with a recent lastEvaluation, in the target with the expected series count, and in the ruler’s own metrics with last_eval_success = 1. A failure at any one of these three signals is the place to start investigating.

How it can fail

The failure modes cluster around four axes: query correctness, cardinality, target reachability, and ruler capacity.

1. The metric query is actually a log query. A rule whose expr is {job="app"} |= "level=error" produces a log stream, not a metric. The ruler evaluates it; the result is an empty vector; nothing is written. Symptom: the rule appears in the rules API and updates on schedule, but no series ever reaches the target.
2. High-cardinality by clause. A rule whose by includes user_id, trace_id, or any other unbounded label turns one rule into thousands of series per evaluation. Symptom: remote-write queue depth climbs (loki_ruler_remote_write_ queue_depth), the target rejects with 4xx, and the rule eventually back-pressures.
3. Remote-write target unreachable. A Mimir outage or a network partition leaves the ruler evaluating successfully but unable to flush. Symptom: loki_ruler_remote_write_ dropped_samples_total rises; samples older than batch_send_deadline are discarded. Alert evaluation based on the derived metric goes stale.
4. Ruler back-pressure from heavy rules. A rule whose LogQL scans gigabytes per evaluation forces subsequent rules in the same group to wait. Symptom: loki_ruler_evaluation_ missed_total rises; the rules API shows lastEvaluation lagging the configured interval.
5. Group interval too short. interval: 15s against a slow query accumulates overlapping evaluations. Symptom: ruler CPU climbs linearly with rule count and query cost; the ruler eventually OOMs.
6. Remote-write tenant ID mismatch. The ruler writes with tenant_id: prod-eu; the target receives with a different header. Symptom: 401 Unauthorized on every push; no series land; the ruler logs the error and retries.

How to troubleshoot it

The diagnostic order is: is the rule loaded, is the query valid, is the target reachable, is the cardinality bounded.

1. Rule loaded? GET /loki/api/v1/rules. If the rule is absent, check rule_path and the file’s YAML syntax with promtool check rules.
2. Query valid? Submit the same expr directly against /loki/api/v1/query_range with a small time window. A 500 from the query endpoint means a parse or type error in the LogQL expression; a 200 with resultType: streams means the rule is the wrong shape (a log query, not a metric query).
3. Target reachable? curl the remote_write.client.url from the ruler host. A 401 indicates a tenant ID / auth mismatch; a 5xx indicates the receiver is unhealthy.
4. Cardinality bounded? Query the produced series in the target with count by (__name__) ({source="log"}). A series count that grows by evaluation interval is unbounded; tighten the by clause.
5. Ruler capacity? loki_ruler_evaluation_missed_total is the canonical signal. A non-zero rate means the ruler cannot keep up; either raise the group interval or split the group.
6. Inspect ruler logs. /var/log/loki/ruler.log (or the container stdout) records each evaluation failure with the rule name and the upstream error.

Security implications

• The ruler speaks remote-write; the receiver enforces auth. Misconfigured tenant_id or a missing bearer token turns the ruler into an anonymous writer. The receiver should require auth and per-tenant rate limits; the ruler should be the only writer allowed.
• Log-derived metrics carry the same secrets as the logs. A rule whose expr extracts a user_email field and emits it as a label replicates the email into Prometheus / Mimir, into Grafana query caches, into remote-write backups. Audit the expr for fields that should not become labels; prefer [REDACTED] upstream.
• The rules directory is a write surface. A compromised host that can drop a YAML file under rule_path can inject new metrics, redirect remote-write to an attacker-controlled endpoint, or flood the platform with expensive rules. Filesystem-level controls (read-only mount, dedicated service account, change auditing) are appropriate.
• Alerts fired by derived metrics inherit the query budget. An attacker who can influence log volume can drive ruler CPU by triggering a heavy rule on every evaluation. The fix is the same as for any log pipeline: rate-limit at ingestion.

Performance implications

• Ruler CPU scales with rule cost. A rule whose LogQL scans 100 GB of chunks per evaluation is 100x more expensive than a rule that scans 1 GB. The shape that matters: the time window in the bracket, the number of streams the selector opens, and the cardinality of the by clause.
• Stored samples scale with by cardinality. A rule that produces N series per evaluation writes N samples per evaluation interval. Over 30 days, that is 30 * 24 * 60 / N samples per series, multiplied by N. The dominant cost is the series count, not the per-sample cost.
• Remote-write is the bandwidth ceiling. Each push is a snappy-compressed protobuf frame. A thousand-series rule at 60s interval is a few hundred KiB per minute; a hundred-thousand- series rule is tens of MiB per minute and will saturate the link.
• Ruler memory holds the per-group state. A misconfigured rule with a large window holds the working set in memory for the evaluation duration; long windows plus large cardinalities are OOM territory.

Production guidance

• Start with the rule file in source control. Treat the rule file the same way you treat prometheus.yml: code review, signed commits, audit log.
• Treat each log-derived metric as a temporary expedient. The decommission target is native instrumentation; the expiry is written into the rule’s runbook entry.
• Bound the by clause. The allowed labels are the same labels you would put on a native metric. Anything more is a cardinality bug.
• Size the remote-write target for the worst case, not the average. A rule that produces a few series at quiet hours can produce thousands at peak; the target must accept the burst.
• Alert on loki_ruler_evaluation_missed_total and on loki_ruler_remote_write_dropped_samples_total. The metrics exist specifically so this lesson can be applied to them.

Verification

You should now be able to answer:

• What is the difference between a log query and a metrics query in the Loki ruler, and which one does a record rule require?
• What does the ruler do when its remote-write target is unreachable, and which flag controls how long samples are queued?
• Why does the cardinality of the by clause matter more than the cardinality of the source logs?
• What signal confirms a rule has actually evaluated successfully in the last interval?
• When does a log-derived metric deserve a native replacement?

Quiz