Logs vs Native Metrics

A shared Prometheus for the platform team starts missing scrape intervals. Investigation: one team added a derived metric for “log lines per service per request_id”. Another team added a native counter on every endpoint. The native counter is bounded and cheap. The derived metric is unbounded — every request_id is a new series — and the ingester is rejecting 70% of the pushes. The on-call engineer is told to “investigate the high cardinality” and discovers that the two teams’ metrics are both implicated, but only one of them is the right tool.

This is the decision the lesson is about. Native metrics and log-derived metrics are not interchangeable; each has a cost model and a ceiling. Picking the wrong one for the question costs the platform.

What it is

The logs-versus-native-metrics question is “given an operational question that needs a time series, should the series be produced by a Prometheus instrumentation library or by a LogQL query against the log stream?”. The answer is not “always the same”. The two have different cost models:

• Native metrics — sampled at the source by the application’s instrumentation, scraped at a fixed interval, stored as a sample per series per interval. Cost scales linearly with the number of series and is bounded by the cardinality of the labels.
• Log-derived metrics — computed by the Loki ruler evaluating a LogQL expression on a fixed interval, scanning every line in the window. Cost scales with log throughput, with parser cost, and with the cardinality of the by clause.

The substitution works for low-cardinality, low-throughput metrics (error rate, request rate, latency). The substitution fails for high-cardinality, high-throughput metrics (per-user counters, per-trace histograms).

Why a sysadmin cares

The cost of the wrong choice is paid by the platform team and by every team that shares the platform.

• A log-derived metric on a high-cardinality label (per-user count, per-request histogram) is unbounded cardinality in Loki, which becomes unbounded series in the target store. The cost is shared across every dashboard, every alert, every query that scans the affected metric.
• A native metric on a low-cardinality question (service error rate) is one scrape per interval per series, regardless of the request volume. The cost is paid by the producing service in CPU, not by the platform in storage.

The right answer for the platform team is to know the boundary and enforce it: log-derived metrics for what native cannot do, native metrics for everything native can do.

How it works

Two cost models, side by side. For a single metric with N labels and V values per label, scraped at interval I:

NATIVE METRIC                  LOG-DERIVED METRIC
--------------                 ------------------
cost = series * 1 sample       cost = series * (window / I) * lines_per_second
                              where series = selector cardinality
                              and lines_per_second = total log throughput
                              in the matched streams

For 100 series,                For 100 rows over a 5-minute window,
1 sample per 15s:              at 1000 lines/s:
= 100 * (86400/15)             = 100 * (300/60) * 1000
= 576,000 samples/day          = 500,000 line parses per evaluation
                                * 96 evaluations per day
                                = 48,000,000 line parses per day

The native path parses one label set per series and produces one sample per interval. The log-derived path parses every log line in the matching streams, every evaluation, regardless of whether the line is the metric in question.

The substitution works when:

• The labels are bounded and small.
• The log volume is manageable.
• The window is short enough that the parse cost is within the ruler’s CPU budget.

The substitution fails when:

• The labels are unbounded (per-user, per-trace, per-request).
• The log volume is large enough that the parse cost dominates.
• The window is long enough that the parse cost exceeds the ruler’s evaluation interval.

How to configure it

There is no “log-derived metrics” or “native metrics” config block to write. The decision is at design time: when an operational question arrives, the team picks the right tool. The configurations on both sides are documented elsewhere in the course; what matters here is the decision matrix.

                log volume        log volume
                low (<10 KiB/s)   high (>1 MiB/s)
                --------------    --------------
labels    low   EITHER. Native    NATIVE wins on
bounded         is cheaper; log-  cost. Log-derived
                derived is fine   is wasteful.
                for prototyping.
                ---------------   ---------------
labels    high  LOG-DERIVED for   NEITHER. Cardinality
unboun-         prototyping;      bomb at any volume.
ded             native is impos-  Trace the question
                sible.            instead.

Two patterns appear in production:

• The “interim signal” pattern. A new service ships logs before its native counters land. A log-derived metric is added as the interim signal; the rule file includes a decommission: comment with the date and the ticket number for the native metric.
• The “vendor-owned binary” pattern. A mainframe adapter emits a duration in logs but no metric. A log-derived latency rule is added and never decommissioned (no native metric possible). The rule is documented as a permanent log-derived metric in the runbook.

The first pattern has a decommission date. The second does not, and that is fine. What is not fine is a permanent log-derived metric that has forgotten which pattern it is.

The query shapes that work for the substitution:

# Substitutes for a native counter (e.g. http_requests_total).
sum(rate({job="checkout"} | json | status="5xx" [5m])) by (route)

# Substitutes for a native histogram (e.g. http_request_duration_seconds_bucket).
quantile_over_time(0.99, {job="checkout"} | json | unwrap duration_ms [5m]) by (route)

# Substitutes for a native gauge (e.g. active_sessions).
# Approximation only — logs are not a census of current state.
count_over_time({job="checkout"} | json | event="session_start" [5m])
  -
count_over_time({job="checkout"} | json | event="session_end" [5m])

The gauge approximation is the boundary. Logs are append-only; gauges are point-in-time. The substitution works for rate (increments over a window) and percentage (ratio of two rates). It does not work for instantaneous state. A “current sessions” metric from logs is a delta of starts and ends, which drifts on every missed end event.

How to validate it

The validation is a decision review, not a command. Three questions to ask of any proposed log-derived metric:

1. Is there a native metric that answers the same question?
   If yes: use the native metric. Stop here.
   If no: continue.

2. Is the cardinality of the by clause bounded?
   If no: the question is wrong, not the tool. Find the
          bounded subset, or use traces / logs with a join key.
   If yes: continue.

3. Is the log volume inside the window within the ruler's budget?
   If no: shorten the window, narrow the selector, or accept
          that the answer is too expensive to compute at this
          cadence. Consider a sampling strategy at the agent.
   If yes: write the rule with a decommission date.

Concrete commands for the third check:

# READ-ONLY: estimate the parse cost of the proposed query.
# Sum rate over the query window — the result is roughly the
# number of lines the ruler will parse per evaluation.
curl -s -u "$LOKI_USER:$LOKI_PASS" \
  --data-urlencode 'query=sum(rate({job="checkout"} [5m]))' \
  -G http://loki.internal:3100/loki/api/v1/query \
  | jq '.data.result[0].value[1]'
# "105.20"  (about 105 lines/s for the checkout job)

# READ-ONLY: confirm the rule cardinality in the target.
curl -s -u "$MIMIR_USER:$MIMIR_PASS" \
  --data-urlencode 'query=count(app_log_latency_p99)' \
  -G http://mimir.internal:9009/prometheus/api/v1/query \
  | jq '.data.result[0].value[1]'
# "6"

105 lines/s at a 1-minute evaluation interval is roughly 6,300 lines per evaluation — trivial. 100,000 lines/s is 6 million per evaluation — outside the ruler’s comfort zone. The boundary is somewhere in between; the actual number depends on the ruler’s CPU and the network’s remote-write budget.

How it can fail

1. The log-derived metric replaces a native metric that exists. The team does not realise the native counter exists; the derivation is added on top; the cost is paid twice. Symptom: the platform carries two metrics answering the same question; one is the canonical answer; the other is the noisy, expensive copy.
2. The by clause creeps. A rule starts with by (instance). Someone adds , route. Someone adds , user_id. The cardinality is now unbounded. Symptom: loki_ruler_remote_write_dropped_samples_total rises; the target store rejects.
3. Log volume grows. A debug log is added in a hot path; the parse cost of the log-derived metric doubles. Symptom: ruler CPU climbs; rule evaluations miss their interval.
4. The decommission date is missed. The native metric lands; the rule is not deleted. Symptom: the platform pays for two metrics indefinitely; the rule file accumulates zombie rules.
5. The “interim signal” becomes a permanent signal. A team uses the log-derived metric past the migration window because the rule “just works”. Symptom: the decommission date in the runbook is two years past; the rule file is the platform’s largest single source of CPU.
6. The gauge approximation drifts. A “current sessions” metric from starts-minus-ends drifts on every missed end event. Symptom: the gauge reads 50 when the actual count is 3; the dashboard is wrong; the alert based on it is silent.

How to troubleshoot it

The diagnostic order is: is the question right, is the cardinality right, is the cost right, is the migration on schedule.

1. Question right? Is there a native metric that answers the same question with the same labels? If yes, the log- derived metric should not exist.
2. Cardinality right? count by (__name__) of the produced metric in the target store. A number that grows under steady traffic means the by clause has crept.
3. Cost right? loki_ruler_evaluation_seconds_sum over the rule group. A rising cost under steady traffic means log volume has grown or the window has been widened.
4. Migration on schedule? Audit the rule file for decommission: annotations. A rule whose annotation is past is a rule that should have been deleted.

Security implications

• The substitution inherits the labels of the source. A log-derived metric whose by includes user_id replicates user identifiers into the metric store. The same audit that applies to native metrics applies to derived ones.
• The query budget is a budgeted resource. A platform team that allows unbounded log-derived metrics invites a cardinality attack (deliberate log flooding that drives ruler CPU). The fix is the same as for native metrics: a label allowlist, a cardinality budget, an alert on loki_ruler_remote_write_dropped_samples_total.
• The migration comment is an audit artefact. A rule file without decommission dates is a rule file the platform team has stopped reviewing. Audit quarterly.

Performance implications

• Native metric cost is the lower bound. A scrape is one request, one parse, one sample per series. The cost is paid by the source in instrumentation overhead and by the platform in samples-per-second of head writes.
• Log-derived metric cost is the upper bound. A query is one LogQL evaluation per interval, scanning every chunk in the window, parsing every matching line. The cost is paid by the platform in CPU.
• The right metric is the cheaper one. The right answer is the cheaper path when both are feasible. The log-derived path is correct only when the native path is not feasible.
• Cardinality cost is the same in either path. A series costs roughly the same to store, regardless of how it was produced. The cost difference is in the production, not in the storage.

Production guidance

• Maintain a written rule for the substitution. The cardinal rule: native metrics answer native questions; log-derived metrics answer questions native metrics cannot.
• Audit every log-derived metric for a decommission date. A rule without one is a rule the platform team has stopped reviewing.
• Set a cardinality budget per metric. A label policy that allows 100 series per metric is fine; one that allows 1 million is a platform incident waiting to happen.
• Watch loki_ruler_evaluation_seconds_sum. A rising cost is the leading indicator of a query that should not exist.
• Use traces and logs for per-entity questions. The trace ID is the join key. Metrics are the wrong tool.

Verification

You should now be able to answer:

• What is the cardinal rule that decides between a native metric and a log-derived metric, and how is the cost model different in each case?
• Why does a log-derived per-user counter fail before the cardinality calculation, and what is the right tool for the per-entity question?
• What three questions should a team ask before adding a new log-derived metric?
• What is the decommission discipline, and why does it matter?
• How does the gauge approximation from logs drift, and what is the correct tool for an instantaneous-state question?

Quiz