Lab: Derive Metrics from Logs

Objective

By the end of this lab a service that exposes no Prometheus endpoint will have an error rate, a throughput figure and a p99 latency in Prometheus, produced entirely from its log lines by the Loki ruler. You will have verified the latency number against one you computed yourself from the raw file, so you know the pipeline is right rather than merely running.

You will also have seen the three ways this pipeline produces nothing while every component reports healthy: a rule of the wrong shape, an unbounded by clause, and a remote-write target that went away for two minutes. Each of the three is silent, and each is why “the rule is in the file” is not evidence.

Architecture

One host, four containers. The generator stands in for the vendor binary: it writes structured JSON and exposes nothing else. Alloy tails the file into Loki. The Loki ruler evaluates LogQL on a schedule and pushes the results into Prometheus over the remote-write protocol — Prometheus never scrapes Loki for these series, it receives them.

  +------------------+  writes   +---------------------+
  | app (generator)  | ========> | /var/log/app/       |
  | 20 lines/sec     |           |   checkout.log      |
  | JSON per line    |           +----------+----------+
  +------------------+                      | tails
                                            v
                                 +----------+----------+
                                 | alloy               |
                                 |  loki.source.file   |
                                 |  loki.write         |
                                 +----------+----------+
                                            | push
                                            v
                                 +----------+----------+
                                 | loki 3.3            |
                                 |  ingester  :3100    |
                                 |  ruler              |  <-- the component
                                 |   /etc/loki/rules/  |      this lab is about
                                 +----------+----------+
                                            | remote_write
                                            v
                                 +----------+----------+
                                 | prometheus 2.55     |
                                 |  --web.enable-      |
                                 |    remote-write-    |
                                 |    receiver  :9090  |
                                 +---------------------+

Note the direction of the bottom arrow. In every other lab in this course Prometheus pulls; here it is written to. That single reversal is the source of most of the confusion around log-derived metrics, and of two of the three silent failures below.

Requirements

A Linux host with Docker Engine 28.x and Docker Compose v2.
curl and jq on the host. Several steps compute a number from a JSON response and compare it with another number; without jq you are reading the responses by eye and the comparison stops being a measurement.
Free TCP ports 3100, 9090 and 12345, bound to loopback only.
About 1 GiB of free memory and 1 GiB of free disk. The generator writes roughly 15 MiB per hour at the default rate.
No out-of-band access requirement. Nothing outside the lab directory and the compose project is modified, and no host networking, firewall or SSH configuration is touched.

Scenario

The checkout service is a vendor-supplied binary. It writes one JSON line per request to a log file and exposes no /metrics endpoint. The vendor’s answer to the instrumentation request is a roadmap item for next year.

Meanwhile the service is in the paging path. The on-call needs three numbers: errors per second per instance, total requests per second so the errors can be expressed as a proportion, and a p99 latency. All three exist in the log lines. None of them exists as a metric.

You are going to build the bridge, and then price it — because the reason this is a lab rather than a paragraph is that the bridge is easy to build wrong in ways that look exactly like building it right.

Tasks

Task 1: Create the lab directory and the generator

LABDIR="$HOME/rb-obs-logmetrics"
mkdir -p "$LABDIR"
cd "$LABDIR"

gen-logs.sh — a POSIX shell loop that writes the vendor’s log format. The shape matters more than the volume, so read the three line types before running it:

#!/bin/sh
# Log generator for the metrics-from-logs lab.
#
# Emits RATE lines per second in three deterministic proportions per 20 lines:
#   1 line  at level=error            <- the thing we want to count
#   2 lines at level=info, whose msg  <- the decoys: they contain the
#           contains "error_count=0"     substring "error" and are not errors
#  17 lines at level=info, ordinary
#
# duration_ms is bounded except for one slow request in every 200, which is
# what puts a visible tail into the latency quantile.
set -eu

OUT=/var/log/app/checkout.log
RATE=${RATE:-20}
INSTANCE=${INSTANCE:-checkout-01}

mkdir -p "$(dirname "$OUT")"

i=0
while :; do
  n=0
  while [ "$n" -lt "$RATE" ]; do
    i=$((i + 1))
    n=$((n + 1))

    level=info
    msg="checkout handled"
    status=200

    if [ $((i % 20)) -eq 0 ]; then
      level=error
      msg="payment gateway rejected the authorisation"
      status=502
    elif [ $((i % 10)) -eq 5 ]; then
      msg="cache warm complete, error_count=0"
    fi

    if [ $((i % 200)) -eq 0 ]; then
      dur=800
    else
      dur=$(( (i % 97) * 3 + 20 ))
    fi

    printf '{"ts":"%s","level":"%s","service":"checkout","instance":"%s","request_id":"req-%s-%d","status":%d,"duration_ms":%d,"msg":"%s"}\n' \
      "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$level" "$INSTANCE" \
      "$(date -u +%s)" "$i" "$status" "$dur" "$msg" >> "$OUT"
  done
  sleep 1
done

chmod +x gen-logs.sh

Twenty lines per second gives exactly one error per second and two decoy lines per second. Those two numbers are the answer key for Task 4; write them down now, because the point of that task is that a plausible query returns neither of them.

Task 2: Configure Loki with the ruler switched on

loki-config.yaml. The ruler block is the whole subject of this lab, so the comments there are load-bearing:

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096
  log_level: info

common:
  instance_addr: 127.0.0.1
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: "2024-01-01"
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

limits_config:
  allow_structured_metadata: true
  reject_old_samples: false

ruler:
  # Exposes /loki/api/v1/rules. Without it the only way to know whether a rule
  # loaded is to read the Loki log, which is the habit this lab is trying to
  # break.
  enable_api: true

  storage:
    type: local
    local:
      # Rule files live under <directory>/<tenant>/. auth_enabled is false, so
      # the tenant is the Loki default, "fake". A file dropped directly into
      # this directory rather than into the tenant subdirectory is never seen.
      directory: /etc/loki/rules

  # Scratch space the ruler uses while evaluating. Not the rule store, despite
  # the name; it must be writable and it must not be the rule directory.
  rule_path: /loki/rules-scratch

  # A write-ahead log for samples awaiting remote write. Without it a restart
  # loses whatever had not been flushed.
  wal:
    dir: /loki/ruler-wal

  evaluation_interval: 1m
  poll_interval: 30s

  remote_write:
    enabled: true
    clients:
      prom:
        url: http://prometheus:9090/api/v1/write

analytics:
  reporting_enabled: false

Task 3: Configure Alloy, Prometheus and compose

config.alloy — deliberately minimal. Everything this pipeline needs is a bounded stream label set; no field is promoted, because the ruler parses the line itself at query time:

local.file_match "app" {
  path_targets = [{
    __path__ = "/var/log/app/*.log",
    job      = "checkout",
    service  = "checkout",
    env      = "lab",
  }]
}

loki.source.file "app" {
  targets    = local.file_match.app.targets
  forward_to = [loki.write.local.receiver]
}

loki.write "local" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
  }
}

prometheus.yml — Prometheus scrapes Loki and Alloy for their own health, but the derived series arrive by remote write, not by scrape:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: loki
    static_configs:
      - targets: ['loki:3100']

  - job_name: alloy
    static_configs:
      - targets: ['alloy:12345']

compose.yaml:

name: rb-obs-logmetrics

services:
  app:
    image: alpine:3.20
    command: ['/bin/sh', '/gen-logs.sh']
    environment:
      RATE: '20'
      INSTANCE: 'checkout-01'
    volumes:
      - ./gen-logs.sh:/gen-logs.sh:ro
      - applogs:/var/log/app

  loki:
    image: grafana/loki:3.3.0
    command: -config.file=/etc/loki/loki-config.yaml
    volumes:
      - ./loki-config.yaml:/etc/loki/loki-config.yaml:ro
      - ./rules:/etc/loki/rules:ro
      - loki-data:/loki
    ports:
      - '127.0.0.1:3100:3100'
    depends_on:
      - prometheus

  alloy:
    image: grafana/alloy:latest
    command:
      - 'run'
      - '--server.http.listen-addr=0.0.0.0:12345'
      - '--storage.path=/var/lib/alloy/data'
      - '/etc/alloy/config.alloy'
    volumes:
      - ./config.alloy:/etc/alloy/config.alloy:ro
      - applogs:/var/log/app:ro
      - alloy-data:/var/lib/alloy/data
    ports:
      - '127.0.0.1:12345:12345'
    depends_on:
      - loki

  prometheus:
    image: prom/prometheus:v2.55.1
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention.time=3h'
      # Without this flag /api/v1/write returns 404 and the ruler's pushes
      # fail with a message that mentions neither Prometheus nor this flag.
      - '--web.enable-remote-write-receiver'
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prom-data:/prometheus
    ports:
      - '127.0.0.1:9090:9090'

volumes:
  applogs:
  loki-data:
  alloy-data:
  prom-data:

Create the tenant rule directory before starting, because Loki mounts it read-only and will not create it for you:

mkdir -p "$LABDIR/rules/fake"

Service impact possiblelab host— Pulls four images and binds three loopback ports

$ docker compose up -d

sleep 30
curl -sf http://127.0.0.1:3100/ready && echo LOKI-READY
curl -sf http://127.0.0.1:12345/-/ready && echo ALLOY-READY
curl -sf http://127.0.0.1:9090/-/ready && echo PROM-READY
docker compose ps

Confirm lines are actually arriving before going further. A rule over an empty stream is indistinguishable from a broken rule:

curl -sfG http://127.0.0.1:3100/loki/api/v1/query_range \
  --data-urlencode 'query={service="checkout"}' \
  --data-urlencode 'since=2m' --data-urlencode 'limit=1' \
| jq -r '.data.result[0].values[0][1]'

One JSON line should come back. If the result is empty, stop here and read docker compose logs alloy.

Task 4: Prove the query before you write the rule

This is the task that decides whether the rest of the lab is worth anything. Two queries, same window, same stream, both plausible, and they do not agree.

Set a window first:

START=$(date -u -d '-5 min' +%Y-%m-%dT%H:%M:%SZ)
END=$(date -u +%Y-%m-%dT%H:%M:%SZ)
echo "window $START .. $END"

Query A — the line filter. This is what almost everybody writes first:

curl -sfG http://127.0.0.1:3100/loki/api/v1/query_range \
  --data-urlencode 'query=sum(rate({service="checkout"} |= "error" [1m]))' \
  --data-urlencode "start=$START" --data-urlencode "end=$END" \
  --data-urlencode 'step=60s' \
| jq -r '.data.resultType, (.data.result[0].values[-1][1] // "no data")'

Query B — the parsed level filter. The parser runs first; the comparison is against a field, not against the bytes of the line:

curl -sfG http://127.0.0.1:3100/loki/api/v1/query_range \
  --data-urlencode 'query=sum(rate({service="checkout"} | json | level=~"error|fatal" [1m]))' \
  --data-urlencode "start=$START" --data-urlencode "end=$END" \
  --data-urlencode 'step=60s' \
| jq -r '.data.resultType, (.data.result[0].values[-1][1] // "no data")'

Query B returns approximately 1.0 — one error per second, which is the number you wrote down in Task 1. Query A returns approximately 3.0, because |= "error" also matches the two decoy lines per second whose message contains error_count=0. The generator’s decoys are not a contrivance; error_count, errorRate, no errors found and ErrorHandler all appear in real application logs, and all of them match a substring filter.

Both queries return resultType: matrix, which is the shape a record rule requires. A query that returns streams is a log query and cannot be recorded.

Confirm the over-count is what you think it is by counting the decoys directly:

curl -sfG http://127.0.0.1:3100/loki/api/v1/query_range \
  --data-urlencode 'query=sum(rate({service="checkout"} | json | level="info" |= "error_count" [1m]))' \
  --data-urlencode "start=$START" --data-urlencode "end=$END" \
  --data-urlencode 'step=60s' \
| jq -r '.data.result[0].values[-1][1]'

Approximately 2.0. One plus two is three, and the arithmetic closes.

Task 5: Write the recording rules and confirm they loaded

rules/fake/checkout.yaml — three rules, each with a job. The naming follows the level:metric:operation convention from the lesson:

groups:
  - name: checkout_log_metrics
    # Matches the rate window below. An interval shorter than the window
    # produces overlapping evaluations for no extra information.
    interval: 1m
    rules:
      # The numerator: errors per second, per instance.
      - record: checkout:log_errors:rate5m
        expr: |
          sum(rate({service="checkout", env="lab"}
                   | json
                   | level=~"error|fatal" [5m])) by (instance)
        labels:
          source: log

      # The denominator: every line, so a dashboard can show a proportion
      # rather than an absolute that moves with traffic.
      #
      # The | json here is doing one job only: instance is a field in the line,
      # not a stream label, so without a parser there is nothing for by
      # (instance) to group on and the series comes back unlabelled - which
      # then refuses to divide against the numerator. The parse is not free;
      # it is the price of per-instance attribution on a stream that does not
      # carry the instance in its label set.
      - record: checkout:log_lines:rate5m
        expr: |
          sum(rate({service="checkout", env="lab"}
                   | json [5m])) by (instance)
        labels:
          source: log

      # The latency quantile, lifted straight out of the duration field.
      - record: checkout:log_duration_ms:p99_5m
        expr: |
          quantile_over_time(0.99,
            {service="checkout", env="lab"}
            | json
            | unwrap duration_ms [5m]) by (instance)
        labels:
          source: log

The source: log label on all three is worth the two characters. Once these series are in Prometheus alongside native ones, {source="log"} is the only way to answer “which of our panels are derived?” — which is the question you will be asked when the ruler starts costing money.

The ruler picks up the file at the next poll_interval, so no restart is needed. Wait, then ask the ruler what it thinks it has:

sleep 45
# What the ruler has loaded, in its own format. This endpoint answers with the
# rule configuration as YAML, not JSON, so read it rather than piping it to jq.
curl -sf http://127.0.0.1:3100/loki/api/v1/rules

# The Prometheus-compatible view. This one is JSON and carries the evaluation
# state, which is the part you actually need.
curl -sf http://127.0.0.1:3100/prometheus/api/v1/rules \
| jq -r '.data.groups[].rules[] | "\(.name)  last=\(.lastEvaluation)  err=\(.lastError // "none")"'

Three rule names, each with a lastEvaluation timestamp inside the last minute and no lastError. This is the only authoritative statement that a rule exists; the file on disk is a request, not a fact.

Now confirm the samples arrived at the other end. They came in by remote write, so they are in Prometheus without Prometheus ever having scraped for them:

for M in checkout:log_errors:rate5m checkout:log_lines:rate5m checkout:log_duration_ms:p99_5m; do
  printf '%-38s ' "$M"
  curl -sfG http://127.0.0.1:9090/api/v1/query \
    --data-urlencode "query=$M" \
  | jq -r '.data.result[0].value[1] // "ABSENT"'
done

Expect roughly 1, 20 and a few hundred. If all three read ABSENT after two minutes, the failure is between the ruler and Prometheus; Task 8 is the diagnosis for that, and you can jump there and come back.

Task 6: Verify the quantile against a number you computed yourself

A derived metric that nobody has checked is a plausible-looking number. Compute the p99 from the raw file and compare.

docker compose exec -T app sh -c 'tail -n 6000 /var/log/app/checkout.log' \
| jq -r '.duration_ms' \
| sort -n \
| awk '{v[NR]=$1} END {printf "p99 from file: %s (n=%d)\n", v[int(NR*0.99+0.5)], NR}'

curl -sfG http://127.0.0.1:9090/api/v1/query \
  --data-urlencode 'query=checkout:log_duration_ms:p99_5m' \
| jq -r '.data.result[] | "p99 from ruler: \(.value[1]) (instance \(.metric.instance))"'

The two will not be identical and should not be: the file sample is the last 6,000 lines and the rule’s window is the last five minutes, so they cover different — overlapping — spans, and quantile_over_time interpolates. They should land within a few tens of milliseconds of each other and on the same side of the slow-request tail. If the ruler’s number is an order of magnitude out, or if it is suspiciously round, the unwrap is not doing what you think.

Why unwrap silently drops lines, and why that matters here

| unwrap duration_ms turns each surviving log line into a (timestamp, value) pair. A line reaches the unwrap only if it parsed as JSON, and it survives the unwrap only if the named field is present and parses as a number. Lines that fail either test are dropped — not counted as zero, not counted as an error, just absent from the series.

That is the correct behaviour and it is also the trap. A quantile over a population that has silently lost 40% of its members is still a number, still a plausible one, and carries no marker saying which lines it is missing. If the vendor ships a build that logs "duration_ms":"124ms" instead of 124, this p99 does not break: it narrows, quietly, to the subset that still parses.

The defence is to watch the parse failures as their own series. {service="checkout"} | json | __error__ != "" selects exactly the lines the parser could not read, and a rate over it is a metric worth recording next to the quantile it protects.

Task 7: Two rules that produce nothing

The first is a rule of the wrong shape. Append it to the same file:

      # Deliberately wrong: the expr returns log lines, not a metric.
      - record: checkout:log_broken:rate5m
        expr: |
          {service="checkout", env="lab"} |= "error"

Wait for the poll, then read what your Loki version did with it:

sleep 45
curl -sf http://127.0.0.1:3100/prometheus/api/v1/rules \
| jq -r '.data.groups[] | "\(.name): \(.rules | length) rule(s)"'

docker compose logs --tail=40 loki | grep -i 'rule\|group' || echo "nothing in the log"

Two outcomes are possible and both are worth seeing. Either the ruler refuses the group and the other three rules go with it — one bad rule taking down its whole group is the important part — or the group loads and the fourth rule never produces a sample. Record which one happened on your version; it changes how you would notice it in production. Then remove the rule and confirm the group recovers:

# Delete the four appended lines, then wait for the poll.
sed -i '/checkout:log_broken:rate5m/,$d' "$LABDIR/rules/fake/checkout.yaml"
sed -i '/Deliberately wrong/d' "$LABDIR/rules/fake/checkout.yaml"
sleep 45
curl -sf http://127.0.0.1:3100/prometheus/api/v1/rules \
| jq -r '.data.groups[].rules[].name'

The second is the unbounded by clause. Record the current series count first:

curl -sfG http://127.0.0.1:9090/api/v1/query \
  --data-urlencode 'query=count({source="log"})' | jq -r '.data.result[0].value[1]'
curl -sfG http://127.0.0.1:9090/api/v1/query \
  --data-urlencode 'query=prometheus_tsdb_head_series' | jq -r '.data.result[0].value[1]'

Now append the rule the lesson warns about — one series per request id, forever:

      # Deliberately unbounded: request_id is unique per line.
      - record: checkout:log_requests:rate5m
        expr: |
          sum(rate({service="checkout", env="lab"}
                   | json [5m])) by (request_id)
        labels:
          source: log

Wait through two evaluations and take the same two measurements again:

sleep 150
curl -sfG http://127.0.0.1:9090/api/v1/query \
  --data-urlencode 'query=count({source="log"})' | jq -r '.data.result[0].value[1]'
curl -sfG http://127.0.0.1:9090/api/v1/query \
  --data-urlencode 'query=prometheus_tsdb_head_series' | jq -r '.data.result[0].value[1]'

Two evaluations over a five-minute window at twenty lines per second is on the order of six thousand distinct request ids, and the head series count moves by about that much. Nothing errors. Prometheus accepts every one of them, because from its side these are ordinary series arriving over an ordinary protocol.

Remove this rule before continuing

Left running, this rule adds several thousand series per evaluation for as long as the lab is up. On a lab host that is a nuisance; the same rule on a shared Prometheus is the incident. Remove it now:

sed -i '/checkout:log_requests:rate5m/,$d' "$LABDIR/rules/fake/checkout.yaml"
sed -i '/Deliberately unbounded/d' "$LABDIR/rules/fake/checkout.yaml"

The series already written do not disappear when the rule does. They stop receiving samples, go stale, and remain queryable for the retention period — three hours in this compose file, and typically weeks in production. The rule was live for five minutes; the cost is paid for the whole retention window.

Task 8: Take the remote-write target away

The ruler is a writer. When the thing it writes to stops answering, rule evaluation carries on succeeding — the LogQL still runs, the samples are still produced — and nothing downstream receives them.

Service impact possiblelab host— Stops the remote-write target; the ruler keeps evaluating into a void

$ docker compose stop prometheus

sleep 120
docker compose logs --tail=30 loki | grep -iE 'remote|write|push' || echo "no remote-write lines logged"

curl -s http://127.0.0.1:3100/metrics \
| grep -E '^(loki_ruler_|prometheus_remote_storage_)' | head -20

Read what your build actually exposes rather than looking for a name you expect. The Loki ruler embeds the Prometheus remote-write client, so the queue and failure counters usually appear under the prometheus_remote_storage_ prefix inside Loki’s own /metrics, while rule scheduling appears under loki_ruler_. Note the names your version uses; they are what you would alert on.

Bring it back and look at the shape of the recovery:

docker compose start prometheus
sleep 90

curl -sfG http://127.0.0.1:9090/api/v1/query_range \
  --data-urlencode 'query=checkout:log_errors:rate5m' \
  --data-urlencode "start=$(date -u -d '-10 min' +%Y-%m-%dT%H:%M:%SZ)" \
  --data-urlencode "end=$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
  --data-urlencode 'step=60s' \
| jq -r '.data.result[0].values[] | "\(.[0]) \(.[1])"'

Whether the outage window is a gap or is backfilled from the ruler WAL depends on how long the target was away against how much the WAL held. Report what you see rather than what you hoped for — that distinction is the whole content of the post-incident question “did we lose the data or delay it?”.

What happens if an alert depends on this series?

A Prometheus alert on a series that stops receiving samples does not fire. It goes to no-data, which is a different state from “the condition is false” and is treated differently by every alerting stack. So a two-minute outage of the remote-write target silences every alert built on every log-derived metric for those two minutes, and the silence is indistinguishable from health on the dashboard.

A native metric does not have this dependency. The scrape either succeeds or up goes to zero, and up == 0 is itself alertable. The derived path has no equivalent: there is no up for a rule. The closest substitute is an alert on the age of the derived series itself — time() - timestamp(checkout:log_errors:rate5m) > 300 — and you have to write it deliberately, because nothing produces it for you.

Validation

1. Three rules loaded and evaluating. Every rule reports a lastEvaluation within the last two minutes and no lastError:

curl -sf http://127.0.0.1:3100/prometheus/api/v1/rules \
| jq -r '.data.groups[].rules[] | select(.lastError != null and .lastError != "")
         | "FAIL \(.name): \(.lastError)"' \
| grep . || echo "PASS: no rule is reporting an error"

2. All three series present in Prometheus, and only those three. The unbounded rule and the broken rule are gone:

curl -sfG http://127.0.0.1:9090/api/v1/query \
  --data-urlencode 'query=count by (__name__) ({source="log"})' \
| jq -r '.data.result[] | "\(.metric.__name__) \(.value[1])"'

Expect exactly three metric names, each with a series count equal to the number of instances — one, here. A fourth name, or a count in the thousands, means a rule from Task 7 is still live.

3. The error rate matches the generator. The rule’s value is within ten per cent of one error per second:

curl -sfG http://127.0.0.1:9090/api/v1/query \
  --data-urlencode 'query=checkout:log_errors:rate5m' \
| jq -r '.data.result[0].value[1] | tonumber
         | if . > 0.9 and . < 1.1 then "PASS \(.)" else "FAIL \(.)" end'

4. The proportion is computable. The denominator series makes the error rate expressible as a fraction, which is the form an SLO needs:

curl -sfG http://127.0.0.1:9090/api/v1/query \
  --data-urlencode 'query=checkout:log_errors:rate5m / checkout:log_lines:rate5m' \
| jq -r '.data.result[0].value[1]'

Approximately 0.05 — one line in twenty.

5. The line filter and the parsed filter are both written down, with the ratio between them, in your notes. This is the deliverable that survives the lab.

Expected Outcome

Four containers running, with a Loki whose ruler holds one group of three rules and reports recent successful evaluations for all of them.
Three series in Prometheus — checkout:log_errors:rate5m, checkout:log_lines:rate5m, checkout:log_duration_ms:p99_5m — all carrying source="log", all arriving by remote write and none by scrape.
A recorded comparison showing the line filter over-counts errors roughly threefold against the parsed filter on the same window.
A p99 from the ruler and a p99 computed from the raw file, in the same ballpark, both written down.
Notes on what your Loki version did with the wrong-shaped rule, what the unbounded rule cost in series, and whether the remote-write outage produced a gap or a backfill.

Troubleshooting

The rules endpoints return nothing, or an empty group list. The file is in the wrong directory. Rule files live under <storage.local.directory>/<tenant>/, and with auth_enabled: false the tenant is fake. Confirm with docker compose exec loki ls -R /etc/loki/rules.

Loki refuses to start after the ruler block is added. Read the error; it names the field. Then validate the file directly: docker compose run --rm loki -config.file=/etc/loki/loki-config.yaml -verify-config. Configuration keys under ruler have moved between Loki releases, so check the key against the configuration reference for the version you are running rather than against a blog post.

The rules load and evaluate but no series reach Prometheus. Prometheus is missing --web.enable-remote-write-receiver, so /api/v1/write returns 404. Test it directly: curl -s -o /dev/null -w '%{http_code}\n' -XPOST http://127.0.0.1:9090/api/v1/write returns 404 when the receiver is off and 400 when it is on and you sent it nonsense — 400 is the healthy answer here.

A rule’s lastError mentions parse or type errors. Run the same expr against /loki/api/v1/query_range by hand. A LogQL error is much easier to read from the query API than from a rule evaluation.

resultType is streams when you expected matrix. The expression is a log query. It needs an aggregation over a range — rate, count_over_time, quantile_over_time — before a record rule can use it.

The p99 series is absent while the other two are present. Either no line parsed, or duration_ms is not a number in the lines that did. Check with {service="checkout"} | json | __error__ != "" over the same window; a non-empty result names the parse failure.

The p99 value is far below the file-computed one. The unwrap is dropping the slow requests rather than including them. Confirm the tail exists in the source: docker compose exec -T app sh -c 'tail -n 2000 /var/log/app/checkout.log' | jq -r '.duration_ms' | sort -n | tail -3.

Alloy starts but nothing is ingested. The applogs volume is mounted read-only into Alloy and read-write into app; if the generator container is not running there is nothing to tail. docker compose ps and docker compose logs app.

Cleanup

Data-loss risklab host— Removes four containers and this project's four named volumes

$ cd ~/rb-obs-logmetrics && docker compose down -v

docker volume ls | grep rb-obs-logmetrics || echo "volumes gone"

Keep the rule file and your measurements, then remove the directory:

mkdir -p "$HOME/rb-obs-deliverables"
cp -a "$HOME/rb-obs-logmetrics/rules/fake/checkout.yaml" \
      "$HOME/rb-obs-deliverables/checkout-log-rules.yaml"

rm -rf "$HOME/rb-obs-logmetrics"

The images stay in the local cache. Remove them if you want the disk back:

docker image rm grafana/loki:3.3.0 grafana/alloy:latest \
  prom/prometheus:v2.55.1 alpine:3.20

Production notes

Every derived metric needs a decommission date in the rule file. The reason this pipeline exists is that the vendor has no /metrics endpoint. When the endpoint arrives, the rule should be deleted in the same commit that adds the scrape config — otherwise you are paying for the same number twice and, worse, you now have two numbers that will eventually disagree. Put the date in a comment above the rule, and treat a rule past its date as a review item rather than as furniture.

The change window is short and the blast radius is not. Dropping a rule file into the ruler’s directory takes effect at the next poll with no restart, which makes it feel like a small change. It is not: a single unbounded by clause is a cardinality incident on a shared Prometheus, and you measured how fast it moves. Treat the rule directory the way you treat prometheus.yml — code review, a check of the by clause against the allowed label list, and a first deploy to a staging tenant.

Alert on the pipeline, not only through it. Three signals are worth having before you depend on a derived metric in the paging path: the rule’s own lastError and evaluation lag from the ruler API, the remote-write failure counters from Loki’s /metrics, and the age of the derived series in Prometheus. The third catches everything the first two miss, including the case where the whole ruler is gone.

Size the derived path against peak, not average. The cost of a rule is the number of lines inside the window, evaluated on every interval, whether or not anybody is looking at the panel. A service that triples its log volume during an incident triples the cost of every rule that reads it — at exactly the moment the ruler’s output matters most. That is the argument for keeping the derived set small and the windows short, and for putting the migration to native instrumentation on somebody’s roadmap rather than on the backlog.

Objectives

Prerequisites