Log-Based Alerting

The cluster has been silent for ten minutes. No metric series from the checkout pods; the Prometheus scrape target list shows the targets as down; the alert that should have fired when the targets went down at 02:53 has not fired. Investigation: the Prometheus alerting rules are configured correctly, but the targets went down because the entire node went down, and the rule that depends on up == 0 from Prometheus cannot fire because Prometheus cannot reach the network to scrape. The alert that would have fired is the one watching the log stream: “no logs received from this service in the last five minutes”.

Loki-based alerting is not a substitute for native alerting; it is the second pair of eyes that fires when the native path is its own the thing that broke. Used sparingly and with care, it catches the failure shape that nothing else can.

What it is

A log-based alert is a Loki ruler rule whose expr returns a metric (a number per series per evaluation) and whose threshold compares that metric to a value. When the comparison holds for the configured for duration, the ruler fires the alert and forwards it to the configured alertmanager backend.

The Loki ruler uses the same rule format as Prometheus, with two extensions:

• The expr is a LogQL expression, not PromQL.
• The alert forwards to alertmanager (or any compatible webhook), not to the Prometheus alertmanager via the same /api/v1/alerts endpoint that Prometheus uses. The ruler exposes its rules via /loki/api/v1/rules.

# /etc/loki/rules/prod-eu/alerts.yaml
groups:
  - name: app_alerts
    interval: 1m
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate({job="checkout", cluster="prod-eu"}
                   | json
                   | level=~"error|fatal" [5m])) by (instance)
            > 0.5
        for: 5m
        labels:
          severity: warning
          team: checkout
        annotations:
          summary: 'High error rate on {{ $labels.instance }}'
          description: '{{ $value }} errors/s on {{ $labels.instance }}'

The query reads as: “if the error rate per instance is above 0.5/s for five minutes, fire the alert”. The for: 5m suppresses single-evaluation noise.

Why a sysadmin cares

Loki-based alerts earn their place in three specific situations:

• The native alerting path is the thing that broke. A node failure, a Prometheus outage, or a service crash means the native metrics are missing. The Loki ruler is independent; an alert on the log stream is the alert that fires when the service is silent.
• A new service has logs but no native counters. Until the native instrumentation lands, the log-based alert is the only alert. It is the interim signal.
• A question that native metrics cannot answer. “Did this specific error message appear in the last hour?” — a metric counter for an error message would explode cardinality; a log-based alert on the message pattern is bounded.

Outside these cases, native alerting is the right answer. The Loki ruler evaluates every alert every minute against every matching chunk in the window; the cost is paid whether the alert fires or not.

How it works

The Loki ruler evaluates alert rules on the same interval used for recording rules. Each evaluation produces a vector; the alert compares each series in the vector to the threshold. Series that hold the comparison for for are marked pending, then firing. Firing alerts are forwarded to alertmanager.

   LogQL query             Loki ruler                Alertmanager
   ------------            ----------                ------------
 rate(... | json ...)  -->  evaluate @ t     -->  series > threshold?
                                |                     |
                                |  for: 5m            |
                                v                     v
                           pending state        firing state
                                                     |
                                                     v
                                              /loki/api/v1/alerts
                                              (or webhook)

Three pieces to understand:

• for: is the dwell time. The alert must hold the threshold for the full for duration before firing. A single-evaluation spike does not fire.
• keep_firing_for: (Loki 3.x) extends the firing window past the first evaluation where the alert no longer holds. Useful when the alert flaps; without it, a brief drop resolves the alert and the next spike fires a new alert.
• expr returns a metric. A log query ({job="x"} |= "y") is not a metric; an alert on it produces nothing.

The alert label set (severity, team) is what alertmanager uses to route the alert. A Loki alert without severity is routed by default to the alertmanager default route; a Loki alert without team is unattributed.

How to configure it

The alert rule (already shown above) and the Loki ruler config:

# /etc/loki/config.yaml (microservices mode excerpt)
ruler:
  enabled: true
  storage:
    type: local
    local:
      directory: /etc/loki/rules
  rule_path: /etc/loki/rules/fake
  alertmanager_url: http://alertmanager.internal:9090
  # Optional: relabel alerts before they reach alertmanager.
  alert_relabel_configs:
    - source_labels: [severity]
      regex: warning
      action: keep
    - source_labels: [team]
      regex: .+
      action: keep
  evaluation_interval: 1m
  poll_interval: 1m
  query_timeout: 30s
  # Notification rate limit. Defaults to 1 per group interval.
  alertmanager_config: |
    basic_auth:
      username: loki-ruler
      password: ${ALERTMANAGER_PASSWORD}

A complete alert rule set for the production use cases:

# /etc/loki/rules/prod-eu/alerts.yaml
groups:
  - name: app_alerts
    interval: 1m
    rules:

      # Error rate above 0.5/s for 5 minutes.
      - alert: HighErrorRate
        expr: |
          sum(rate({job=~"checkout|api", cluster="prod-eu"}
                   | json
                   | level=~"error|fatal" [5m])) by (instance)
            > 0.5
        for: 5m
        keep_firing_for: 5m
        labels:
          severity: warning
          team: checkout
        annotations:
          summary: 'High error rate on {{ $labels.instance }}'
          description: 'Rate {{ $value | printf "%.2f" }} errors/s for 5m'

      # Tenant-level volume above 75% of the configured cap.
      - alert: LokiIngestionApproachingLimit
        expr: |
          sum(rate({cluster="prod-eu"} [5m]))
            / 16777216 > 0.75
        for: 5m
        labels:
          severity: warning
          team: observability
        annotations:
          summary: 'Loki ingestion at {{ $value | humanizePercentage }} of cap'

      # NO LOGS at all — the alert that catches complete silence.
      # count_over_time returns the number of lines in the window;
      # ==0 means no logs received.
      - alert: NoLogsReceived
        expr: |
          count_over_time({job="checkout", cluster="prod-eu"}[5m]) == 0
        for: 5m
        labels:
          severity: critical
          team: checkout
        annotations:
          summary: 'No logs received from {{ $labels.job }} on {{ $labels.instance }}'
          description: 'count_over_time == 0 for 5m; service may be silent or down'

The “no logs” alert deserves a closer look. count_over_time counts the lines that match the selector over the window. When the count is zero for the window, the alert fires; when the count is non-zero, it does not. The alert is the canonical “silence detector” — the alert that catches the service that has gone dark, including the case where Prometheus cannot scrape because the service is unreachable.

A few notes on the shape:

• keep_firing_for: 5m suppresses flap. A brief dip below the threshold does not resolve the alert; the alert stays firing for the additional 5 minutes.
• for: 5m is the dwell. A single evaluation that crosses the threshold does not fire; five consecutive evaluations do.
• The “no logs” alert is critical severity. A service that has stopped emitting logs is a service that has stopped observability; the alert is the only signal.
• The threshold for the error rate is a per-instance rate, not a percentage. A 0.5 errors/s threshold means “any instance producing more than one error every two seconds”. A percentage threshold would need a denominator (total log volume per instance) which is a second rule.

How to validate it

Three signals confirm the alert is live:

# READ-ONLY: the alert is loaded and evaluates.
curl -s -u "$LOKI_USER:$LOKI_PASS" \
  http://loki.internal:3100/loki/api/v1/rules \
  | jq '.data.groups[] | select(.name=="app_alerts") | .rules[] | {name, state, lastEvaluation, alerts: .alerts[]}'

# READ-ONLY: the alertmanager received the alert (if firing).
curl -s -u "$AM_USER:$AM_PASS" \
  http://alertmanager.internal:9090/api/v2/alerts \
  | jq '.[] | select(.labels.alertname=="HighErrorRate") | {labels, status}'

# READ-ONLY: the ruler is forwarding alerts at the expected rate.
curl -s http://loki.internal:3100/metrics \
  | grep -E '^loki_ruler_alerts_(sent|invalid)_total' \
  | head

A working alert appears in the rules API with a recent lastEvaluation, in alertmanager when the threshold is held, and in the ruler’s own metrics as a positive alerts_sent_total.

The “no logs” alert has a separate validation. Trigger the alert by silencing the agent for five minutes and confirm the alert fires; restore the agent and confirm the alert resolves within keep_firing_for.

How it can fail

1. The alert is on a log query, not a metric. A rule whose expr is {job="x"} |= "error" produces a stream, not a metric. The threshold comparison produces nothing. Symptom: the rule evaluates every minute but never fires.
2. The alert never fires because the threshold is wrong. A threshold of > 100 against an error rate that tops out at 0.5/s never holds. Symptom: the rule evaluates; no alert ever appears; the team believes Loki alerting is broken.
3. The alert floods. A threshold of > 0 on a query that matches every noisy service fires on every evaluation; the alertmanager receiver is overwhelmed. Symptom: for: 5m does not help because the alert holds the threshold permanently.
4. Ruler back-pressure from a heavy alert. An alert whose LogQL scans gigabytes per evaluation forces subsequent rules to wait. Symptom: loki_ruler_evaluation_missed_total rises; some alerts lag by several minutes.
5. The alertmanager URL is wrong. A typo in the URL or a missing basic-auth credential means the ruler evaluates but cannot forward. Symptom: loki_ruler_alerts_invalid_total rises; no alerts appear in alertmanager; the rules API shows the alert in firing state.
6. The “no logs” alert fires during a deploy. A deploy takes longer than for: 5m. Symptom: the alert fires during routine work; the team learns to ignore it.

How to troubleshoot it

The diagnostic order: is the rule loaded, is the alert firing, is the alertmanager receiving, is the alert correct.

1. Rule loaded? GET /loki/api/v1/rules. A missing lastEvaluation means the ruler is stuck; check the file syntax with promtool check rules.
2. Alert firing? The rules API returns the alert state (inactive, pending, firing). An alert that never leaves inactive means the threshold is wrong; an alert that stays in pending means for is too long.
3. Alertmanager receiving? GET /api/v2/alerts on alertmanager. A firing Loki alert not appearing here means either a network failure or a label-routing problem on alertmanager.
4. Alert correct? Compare the alert’s evaluation against the same expr at the Loki query API. A difference between the two means the ruler is evaluating with a different evaluation timestamp or a different label set.
5. Ruler capacity? loki_ruler_evaluation_missed_total. Non-zero means the ruler cannot keep up.
6. Inspect ruler logs. /var/log/loki/ruler.log records each alert state transition with the rule name and the evaluation timestamp.

Security implications

• The alert carries the labels of the source. A Loki alert whose expr extracts user_email and emits it as a label replicates the email into alertmanager, into the alert notification, into PagerDuty / Slack. Audit the alert’s by clause and the alert template for sensitive labels.
• The query budget is budgeted. A platform that allows arbitrary Loki alerts invites a ruler-CPU attack (deliberate log flooding that drives heavy alert evaluations). The fix is rate-limiting at the agent and a cardinality budget per alert.
• The “no logs” alert is a denial-of-service signal. A legitimate service that goes silent during a deploy can page the on-call; a malicious actor who can suppress log emission can trigger the alert deliberately. The right posture is to treat the “no logs” alert as a service-down alert, not as a security alert, and to back it with the for: window.

Performance implications

• Ruler CPU scales with alert cost. A heavy alert (one whose LogQL scans gigabytes) forces the ruler to wait; subsequent alerts in the same group lag. Match the alert’s parse cost to the evaluation interval.
• Alertmanager is the rate-limit. Every firing alert is a POST to /api/v1/alerts; a flood of alerts overwhelms alertmanager. Use keep_firing_for to suppress flap and for to suppress single-evaluation spikes.
• Alert storage scales with alert count. Each firing alert is a row in the ruler’s storage. A long-lived alert that stays firing for a week is one row; a flapping alert that fires and resolves ten times is ten rows. Prefer keep_firing_for over resolution flapping.
• The “no logs” alert is cheap. A count_over_time query is an index lookup, not a full scan. The cost is bounded by the selector cardinality.

Verification

You should now be able to answer:

• When does a log-based alert earn its place, and when is a native alert the right answer?
• What does for: control, and what does keep_firing_for: add on top of it?
• What is the shape of the alert that catches “no logs at all”, and what window should it use?
• Which Loki self-metric catches a ruler that cannot keep up with heavy alert evaluations?
• What is the failure shape when the alertmanager URL is wrong, and how does it show up in the ruler’s metrics?

Quiz