Log-based alerting — the queries and the alerts

Log-based alerting is the discipline of alerting on the logs. The LogQL queries, the Loki ruler, the Alertmanager integration are the components. This lesson walks the queries, the ruler, the Alertmanager integration, and the production patterns.

The log-based alerting flow

The log-based alerting flow:

flowchart LR
    A[Logs] --> B[Loki]
    B --> C[Loki ruler]
    C --> D[Alert]
    D --> E[Alertmanager]
    E --> F[Slack]
    E --> G[PagerDuty]

The flow is the alert path.

The LogQL queries

The LogQL queries:

# Count errors per second
sum(rate({service="nginx"} |= "error" [5m]))

# Count exceptions
sum(rate({service="nginx"} |~ "Exception|Error" [5m]))

# Count specific errors
sum(rate({service="nginx"} |= "connection refused" [5m]))

# Count per namespace
sum(rate({namespace="team-a"} |= "error" [5m])) by (namespace)

The queries are the input for the alerts.

The Loki ruler

The Loki ruler:

apiVersion: monitoring.coreos.com/v1
kind: LokiRuler
metadata:
  name: my-loki-ruler
spec:
  rules:
    groups:
    - name: log-alerts
      rules:
      - alert: HighErrorRate
        expr: |
          sum(rate({service="nginx"} |= "error" [5m])) > 1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High error rate from nginx logs"
          runbook_url: "https://runbook.example.com/error-rate"

The Loki ruler evaluates the rules against the logs.

The Loki ruler config

The Loki ruler config:

# Loki config
ruler:
  storage:
    type: local
    local:
      directory: /etc/loki/rules
  rule_path: /tmp/loki/rules-temp
  alertmanager_url: http://alertmanager:9093
  external_url: http://loki:3100

The config tells the Loki where to load the rules and where to send the alerts.

The Alertmanager integration

The Alertmanager integration:

sequenceDiagram
    participant L as Loki ruler
    participant A as Alertmanager
    participant S as Slack
    L->>A: alert
    A->>A: route by severity
    A->>S: notification

The Alertmanager receives the alerts from the Loki ruler.

The alert routing

The alert routing:

# Alertmanager config
route:
  receiver: default
  routes:
  - matchers:
    - severity = warning
    receiver: slack-warnings
  - matchers:
    - severity = critical
    receiver: pagerduty-critical

receivers:
- name: slack-warnings
  slack_configs:
  - apiURL: https://hooks.slack.com/services/...
    channel: "#alerts-warnings"

The routing is the Alertmanager’s responsibility.

The alert silencing

The alert silencing:

amtool silence add --alertmanager http://alertmanager:9093 \
  --comment "Maintenance window" \
  --duration 1h \
  --start "2026-08-16T10:00:00Z" \
  --match "alertname=HighErrorRate"

The silencing is the temporary suppression.

The log-based recording rules

The log-based recording rules:

- record: nginx:error_rate:rate5m
  expr: sum(rate({service="nginx"} |= "error" [5m]))

- record: nginx:exceptions:rate5m
  expr: sum(rate({service="nginx"} |~ "Exception|Error" [5m]))

The recording rules are the input for the dashboards.

The alert testing

The alert testing:

# Verify the alert is configured
logcli query --addr http://loki:3100 '{service="nginx"} |= "error"' --since=1h

# Test the alert rule
logcli rules --addr http://loki:3100

The testing is direct.

The production patterns

The production patterns:

flowchart LR
    A[Log pattern] --> B[LogQL query]
    B --> C[Alert rule]
    C --> D[Loki ruler]
    D --> E[Alertmanager]
    E --> F[Notification]

The pattern is the production flow.

The cross-course references

• The Loki course covers the LogQL queries.
• The Alertmanager course covers the routing.
• The Grafana course covers the dashboards.

Quiz

Production discipline

• Define the LogQL queries. The error patterns.
• Configure the Loki ruler. The alert rules.
• Integrate with the Alertmanager. The routing.
• Configure the receivers. Slack, PagerDuty.
• Test the alerts. Verify the alerts are firing.
• Document the alerting. The rules, the routes, the receivers.

The log-based alerting is the production pattern. Operating it well is the LogQL queries, the Loki ruler, and the Alertmanager integration.