Recording rules — pre-computing the metrics

Prometheus recording rules pre-compute the metrics. The PrometheusRule CRD is the Operator-style. The eval_interval is the evaluation interval. This lesson walks the recording rules, the CRD, the evaluation interval, and the production patterns.

What recording rules do

The recording rules pre-compute the metrics:

flowchart LR
    A[Raw metrics] --> B[Recording rule]
    B --> C[Pre-computed metrics]
    C --> D[Dashboards]
    C --> E[Alerts]

The pre-computed metrics are faster to query.

The PrometheusRule CRD

The PrometheusRule CRD:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: my-rules
  labels:
    release: my-prometheus
spec:
  groups:
  - name: my-rules
    interval: 30s
    rules:
    - record: my_app:http_requests:rate5m
      expr: sum(rate(http_requests_total[5m])) by (status)

The PrometheusRule is the Operator-style.

The rule file

The rule file is the alternative:

# /etc/prometheus/rules/my-rules.yaml
groups:
- name: my-rules
  interval: 30s
  rules:
  - record: my_app:http_requests:rate5m
    expr: sum(rate(http_requests_total[5m])) by (status)

The rule file is mounted into the Prometheus.

The PrometheusRule structure

The PrometheusRule structure:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: my-rules
spec:
  groups:
  - name: <group-name>
    interval: <eval-interval>
    rules:
    - record: <metric-name>
      expr: <PromQL>
      labels:
        <label>: <value>
    - alert: <alert-name>
      expr: <PromQL>
      for: <duration>
      labels:
        severity: <level>
      annotations:
        summary: <summary>

The structure is the Prometheus recording rules.

The evaluation interval

The evaluation interval:

spec:
  groups:
  - name: my-rules
    interval: 30s

The default is the global evaluation interval. The per-group interval overrides the default.

The recording pattern

The recording pattern:

# Pre-compute the rate
record: my_app:http_requests:rate5m
expr: sum(rate(http_requests_total[5m])) by (status)

# Use the pre-computed metric
record: my_app:http_requests:avg_rate5m
expr: avg(my_app:http_requests:rate5m)

The pattern is the layered metrics.

The cardinality

The cardinality:

# Low cardinality (good)
record: my_app:http_requests:rate5m
expr: sum(rate(http_requests_total[5m])) by (status)
# Output: 2 series (2xx, 5xx)

# High cardinality (bad)
record: my_app:http_requests:rate5m
expr: sum(rate(http_requests_total[5m])) by (status, pod)
# Output: N * 2 series (N pods * 2 statuses)

The cardinality is the storage cost.

The query

The query:

my_app:http_requests:rate5m

The query is the input for the dashboards and alerts.

The SLO recording rules

The SLO recording rules:

# SLI: success rate
sum(rate(http_requests_total{status=~"2.."}[5m]))
/
sum(rate(http_requests_total[5m]))

# Error budget
1 - {
  sum(rate(http_requests_total{status=~"2.."}[30d]))
  /
  sum(rate(http_requests_total[30d]))
}

# Burn rate
1 - {
  sum(rate(http_requests_total{status=~"2.."}[1h]))
  /
  sum(rate(http_requests_total[1h]))
}

The SLO recording rules are the input for the SLO queries.

The recording rule evaluation

The recording rule evaluation:

flowchart LR
    A[30s timer] --> B[Recording rule evaluator]
    B --> C[PromQL expression]
    C --> D[Result]
    D --> E[Store as new metric]

The evaluation is periodic.

The production patterns

The production patterns:

flowchart LR
    A[Raw metrics] --> B[Layer 1: rate]
    B --> C[Layer 2: aggregation]
    C --> D[Layer 3: SLO]
    D --> E[Dashboards]
    D --> F[Alerts]

The layered metrics are the production pattern.

The PrometheusRule validation

The validation:

promtool check rules /etc/prometheus/rules/my-rules.yaml

The validation checks the syntax.

Cross-course references

• The Prometheus Operator course covers the CRDs.
• The Grafana course covers the dashboards.
• The SLO/SLI course covers the burn rate.

Quiz

Production discipline

• Use the PrometheusRule CRD. The Operator style.
• Layer the recording rules. Layer 1: rate; Layer 2: aggregation; Layer 3: SLO.
• Control the cardinality. Use the by clause.
• Validate the rules. Use promtool check rules.
• Document the recording rules. The metric names, the expressions.
• Test the recording rules. Verify the metrics are computed.

The recording rules are the cluster’s pre-computed metrics. Operating it well is via the PrometheusRule CRD, with the layered recording pattern, and the cardinality control.