Alerting in Grafana

A paged alert at 02:00 reads “CPU above 80%”. The on-call engineer opens the rule and the message is the message Grafana sent; the notification policy is what decided how the message was routed; the contact point is the integration that delivered it. Knowing which of those three to edit is the difference between a five-minute fix and a five-hour raid.

This lesson is about the three pieces: rules (the what), notification policies (the who and when), and contact points (the where). It closes with the boundary with Prometheus Alertmanager — the place where most Grafana alert teams go wrong.

What alerting in Grafana is

Grafana Alerting is the in-Grafana alerting subsystem, separate from Prometheus’s /alertmanager and from any externally run Alertmanager. It is “unified” in that the same alert rule can be expressed as a Prometheus query, a Loki query, or a Tempo query behind the same rule model. Three objects make up the subsystem:

+--- Rule group (file on disk or in storage) ------------------+
|                                                              |
|   +--- Rule ------- +--- Rule ------- +--- Rule -----+      |
|   |  expr           |  expr           |  expr          |      |
|   |  for: 5m        |  for: 10m       |  for: 2m       |      |
|   |  noDataState:   |  execErrState:  |  labels:       |      |
|   |   NoData        |   Alerting      |    team: sre   |      |
|   +-----------------+-----------------+-----------------+     |
+--------------------------------------------------------------+
                              |
                              v
+--- Notification policy (storage) ----------------------------+
|                                                              |
|   matchers: team=sre             ->   contact point "sre-pager"|
|   matchers: severity=high        ->   contact point "war-room"  |
|   fallback:                     ->   contact point "default"   |
|                                                              |
+--------------------------------------------------------------+
                              |
                              v
+--- Contact point (integration) ------------------------------+
|                                                              |
|   types: webhook | pagerduty | slack | email | teams | opsgenie|
|   settings:    URL, channel, API key stored in secureJsonData |
|                                                              |
+--------------------------------------------------------------+

A rule evaluates a query on its interval, classifies the result as one of Normal, Pending, Alerting, No Data, or Error, and (for the Alerting and Error cases) emits a notification. The notification policy decides which contact point receives the notification, and silences or inhibition modify the flow.

Why a sysadmin cares

Alerting failure shapes map directly to money and reputation:

1. Alert rule is firing but no one is paged. Caused by a missing or mis-routed contact point, or a notification policy that suppresses everything.
2. Alert is paging but the on-call does not know what to do. The notification template is silent on what the rule means. The fix is template work, not contact-point work.
3. Alert fires only after the incident is over. A for: that is too long for the SLO and a noDataState that is set to OK instead of NoData.
4. Alert flaps every minute. A near-threshold expression and a short evaluation interval cause perpetual state-changing notifications. The fix is for: plus a keep_firing_for.

These are ticket templates. The lesson is the diagnostic path between the symptom and the field.

How it works

The evaluation engine is a Go service inside the Grafana process. The path from a query to a paged human looks like this:

  every interval                 every interval
+--------------+           +-------------------+
|  rule group  |-----------> |  eval engine      |
|  (yaml or    |  turn-on  |                   |
|  storage)    |           |  query datasource |
+--------------+           |  -> data source    |
                            |  -> classification |
                            +-------------------+
                                      |
                          classifier (Normal/Pending/Alerting/NoData/Error)
                                      |
                                      v
                            +-------------------+
                            |  state store       |
                            |  + active rules    |
                            |  + silences        |
                            |  + history         |
                            +-------------------+
                                      |
                                      v
                            +-------------------+
                            |  notification      |
                            |  policy + matcher  |
                            +-------------------+
                                      |
                                      v
                            +-------------------+
                            |  contact points     |
                            |  (pagerduty, slack, |
                            |  teams, opsgenie)   |
                            +-------------------+

Three numbers characterise every rule:

• interval — how often the rule evaluates the query. The default of 1m is fine for SLI / SLO rules; some high-frequency rules drop to 30s. Once below 10s, the rule is system-induced.
• for — how long the state must remain Alerting before a notification is emitted. A for: 5m against a transient spike does not page; a for: 0s would.
• keep_firing_for — for how long the rule continues to notify after the state goes back to Normal. Use this to guarantee a single notification per incident, not per state-change.

How to configure it

Three provisioning files describe the complete alerting landscape: groups of rules, the notification policy, and the contact points. This is the file layout Grafana expects under /etc/grafana/provisioning/alerting/.

# /etc/grafana/provisioning/alerting/contact_points.yaml
apiVersion: 1
contactPoints:
  - orgId:       1
    name:        sre-pager
    receivers:
      - uid:            pd-sre
        type:           pagerduty
        settings:
          # The integration key lives in secureJsonData so the
          # plaintext form does not appear in provisioning diffs.
          service:        SRE primary
          severity:       critical
        secureSettings:
          integrationKey: ${PAGERDUTY_INTEGRATION_KEY_SRE}

  - orgId:       1
    name:        default
    receivers:
      - uid:    slack-default
        type:   slack
        settings:
          url:    https://hooks.slack.com/services/${SLACK_TID_SRE}/${SLACK_TOKEN_SRE}
          channel: #sre-alerts

# /etc/grafana/provisioning/alerting/routes.yaml
apiVersion: 1
routes:
  - receiver:        sre-pager
    group_by:        [grafana_folder, alertname]
    group_wait:      30s
    group_interval:  5m
    repeat_interval: 4h
    object_matchers:
      - [team,   "=", sre]
      - [severity, "=", critical]
  - receiver:        default
    object_matchers:
      - [severity, "=", info]

# /etc/grafana/provisioning/alerting/rules.yaml
apiVersion: 1
groups:
  - orgId:       1
    name:        sre-error-rate
    folder:      SRE
    interval:    1m
    rules:
      - uid:             high-error-budget-burn
        title:           High error budget burn on checkout
        condition:       C
        data:
          - refId:        A
            relativeTimeRange:
              from:       600
              to:         0
            datasourceUid:  prom-prod
            model:
              refId:        A
              expr:         |
                sum(rate(checkout_requests_total{status=~"5.."}[5m]))
                /
                sum(rate(checkout_requests_total[5m]))
                >
                0.01
              instant:      true
              intervalMs:   1000
              maxDataPoints: 43200
          - refId:        B
            datasourceUid: __expr__
            model:
              refId:      B
              type:       threshold
              conditions:
                - evaluator: { type: gt, params: [0.001] }
                  operator:  { type: and }
                  query:     { params: [A] }
                  reducer:   { type: last }
        for:             2m
        keep_firing_for: 5m
        noDataState:    OK
        execErrState:   Alerting
        annotations:
          summary:       "checkout error rate above 1% for 2m"
          runbook_url:   https://runbooks.example/checkout-error-rate
          description:   "Error rate = {{ $value | humanizePercentage }}"
        labels:
          team:          sre
          severity:      critical

Three things to read in this YAML:

• The model.expr is the expression. instant: true produces a single value; an interval query produces a series. The threshold type is hidden inside the second refId (a server-side expression).
• noDataState: OK is the dangerous default for SLO rules; setting it to NoData makes a missing data source emit a notification (this is what you want for critical rules).
• runbook_url is the field the notification templates typically surface; populate it consistently.

How to validate it

Five checks confirm the alerting subsystem end-to-end.

Severity: READ-ONLY unless noted.

# 1. The alertmanager's view of itself.
curl -sf -u "grafana-admin:$GF_ADMIN_PASSWORD" \
  http://grafana:3000/api/v1/provisioning/alertmanager/config | jq

{
  "alertmanager_config": {
    "org_id":   1,
    "alertmanager_name": "grafana",
    "default":  "default"
  }
}

# 2. The contact-point list.
curl -sf -u "grafana-admin:$GF_ADMIN_PASSWORD" \
  http://grafana:3000/api/v1/provisioning/contact-points | jq

[
  { "uid":  "pd-sre",         "name": "sre-pager",   "type": "pagerduty" },
  { "uid":  "slack-default",  "name": "default",     "type": "slack" }
]

# 3. The notification policy.
curl -sf -u "grafana-admin:$GF_ADMIN_PASSWORD" \
  http://grafana:3000/api/v1/provisioning/folder/1/rule-groups | jq

# 4. The rule group (and per-rule UID).
curl -sf -u "grafana-admin:$GF_ADMIN_PASSWORD" \
  http://grafana:3000/api/v1/provisioning/folder/1/rule-groups/sre-error-rate \
  | jq '.rules[] | {uid, title, for}'

{
  "uid":   "high-error-budget-burn",
  "title": "High error budget burn on checkout",
  "for":   "2m"
}

# 5. The state of a known rule.
# RULE_UID is the uid reported by the rule group listing in step 4.
RULE_UID=high-error-budget-burn

curl -sf -u "grafana-admin:$GF_ADMIN_PASSWORD" \
  "http://grafana:3000/api/v1/provisioning/alert-rules/$RULE_UID/state"

{ "state": "inactive", "lastEvaluated": "2026-08-13T16:00:00Z" }

A rule with state: active and no notification has only one explanation: the notification policy matches, but the contact point’s secret is empty or the integration endpoint is unreachable. Inspect the [Alerting] block of the Grafana log for the relevant error.

How it can fail

Six failure modes that recur in real fleets:

1. noDataState mis-set. A rule with noDataState: OK flattens to “no notification” when the data source stops responding. The cluster appears to recover; the alert does not fire. Set noDataState: NoData for SLO rules.
2. Contact-point secret missing. A PagerDuty contact point with an empty integrationKey silently fails to deliver. The state is normal; the notification policy matched; the delivery failed. Inspect the log line Notifier notifier=pagerduty returned an error.
3. Notification policy does not match. The rule has the correct labels; the route’s object_matchers does not include this rule; the fallback fires only if a fallback route exists. Without one, the notification is dropped.
4. Grafana and Prometheus alerting both firing. A duplicated rule produces two notifications for the same condition. The on-call is paged twice. Disable one of the two.
5. evaluation_interval too long for the SLO. A 5-minute interval against a 5-minute error-budget burn window detects the burn too late. Lower the interval or move to a Prometheus recording rule.
6. keep_firing_for produces a notification gap. A 5-minute keep_firing_for after the alert returns to Normal. If the next spike starts within the window, no second notification is emitted. Tune the value or remove it.

How to troubleshoot it

The order works because each step eliminates a layer.

1. Does the rule evaluate? The rule state in /api/v1/provisioning/alert-rules/<uid>/state shows the last evaluation result. lastEvaluated: never means the rule evaluator was not scheduled; check the Grafana log.
2. Is the query firing? Use the same query in Explore and watch for the condition value. A rule over a label the data source does not emit always evaluates to “empty” with noDataState.
3. Does the notification policy match? Read the rules in /api/v1/provisioning/folder/1/rule-groups/<group> and check the rule’s labels against the route’s object_matchers. A label-as-string mismatch (team=sre vs team="sre") is a common fault.
4. Did the contact point deliver? Check /api/v1/provisioning/contact-points and the [Alerting] block of the Grafana log for Notifier notifier=<type> returned an error.
5. Is the integration endpoint healthy? From the Grafana host, curl -i the integration URL with the secureSettings substituted. PagerDuty’s Events API /v2/enqueue returns a 202 Accepted on success; Slack’s webhook returns a 200 OK with empty body.
6. Is the rate-limiter silencing? Add repeat_interval 30s instead of 4h in the route, watch the next notification, restore the larger value once delivered.

Security implications

• Notification channels leak data. A PagerDuty alert with a full notification payload including customer IDs is a regulated-data disclosure via a non-regulated channel. Template the notification to strip or hash identifiers.
• Contact-point secrets are stored in secureJsonData and encrypted with secret_key. They are reversible by the server, not by a vault. Treat them as host-resident secrets. Use the integration secrets PagerDuty / Slack / OpsGenie provision per environment; do not re-use the production PagerDuty key in staging.
• Webhook endpoints. Each webhook contact point is an outbound request from Grafana to a URL. The URL is part of the configuration. It is not an authenticated server and any compromise of the contact-point URL or its secret is a path to a forged notification. Validate URL ownership in the integration setup.
• OTLP / external Alertmanager. If Grafana’s alertmanager URL is pointed at an external Alertmanager for legacy setups, the URL is in [unified_alerting] and the connection is auth-stripped. Network-segment Grafana from Alertmanager with the same isolation as a database.

Performance implications

• A rule group with a slow query holds the evaluator thread. Move expr computations to recording rules for a hot path.
• A high-frequency interval (sub-30s) wastes both CPU and data-source bandwidth. The rule engine has no built-in deduplication across rules; if two rules share an expression, define a recording rule and reference it.
• The state store is bounded by the number of currently-firing alerts, not the total number of rules. A Grafana with 10,000 rules and 50 firing alerts uses the same memory as one with 50 rules and 50 firing alerts.

Production guidance

• Adopt provisioning for every alert definition that is part of an SLO. UI-created rules are version-control-hostile; their drift from the team’s intent is hard to spot.
• Set noDataState on every rule. The default (OK) is almost never right for SLO rules; the discipline is to be explicit.
• Maintain a notification policy that is small, named, and reviewed. A sprawling matchers tree is the source of the “notification went nowhere” tickets.
• Pre-wire [Alerting].secondary_resource to an S3 bucket or similar for the alertmanager state file. A state-store corruption on the primary node is otherwise a rule-history loss.
• Define keep_firing_for per SLO family. The default of 0 is fine for paging alerts; longer windows suit SLO notifications where the operator is expected to investigate during the burn.

Verification

You should now be able to answer:

• What are the three objects the unified alerting model owns?
• What does noDataState: OK do for a missing-data signal?
• What is the difference between a notification policy and a contact point?
• When is Grafana-managed alerting the wrong choice and Prometheus the right one?
• Which API endpoint lists the contact points and which lists the rule groups?

Quiz