Grafana and alerting - dashboards and notification

Grafana turns Prometheus metrics into dashboards. Alertmanager routes alerts to the right channel. This lesson covers setting up both.

Install Grafana

# Debian/Ubuntu
sudo apt install grafana

# Or download from grafana.com
sudo systemctl enable --now grafana-server

Grafana listens on port 3000 by default. Open http://localhost:3000 and log in (default admin/admin, change on first login).

Add Prometheus as a data source

1. Go to Configuration > Data sources > Add data source.
2. Select Prometheus.
3. Set URL: http://localhost:9090.
4. Save and test.

Build a host dashboard

A useful host dashboard has:

• CPU usage per core.
• Memory available.
• Disk usage per mount.
• Network throughput.
• Load average.
• System uptime.

Create panels with PromQL queries:

# CPU usage
100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Memory available (GB)
node_memory_MemAvailable_bytes / 1024^3

# Disk usage %
(1 - node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100

# Network receive (Mbps)
rate(node_network_receive_bytes_total[5m]) * 8 / 1024^2

Alertmanager

Prometheus sends alerts to Alertmanager. Alertmanager deduplicates, groups, and routes them.

# /etc/alertmanager/alertmanager.yml
global:
  resolve_timeout: 5m
  smtp_smarthost: 'smtp.example.com:587'
  smtp_from: 'alerts@example.com'
  smtp_auth_username: 'alerts@example.com'
  # Never inline the password: this file belongs in version control.
  # The referenced file is mode 0600, owned by the alertmanager user,
  # and delivered by the secret store rather than by the repository.
  smtp_auth_password_file: /etc/alertmanager/secrets/smtp_password
  smtp_require_tls: true

route:
  receiver: 'team-platform'
  group_by: ['alertname', 'instance']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    # `matchers:` - the `match:`/`match_re:` forms have been deprecated
    # since Alertmanager 0.22 and log a warning on load.
    - matchers:
        - severity = "critical"
      receiver: 'pagerduty'

# Do not page for the symptom when you have already paged for the cause.
inhibit_rules:
  - source_matchers:
      - severity = "critical"
    target_matchers:
      - severity = "warning"
    equal: ['alertname', 'instance']

receivers:
  - name: 'team-platform'
    email_configs:
      - to: 'platform@example.com'

  - name: 'pagerduty'
    pagerduty_configs:
      # `routing_key` is the Events API v2 field. `service_key` is v1,
      # which PagerDuty has retired - a v1 key on a current integration
      # fails at delivery time, when the page is the thing being lost.
      - routing_key_file: /etc/alertmanager/secrets/pagerduty_routing_key

Alertmanager sends:

• Critical alerts to PagerDuty (on-call wakes up).
• Non-critical to email.
• Groups by alertname and instance to avoid noise.

Run Grafana alerting

Grafana can also alert (independent of Prometheus):

1. Open a panel.
2. Alert tab.
3. Set condition: avg(cpu_usage) > 80 for 5m.
4. Configure notification: contact point (Slack, PagerDuty,
   email).
5. Save.

Grafana alerting is good for dashboard-driven alerting. Prometheus + Alertmanager is better for fleet-wide alerting.

Best practices

• One dashboard per host type: web, db, etc.
• Variables for fleet-wide dashboards: $instance, $mountpoint.
• Reuse panels: shared libraries for common queries.
• Alert on user impact: latency, errors, not CPU.
• Document alerts: each alert has a runbook.

Knowledge check