Reference monitoring — Prometheus, Grafana, Loki, and Tempo for the production estate

A reference monitoring stack is not a collection of tools. It is a working system that the operator can clone, deploy, and adapt to a real-world estate. The RunBook Academy reference monitoring stack is the canonical observability layer for a production VyOS 1.5 LTS estate: Prometheus for metrics, Grafana for visualisation, Loki for logs, Tempo for traces, and Alertmanager for alerting.

This lesson is the reference monitoring stack: the components, the configuration, the dashboards, and the production discipline that turns observability into evidence rather than a guess.

The stack components

flowchart LR
  subgraph EXPORTERS["Exporters on every router"]
    NE["node_exporter<br/>:9100"]
    VYOS["prometheus-vyos-exporter<br/>:9102"]
    SE["snmp_exporter<br/>:9116"]
  end
  subgraph CORE["Core stack"]
    PROM["Prometheus<br/>(scrape, record, alert)"]
    AM["Alertmanager<br/>(route, group, inhibit)"]
  end
  subgraph VIS["Visualisation"]
    GRAF["Grafana<br/>(dashboards, alerts)"]
  end
  subgraph LOG["Logs and traces"]
    LOKI["Loki<br/>(log aggregation)"]
    TEMPO["Tempo<br/>(trace storage)"]
  end
  subgraph CONSUME["Consumers"]
    ONCALL["On-call rotation"]
    POSTMORTEM["Post-incident review"]
  end
  NE --> PROM
  VYOS --> PROM
  SE --> PROM
  PROM --> AM
  PROM --> GRAF
  AM --> ONCALL
  PROM --> POSTMORTEM
  LOKI --> GRAF
  TEMPO --> GRAF

Prometheus is the metrics engine. It scrapes the exporters on every router on a configurable cadence (typically 15 seconds), stores the metrics in a time-series database, evaluates recording rules for derived metrics, and evaluates alerting rules for incident detection.

Grafana is the visualisation layer. It connects to Prometheus (for metrics), Loki (for logs), and Tempo (for traces), and provides the dashboards that the operator uses to inspect the estate. Grafana also supports alerting (the operator can define Grafana alerts in addition to the Prometheus alerts).

Loki is the log aggregation layer. It ingests logs from the FRR, keepalived, WireGuard, and system log streams, and provides the log-based queries that the operator uses to correlate the logs with the metrics.

Tempo is the trace storage layer. It ingests the distributed traces from the application layer, and provides the trace-based queries that the operator uses to follow a request through the estate.

Alertmanager is the alerting routing layer. It receives alerts from Prometheus (and from Grafana), groups them by labels, inhibits redundant alerts, and routes them to the on-call rotation (PagerDuty, OpsGenie, Slack, etc.).

Prometheus configuration

The Prometheus server’s configuration is the canonical pattern for scraping the exporters on every router. The configuration is stored in /etc/prometheus/prometheus.yml:

global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    cluster: prod-east
    region: us-east-1

scrape_configs:
  # node_exporter on every router
  - job_name: 'node'
    static_configs:
      - targets:
        - '10.0.1.1:9100'  # R1-A
        - '10.0.1.2:9100'  # R1-B
        - '10.0.2.1:9100'  # R2-A
        - '10.0.2.2:9100'  # R2-B

  # prometheus-vyos-exporter on every router
  - job_name: 'vyos'
    static_configs:
      - targets:
        - '10.0.1.1:9102'  # R1-A
        - '10.0.1.2:9102'  # R1-B
        - '10.0.2.1:9102'  # R2-A
        - '10.0.2.2:9102'  # R2-B

  # snmp_exporter on every router
  - job_name: 'snmp'
    static_configs:
      - targets:
        - '10.0.1.1:9116'  # R1-A
        - '10.0.1.2:9116'  # R1-B
        - '10.0.2.1:9116'  # R2-A
        - '10.0.2.2:9116'  # R2-B
    metrics_path: /snmp
    params:
      module: [if_mib]
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 10.0.1.50:9116  # snmp_exporter service

# Alerting rules
rule_files:
  - '/etc/prometheus/rules/*.yml'

# Alertmanager
alerting:
  alertmanagers:
    - static_configs:
        - targets: ['10.0.0.50:9093']

The Prometheus server scrapes the three exporters on every router. The external_labels are used by the Alertmanager to route alerts based on the cluster and region.

Recording rules and alerting rules

The Prometheus server evaluates recording rules and alerting rules. The recording rules compute derived metrics; the alerting rules detect incidents.

# /etc/prometheus/rules/network.yml
groups:
  - name: network
    interval: 30s
    rules:
      # 5-minute average of interface receive bytes per second
      - record: instance:node_network_receive_bytes:rate5m
        expr: rate(node_network_receive_bytes_total[5m])

      # 5-minute average of interface transmit bytes per second
      - record: instance:node_network_transmit_bytes:rate5m
        expr: rate(node_network_transmit_bytes_total[5m])

      # BGP session state
      - record: instance:vyos_bgp_session_up:max
        expr: max by (instance, peer) (vyos_bgp_session_up)

      # Alert: BGP session down
      - alert: BGPDown
        expr: vyos_bgp_session_up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: 'BGP session {{ $labels.peer }} on {{ $labels.instance }} is down'
          description: 'BGP session {{ $labels.peer }} on {{ $labels.instance }} has been down for more than 1 minute.'

      # Alert: High CPU
      - alert: HighCPU
        expr: 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: 'High CPU on {{ $labels.instance }}'
          description: 'CPU usage on {{ $labels.instance }} is above 80% for 5 minutes.'

      # Alert: Disk almost full
      - alert: DiskAlmostFull
        expr: (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100 < 10
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: 'Disk almost full on {{ $labels.instance }}'
          description: 'Disk / on {{ $labels.instance }} is above 90% usage.'

The recording rules compute the 5-minute rate of the interface bytes and the BGP session state. The alerting rules detect the BGP session down, the high CPU, and the disk almost full conditions.

Grafana dashboards

The Grafana dashboards provide the visualisation layer for the reference topology. The canonical dashboards:

# /etc/grafana/provisioning/dashboards/vyos-overview.yaml
apiVersion: 1
providers:
  - name: 'VyOS Overview'
    folder: 'VyOS'
    type: file
    options:
      path: /var/lib/grafana/dashboards/vyos

The canonical dashboards:

• VyOS — Overview. Per-router CPU, memory, disk, and uptime. The operator uses this dashboard to see the health of every router at a glance.
• VyOS — Interface Metrics. Per-interface traffic (in/out), errors, discards. The operator uses this dashboard to identify the interfaces that are saturated or dropping packets.
• VyOS — BGP. Per-peer session state, prefix count, update rate. The operator uses this dashboard to identify the BGP peers that are flapping or losing routes.
• VyOS — OSPF. Per-area LSDB count, SPF run rate, neighbour state. The operator uses this dashboard to identify the OSPF areas that are unstable.
• VyOS — VRRP. Per-group master/backup state, transition count, gratuitous ARP delay. The operator uses this dashboard to identify the VRRP groups that are flapping.

The dashboards are provisioned from YAML files; the operator can version-control the dashboards in Git and roll them out via Ansible.

Loki log aggregation

The Loki log aggregation ingests the FRR, keepalived, WireGuard, and system log streams. The Promtail agent runs on every router and ships the logs to Loki.

# /etc/promtail/config.yml
server:
  http_listen_port: 9080

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://10.0.0.50:3100/loki/api/v1/push

scrape_configs:
  # FRR log
  - job_name: frr
    static_configs:
      - targets: [localhost]
        labels:
          job: frr
          __path__: /var/log/frr/*.log

  # keepalived log
  - job_name: keepalived
    static_configs:
      - targets: [localhost]
        labels:
          job: keepalived
          __path__: /var/log/keepalived.log

  # WireGuard log
  - job_name: wireguard
    static_configs:
      - targets: [localhost]
        labels:
          job: wireguard
          __path__: /var/log/syslog

  # System log
  - job_name: syslog
    static_configs:
      - targets: [localhost]
        labels:
          job: syslog
          __path__: /var/log/syslog

The Promtail agent ships the logs to Loki, and the operator uses Grafana to query the logs ({job="frr"} |~ "BGP session"). The discipline: use the job label to distinguish the log sources; use the operator’s incident ID as a label to correlate the logs with the incident.

Alertmanager routing

The Alertmanager routes alerts based on the labels. The canonical routing:

# /etc/alertmanager/alertmanager.yml
route:
  receiver: 'default'
  group_by: ['alertname', 'instance']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    # Critical alerts go to PagerDuty
    - match:
        severity: critical
      receiver: 'pagerduty'
    # Warning alerts go to Slack
    - match:
        severity: warning
      receiver: 'slack'
    # BGP alerts go to the network on-call
    - match:
        alertname: BGPDown
      receiver: 'network-oncall'

receivers:
  - name: 'default'
    webhook_configs:
      - url: 'http://10.0.0.50:5001/alerts'
  - name: 'pagerduty'
    pagerduty_configs:
      - service_key: '<pagerduty-service-key>'
  - name: 'slack'
    slack_configs:
      - api_url: '<slack-webhook-url>'
        channel: '#network-alerts'
  - name: 'network-oncall'
    pagerduty_configs:
      - service_key: '<network-oncall-service-key>'

inhibit_rules:
  # Inhibit the per-peer BGP alert if the per-router alert is firing
  - source_match:
      alertname: RouterDown
    target_match:
      alertname: BGPDown
    equal: ['instance']

The Alertmanager routes critical alerts to PagerDuty, warning alerts to Slack, and BGP alerts to the network on-call. The inhibit_rules prevent the per-peer BGP alert from firing if the per-router alert is already firing (a router down implies BGP down).

Production failure modes

The monitoring stack failure modes the operator encounters:

• Prometheus disk full. The time-series database grows; the disk is full; Prometheus stops recording. Fix: configure the retention policy (--storage.tsdb.retention.time=30d); monitor the disk usage; alert on Prometheus disk usage > 80%.
• Loki disk full. The log storage grows; the disk is full; Loki stops ingesting. Fix: configure the retention policy; compress the logs; alert on Loki disk usage > 80%.
• Prometheus OOM. The Prometheus process consumes too much memory; the OOM killer kills Prometheus. Fix: increase the memory limit (--storage.tsdb.max-block-chunk-segment-size); reduce the cardinality of the metrics.
• Alertmanager misrouted alert. The alert is routed to the wrong on-call rotation; the operator is not notified. Fix: test the Alertmanager routing; review the routes; alert on the Alertmanager’s own metrics.
• Grafana dashboard stale. The dashboard is showing stale data because the underlying data source is stale. Fix: configure the data source refresh interval; alert on the data source’s last successful scrape.
• Promtail down. The log shipping agent is down; the logs are not ingested. Fix: monitor the Promtail process; alert on Promtail down.

Rollback

The rollback for a monitoring stack change is the rollback of the configuration. The discipline:

• Prometheus configuration — the operator must version-control the configuration in Git; the rollback is git revert and prometheus reload.
• Grafana dashboards — the operator must version-control the dashboards in Git; the rollback is git revert and grafana reload.
• Loki configuration — the operator must version-control the configuration in Git; the rollback is git revert and loki reload.
• Alertmanager configuration — the operator must version-control the configuration in Git; the rollback is git revert and alertmanager reload.

Production discipline

Cross-course references

• Part LVII-01 (vyos-lvii-01-reference-topology) covers the topology that this lesson monitors.
• Part XLIX (vyos-xlix-01-interface-metrics) covers the interface metrics that this lesson aggregates.
• Part XLIX-05 (vyos-xlix-05-system-metrics) covers the system metrics that this lesson aggregates.
• The Observability course covers the consumer side: Prometheus, Grafana, Loki, Tempo.
• The Ansible course’s XLII-Ansible-BeyondLinux covers the automation hand-off (rolling out the stack to a fleet via a single playbook).

Quiz