Prometheus and node_exporter - the standard Linux monitoring stack

Prometheus and node_exporter are the de facto standard for Linux monitoring. This lesson covers how they work and how to set up a basic stack.

Prometheus architecture

  +---------+     +---------+     +---------+
  | Host A  |     | Host B  |     | Host C  |
  | (node_  |     | (node_  |     | (node_  |
  | exporter|     | exporter|     | exporter|
  +---------+     +---------+     +---------+
       |              |              |
       +-------+------+------+-------+
               |
        +-------------+
        |  Prometheus  |
        | (scrape     |
        |  storage)   |
        +-------------+
               |
        +-------------+
        |   Grafana    |
        | (dashboard) |
        +-------------+

• node_exporter: runs on each host, exposes metrics over HTTP.
• Prometheus: scrapes each node_exporter, stores the data in a time-series database.
• Grafana (or similar): queries Prometheus, displays dashboards.

Install node_exporter

# Debian/Ubuntu
sudo apt install prometheus-node-exporter

# Or download from GitHub releases
wget https://github.com/prometheus/node_exporter/releases/download/v*/node_exporter-*.linux-amd64.tar.gz
tar xzf node_exporter-*.linux-amd64.tar.gz
sudo cp node_exporter-*/node_exporter /usr/local/bin/

node_exporter listens on port 9100 by default. Verify:

curl http://localhost:9100/metrics

Metric format

Prometheus metrics are lines like:

node_cpu_seconds_total{cpu="0",mode="user"} 1234.56
node_memory_MemTotal_bytes 67108864000
node_filesystem_size_bytes{device="/dev/sda1",mountpoint="/"} 100000000000

Format: metric_name{label1="value1",label2="value2"} value timestamp.

Metric types:

• Counter: monotonically increasing. node_cpu_seconds_total.
• Gauge: arbitrary value. node_memory_MemAvailable_bytes.
• Histogram: bucketed observations. For latency.
• Summary: percentiles.

Configure Prometheus to scrape

# /etc/prometheus/prometheus.yml
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'node'
    static_configs:
      - targets:
        - 'host-a:9100'
        - 'host-b:9100'
        - 'host-c:9100'

Prometheus scrapes each target every 15s.

PromQL queries

# Host CPU busy %, per instance.
# node_cpu_seconds_total is per-CPU-per-mode, so it MUST be aggregated:
# without `avg by (instance)` you get one series per core, and a host with
# one saturated core out of 32 looks 100% busy on that series.
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Memory used percentage
(1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100

# Disk used percentage - exclude pseudo-filesystems, never allow-list one fstype
100 * (1 - node_filesystem_avail_bytes{fstype!~"tmpfs|devtmpfs|overlay|squashfs|autofs"}
          / node_filesystem_size_bytes{fstype!~"tmpfs|devtmpfs|overlay|squashfs|autofs"})

# Inode used percentage - a separate exhaustion mode with its own metric
100 * (1 - node_filesystem_files_free{fstype!~"tmpfs|devtmpfs|overlay"}
          / node_filesystem_files{fstype!~"tmpfs|devtmpfs|overlay"})

# Network utilisation
rate(node_network_receive_bytes_total[5m]) * 8

Alerts

# /etc/prometheus/rules/host.yml
groups:
  - name: host
    rules:
      - alert: HostDown
        expr: up == 0
        for: 2m
        annotations:
          summary: 'Host {{ $labels.instance }} is down'

      - alert: FilesystemSpaceLow
        expr: |
          100 * (1 - node_filesystem_avail_bytes{fstype!~"tmpfs|devtmpfs|overlay|squashfs|autofs"}
                    / node_filesystem_size_bytes{fstype!~"tmpfs|devtmpfs|overlay|squashfs|autofs"}) > 85
        for: 15m
        annotations:
          summary: 'Filesystem {{ $labels.instance }} {{ $labels.mountpoint }} > 85% full'

      - alert: FilesystemInodesLow
        expr: |
          100 * (1 - node_filesystem_files_free{fstype!~"tmpfs|devtmpfs|overlay"}
                    / node_filesystem_files{fstype!~"tmpfs|devtmpfs|overlay"}) > 85
        for: 15m
        annotations:
          summary: 'Inodes {{ $labels.instance }} {{ $labels.mountpoint }} > 85% used'

      - alert: FilesystemWillFillIn4h
        expr: predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|devtmpfs|overlay"}[6h], 4*3600) < 0
        for: 30m
        annotations:
          summary: '{{ $labels.instance }} {{ $labels.mountpoint }} fills within 4h at current rate'

      - alert: FilesystemReadOnly
        expr: node_filesystem_readonly{fstype!~"tmpfs|devtmpfs|overlay"} == 1
        for: 1m
        annotations:
          summary: '{{ $labels.instance }} {{ $labels.mountpoint }} remounted read-only'

      - alert: HighCPU
        expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 10m
        annotations:
          summary: 'CPU on {{ $labels.instance }} > 80%'

Alerts are evaluated periodically. When conditions are met for the for duration, the alert fires.

The four filesystem rules cover four different failures, and none of them substitutes for another:

• Space is the one everybody has.
• Inodes exhaust independently. A filesystem with free blocks and zero free inodes reports normal disk usage, refuses every new file, and pages nobody.
• predict_linear turns a threshold into lead time. A volume at 60% that is filling fast is a more urgent page than one parked at 86%.
• node_filesystem_readonly catches the remount the kernel does after an I/O error. Usage stops changing, so the space alert never fires, and the host quietly stops accepting writes.

Knowledge check