Lab: Set up Prometheus and node_exporter on a host

This lab sets up Prometheus, node_exporter, and Grafana on a host. By the end you will have a working monitoring stack, with alert rules that Prometheus has actually loaded and that you have proved fire.

Tasks

Task 1: Install Prometheus

Debian and Ubuntu ship both components in universe, so there is no tarball to unpack:

sudo apt update
sudo apt install -y prometheus prometheus-node-exporter prometheus-alertmanager

If you need a newer release than the distribution carries, download a pinned version - never a wildcard:

PROM_VER=2.53.5
wget "https://github.com/prometheus/prometheus/releases/download/v${PROM_VER}/prometheus-${PROM_VER}.linux-amd64.tar.gz"
tar xzf "prometheus-${PROM_VER}.linux-amd64.tar.gz"
sudo install -m 0755 "prometheus-${PROM_VER}.linux-amd64/prometheus" /usr/local/bin/
sudo install -m 0755 "prometheus-${PROM_VER}.linux-amd64/promtool"  /usr/local/bin/

Now write the configuration. Note the two blocks the naive version of this lab leaves out: rule_files: (without it Prometheus never reads your alert rules) and alerting: (without it a firing alert has nowhere to go).

sudo mkdir -p /etc/prometheus/rules

sudo tee /etc/prometheus/prometheus.yml <<'EOF'
global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - /etc/prometheus/rules/*.yml

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['localhost:9093']

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
  - job_name: 'node'
    static_configs:
      - targets: ['localhost:9100']
EOF

sudo promtool check config /etc/prometheus/prometheus.yml
sudo systemctl restart prometheus

promtool ships with the prometheus package and inside the upstream tarball. Run it before every restart or reload. If command -v promtool finds nothing, you installed only the node exporter.

Task 2: Start node_exporter

The prometheus-node-exporter package installs a systemd unit, so the exporter survives your SSH session ending:

sudo systemctl enable --now prometheus-node-exporter
systemctl is-active prometheus-node-exporter
curl -s localhost:9100/metrics | head -5

Running node_exporter in the foreground, or backgrounding it with &, ties it to your login shell. It dies when the session ends and never comes back after a reboot, which is how a “monitored” host quietly stops being monitored.

Task 3: Verify scraping

Open http://localhost:9090/targets. Both prometheus and node targets should be UP.

Task 4: Query in Prometheus

Open http://localhost:9090/graph. Try:

# CPU usage
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Memory available (GB)
node_memory_MemAvailable_bytes / 1024^3

# Disk usage % - exclude pseudo-filesystems, never allow-list one fstype
100 * (1 - node_filesystem_avail_bytes{fstype!~"tmpfs|devtmpfs|overlay|squashfs|autofs"}
          / node_filesystem_size_bytes{fstype!~"tmpfs|devtmpfs|overlay|squashfs|autofs"})

# Inode usage % - exhausts independently of blocks
100 * (1 - node_filesystem_files_free{fstype!~"tmpfs|devtmpfs|overlay"}
          / node_filesystem_files{fstype!~"tmpfs|devtmpfs|overlay"})

# Network receive rate
rate(node_network_receive_bytes_total[5m]) * 8

{fstype="ext4"} is the filter to avoid. It excludes XFS - the RHEL, Rocky and Alma default root filesystem - plus Btrfs, ext3 and every network mount, so those hosts produce no series and never alert at any fill level. Exclude pseudo-filesystems with fstype!~ instead.

Task 5: Install Grafana

Grafana is not in the Debian or Ubuntu archives, so sudo apt install grafana fails with E: Unable to locate package grafana. Add the vendor repository and its signing key first:

sudo apt install -y wget gpg
sudo mkdir -p /etc/apt/keyrings
wget -qO- https://apt.grafana.com/gpg.key \
  | sudo gpg --dearmor -o /etc/apt/keyrings/grafana.gpg
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" \
  | sudo tee /etc/apt/sources.list.d/grafana.list

sudo apt update
sudo apt install -y grafana
sudo systemctl enable --now grafana-server

The signed-by= clause scopes that key to that one repository. A key dropped into the global trusted set can sign packages for any repository the host uses, which is a supply-chain hole you install by hand.

Open http://localhost:3000. Default admin/admin (change on first login).

Task 6: Add Prometheus data source

1. Configuration > Data sources > Add data source.
2. Type: Prometheus.
3. URL: http://localhost:9090.
4. Save and test.

Task 7: Build a host dashboard

Create a new dashboard. Add panels:

• CPU usage (stat).
• Memory used (gauge).
• Disk usage per mount (barchart).
• Network throughput (timeseries).

For each, use the PromQL from Task 4.

Task 8: Configure alerts

Write the rules into the directory rule_files: already points at. A rules file anywhere else is a text file, not an alert.

sudo tee /etc/prometheus/rules/host.yml <<'EOF'
groups:
  - name: host
    rules:
      - alert: HostDown
        expr: up == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "{{ $labels.instance }} target {{ $labels.job }} is down"

      - alert: FilesystemSpaceLow
        expr: |
          100 * (1 - node_filesystem_avail_bytes{fstype!~"tmpfs|devtmpfs|overlay|squashfs|autofs"}
                    / node_filesystem_size_bytes{fstype!~"tmpfs|devtmpfs|overlay|squashfs|autofs"}) > 85
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.mountpoint }} above 85% used"

      - alert: FilesystemInodesLow
        expr: |
          100 * (1 - node_filesystem_files_free{fstype!~"tmpfs|devtmpfs|overlay"}
                    / node_filesystem_files{fstype!~"tmpfs|devtmpfs|overlay"}) > 85
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.mountpoint }} above 85% of inodes used"

      - alert: FilesystemWillFillIn4h
        expr: predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|devtmpfs|overlay"}[6h], 4*3600) < 0
        for: 30m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.mountpoint }} fills within 4h at current rate"

      - alert: FilesystemReadOnly
        expr: node_filesystem_readonly{fstype!~"tmpfs|devtmpfs|overlay"} == 1
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "{{ $labels.mountpoint }} remounted read-only"
EOF

sudo promtool check rules /etc/prometheus/rules/host.yml
sudo promtool check config /etc/prometheus/prometheus.yml
sudo systemctl reload prometheus

Task 9: Validate - prove the alerts exist and fire

Do not accept the file on disk as evidence. Ask Prometheus what it has actually loaded:

# 1. Config and rules parse cleanly.
promtool check config /etc/prometheus/prometheus.yml
promtool check rules  /etc/prometheus/rules/host.yml

# 2. Prometheus has loaded both rules. Must print HostDown and DiskSpaceLow.
curl -s localhost:9090/api/v1/rules | jq -r '.data.groups[].rules[].name'

# 3. The last reload succeeded, and rule evaluation is not erroring.
curl -s localhost:9090/api/v1/status/config | jq -r '.status'
curl -s 'localhost:9090/api/v1/query?query=prometheus_config_last_reload_successful' \
  | jq -r '.data.result[].value[1]'      # must be 1

# 4. Alertmanager is a reachable target, not just a line in the config.
curl -s localhost:9090/api/v1/alertmanagers | jq -r '.data.activeAlertmanagers[].url'

Then force the alert, which is the only proof that matters:

sudo systemctl stop prometheus-node-exporter
sleep 180                                  # scrape interval + the 'for: 2m' window
curl -s localhost:9090/api/v1/alerts | jq -r '.data.alerts[].labels.alertname'
# expect: HostDown

sudo systemctl start prometheus-node-exporter
sleep 60
curl -s localhost:9090/api/v1/alerts | jq -r '.data.alerts[].labels.alertname'
# expect: no HostDown - it must clear on its own

An alert that fires but never clears is its own incident: the on-call learns to ignore it, and the next real firing looks identical to the stuck one.

Task 10: Document

PROMETHEUS LAB
=============
Host: <host>
Date: 2026-08-09

Components:
- Prometheus: http://localhost:9090
- node_exporter: http://localhost:9100/metrics
- Grafana: http://localhost:3000

Dashboards:
- Host: <name> with CPU, memory, disk, network

Alerts (as reported by /api/v1/rules, not as written to disk):
- HostDown: up == 0 for 2m
- DiskSpaceLow: disk > 80% for 10m

Evidence:
- promtool check config / check rules: pass
- prometheus_config_last_reload_successful: 1
- Alertmanager target: http://localhost:9093
- HostDown observed firing after stopping node_exporter: yes
- HostDown observed clearing after restart: yes

The last three lines are the point. “Configured” is a claim; “observed firing and clearing” is evidence. A monitoring handover that records only the claim is how a team ends up believing it has coverage it has never once seen work.

Validation

Run these from the lab host. Every one of them asks a running process what it believes, rather than reading a file you wrote.

# Both units are enabled, so the stack survives a reboot.
systemctl is-enabled prometheus prometheus-node-exporter grafana-server
systemctl is-active  prometheus prometheus-node-exporter grafana-server

# Config and rules parse.
promtool check config /etc/prometheus/prometheus.yml     # SUCCESS
promtool check rules  /etc/prometheus/rules/host.yml     # SUCCESS: 2 rules found

# Both scrape targets are UP.
curl -s localhost:9090/api/v1/targets \
  | jq -r '.data.activeTargets[] | "\(.labels.job) \(.health)"'
# expect: prometheus up
#         node        up

# Both rules are loaded in the running process.
curl -s localhost:9090/api/v1/rules | jq -r '.data.groups[].rules[].name'
# expect: HostDown
#         DiskSpaceLow

# The last reload actually took.
curl -s 'localhost:9090/api/v1/query?query=prometheus_config_last_reload_successful' \
  | jq -r '.data.result[].value[1]'          # expect: 1

# node_exporter is producing the series the dashboard depends on.
curl -s 'localhost:9090/api/v1/query?query=node_filesystem_avail_bytes' \
  | jq -r '.data.result | length'            # expect: a non-zero count

Then the one that cannot be faked — an alert observed firing and clearing, as performed in Task 9:

sudo systemctl stop prometheus-node-exporter
sleep 180
curl -s localhost:9090/api/v1/alerts \
  | jq -r '.data.alerts[] | "\(.labels.alertname) \(.state)"'
# expect: HostDown firing

sudo systemctl start prometheus-node-exporter
sleep 60
curl -s localhost:9090/api/v1/alerts | jq -r '.data.alerts[].labels.alertname'
# expect: no HostDown

If the target reports up but the alert never fires, check the for: window against how long you actually waited before concluding the rule is broken.

Expected outcome

• Prometheus, node_exporter and Grafana all run under systemd and are enabled, so the stack comes back after a reboot.
• Prometheus scrapes two targets and both report up.
• Two alerting rules — HostDown and DiskSpaceLow — are loaded in the running process, confirmed from /api/v1/rules rather than from the file on disk.
• prometheus_config_last_reload_successful is 1.
• A Grafana dashboard renders CPU, memory, disk and network for the host from live data.
• HostDown has been observed firing when node_exporter is stopped and clearing when it is restarted. This is the definition of done: a rule that has never been seen to fire is not coverage, it is a configuration file.

Cleanup

The lab installs three services that listen on the network. On a shared or internet-facing host, leaving them running is the finding your next audit reports.

# Stop and disable the stack.
sudo systemctl disable --now grafana-server prometheus prometheus-node-exporter

# Remove the packages if this host was not meant to keep them.
sudo apt-get remove --purge -y prometheus prometheus-node-exporter grafana

# Remove the rule file and any drop-ins you added.
sudo rm -f /etc/prometheus/rules/host.yml

# Confirm nothing is still listening on the lab ports.
ss -ltnp | grep -E ':(9090|9100|9093|3000)\b' || echo "all lab ports closed"

If you are keeping the stack, at minimum bind it to localhost or put it behind the host firewall — Prometheus and node_exporter ship with no authentication, and /metrics discloses a detailed inventory of the host to anyone who can reach the port.