Lab: Run node_exporter on Linux

Objective

By the end of this lab a Linux host will be a Prometheus scrape target running node_exporter under a dedicated account with the systemd sandboxing directives applied, and you will be able to say which collector on that specific host is the most expensive — from its own timings, not from a blog post.

Then you will break it in the way that matters. Anyone can stop the service and watch up go to zero; that failure is loud and alerts on itself. This lab spends its second half on the failure that does not: one regex in one flag, the filesystem metrics gone, the service healthy, the scrape succeeding, up at 1, and every disk-space panel on the host quietly empty.

Architecture

node_exporter runs directly on the host, as the lesson prescribes — a static binary under systemd, not a container, because a containerised node_exporter reports the container’s view of the kernel unless you mount half of /proc and /sys into it. Prometheus runs in a container sharing the host network namespace, so it can reach a loopback-bound exporter.

  +----------------------------------------------------------+
  | Linux VM (Debian 12 / Ubuntu 24.04)                       |
  |                                                            |
  |   /proc, /sys, /proc/1/mountinfo    <- kernel surfaces     |
  |            |                                               |
  |            v                                               |
  |   +-------------------------+                              |
  |   | node_exporter 1.8.2     |  systemd unit                |
  |   |   collectors            |  User=node_exporter          |
  |   |   textfile dir  --------+--> /var/lib/node_exporter/   |
  |   |   :9100 (127.0.0.1)     |        textfile_collector/   |
  |   +-----------+-------------+                              |
  |               ^ scrape every 15s                           |
  |   +-----------+-------------+                              |
  |   | prometheus 2.55         |  container, network_mode:    |
  |   |   :9090 (127.0.0.1)     |  host                        |
  |   +-------------------------+                              |
  +------------------------------------------------------------+

The exporter binds loopback only. Prometheus can reach it because it shares the host’s network namespace; nothing else on the network can, which is the posture the lesson recommends for a host whose /metrics enumerates mount points, interfaces and systemd units.

Requirements

A disposable Linux VM with systemd — Debian 12 or Ubuntu 24.04 — and sudo. Declare B-nested and use a VM you are willing to throw away.
Do not run this on your workstation. It creates a system user, installs a binary into /usr/local/bin, writes a systemd unit, and creates a directory under /var/lib. Cleanup removes all four, but a mistake in the middle leaves a service account and a listening socket behind.
Docker Engine 28.x and Docker Compose v2 on the same VM, for Prometheus.
curl, jq and tar on the VM, plus outbound HTTPS to github.com for the release download.
Free TCP ports 9100 and 9090, both bound to loopback.
No out-of-band access requirement. This lab does not touch the network configuration, the firewall or sshd, so no step can lock you out of the VM. It can leave a service running, which Cleanup addresses.
About 200 MiB of disk for the binary, the Prometheus TSDB and the images.

Scenario

A host reports 95% disk full. The on-call opens the dashboard for that host and finds three panels — CPU, memory, network — and nothing about disk. Everyone assumes the disk panel is broken today. It has been broken for eleven weeks, since a change that tightened the filesystem collector’s exclude regex to quiet down a noisy Kubernetes node and was rolled out to the whole fleet.

The incident was not the disk filling up. The incident was that nobody could see it, and nothing alerted, because from Prometheus’s point of view the host had been up and scraping successfully the entire time.

You are going to build that host and then reproduce the eleven weeks in about four minutes.

Tasks

Task 1: Capture the starting state

Cleanup restores what was here, so record it first. This is not ceremony: on a VM that has been used for something else, one of these may already exist and removing it at the end would be a larger change than anything the lab makes.

LABDIR="$HOME/rb-obs-nodeexporter"
mkdir -p "$LABDIR"
cd "$LABDIR"

{
  echo "captured: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
  echo "user node_exporter exists: $(id -u node_exporter >/dev/null 2>&1 && echo yes || echo no)"
  echo "binary present: $(test -x /usr/local/bin/node_exporter && echo yes || echo no)"
  echo "unit present: $(test -f /etc/systemd/system/node_exporter.service && echo yes || echo no)"
  echo "textfile dir present: $(test -d /var/lib/node_exporter && echo yes || echo no)"
  echo "root filesystem type: $(findmnt -no FSTYPE /)"
} | tee pre-lab.txt

sudo ss -ltnp | grep -E ':9100|:9090' || echo "ports 9100 and 9090 are free"

Note the root filesystem type in pre-lab.txt. Task 6 needs it, and reading it now rather than guessing later is the difference between a lab that works on your VM and one that works on the VM it was written on.

Task 2: Install the binary, and verify it

Download the release and its checksum file from the same release, then check one against the other. A download you did not verify is a download you are trusting the network for:

NE_VERSION=1.8.2
BASE="https://github.com/prometheus/node_exporter/releases/download/v${NE_VERSION}"

cd "$LABDIR"
curl -fsSLO "${BASE}/node_exporter-${NE_VERSION}.linux-amd64.tar.gz"
curl -fsSLO "${BASE}/sha256sums.txt"

sha256sum -c --ignore-missing sha256sums.txt

The expected line is node_exporter-1.8.2.linux-amd64.tar.gz: OK. Anything else — and in particular a silent pass because the filename did not match and --ignore-missing skipped it — means stop and re-download. Confirm the check actually checked something:

sha256sum -c --ignore-missing sha256sums.txt 2>/dev/null | grep -c ': OK$'

That must print 1, not 0.

tar xzf "node_exporter-${NE_VERSION}.linux-amd64.tar.gz"
sudo install -o root -g root -m 0755 \
  "node_exporter-${NE_VERSION}.linux-amd64/node_exporter" \
  /usr/local/bin/node_exporter

/usr/local/bin/node_exporter --version

Create the service account and the textfile directory. The account has no shell and no home: it exists to own a process and to read /proc, and it should be able to do nothing else if the process is ever compromised.

Configuration changelab VM— Creates a system account with no login shell and a directory under /var/lib

$ sudo useradd --system --no-create-home --shell /usr/sbin/nologin node_exporter

sudo mkdir -p /var/lib/node_exporter/textfile_collector
sudo chown -R node_exporter:node_exporter /var/lib/node_exporter
sudo chmod 0755 /var/lib/node_exporter/textfile_collector

Task 3: Write the systemd unit and start it

Save this as node_exporter.service in the lab directory. Every flag below is a decision, and the comments after the block say what each one buys:

[Unit]
Description=Prometheus node_exporter
Documentation=https://github.com/prometheus/node_exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter \
  --web.listen-address=127.0.0.1:9100 \
  --collector.filesystem.mount-points-exclude=^/(sys|proc|dev|run|var/lib/docker/.+)($|/) \
  --collector.netdev.device-exclude=^(veth.*|docker.*|lo)$ \
  --collector.diskstats.device-exclude=^(loop.*|ram.*|dm-.*|sr.*)$ \
  --collector.textfile.directory=/var/lib/node_exporter/textfile_collector
Restart=on-failure
RestartSec=5s
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
ReadWritePaths=/var/lib/node_exporter/textfile_collector

[Install]
WantedBy=multi-user.target

--web.listen-address=127.0.0.1:9100 is the single most valuable line here. The default is 0.0.0.0, and /metrics is a readable inventory of the host’s mounts, interfaces and hardware.
The three -exclude flags remove veth/docker interfaces, loop and device-mapper block devices, and the pseudo-filesystems. They are the reason the panels stay readable on a host that runs containers.
ProtectSystem=strict makes the whole filesystem read-only to this process except for the paths named in ReadWritePaths. That single directory is the only thing node_exporter ever needs to write to, and only via the textfile collector.
Note what is not here: --collector.systemd and --collector.processes. Both are opt-in, and Task 5 measures what they cost before you decide.

Install it and start:

Service impact possiblelab VM— Installs a unit file, reloads systemd, and starts a new listening service

$ sudo install -o root -g root -m 0644 node_exporter.service /etc/systemd/system/node_exporter.service && sudo systemctl daemon-reload && sudo systemctl enable --now node_exporter.service

systemctl is-active node_exporter.service
sudo ss -ltnp | grep ':9100'
curl -sf http://127.0.0.1:9100/metrics | grep -cE '^node_'

The third command should print a few hundred: that is the number of metric lines the enabled collectors produced on this host. Then confirm the families the dashboards depend on are all present:

for FAMILY in node_cpu_seconds_total node_memory_MemAvailable_bytes \
              node_filesystem_avail_bytes node_disk_io_time_seconds_total \
              node_network_receive_bytes_total; do
  COUNT=$(curl -sf http://127.0.0.1:9100/metrics | grep -c "^${FAMILY}")
  printf '%-38s %s series\n' "$FAMILY" "$COUNT"
done

Five families, all non-zero. Keep this output; Task 6 runs the same loop and the difference is the whole point.

Task 4: Point Prometheus at it

prometheus.yml:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - /etc/prometheus/rules/*.yml

scrape_configs:
  - job_name: node_exporter
    static_configs:
      - targets: ['localhost:9100']

compose.yaml. Prometheus shares the host network namespace so it can reach a loopback-bound exporter, and binds its own listener to loopback for the same reason the exporter does:

name: rb-obs-nodeexporter

services:
  prometheus:
    image: prom/prometheus:v2.55.1
    network_mode: host
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention.time=6h'
      - '--web.listen-address=127.0.0.1:9090'
      # Lets you reload rules with a POST instead of a restart, which would
      # lose the head block and with it the before/after comparison.
      - '--web.enable-lifecycle'
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./rules:/etc/prometheus/rules:ro
      - prom-data:/prometheus

volumes:
  prom-data:

mkdir -p "$LABDIR/rules"
cd "$LABDIR"
docker compose up -d
sleep 20

curl -sfG http://127.0.0.1:9090/api/v1/query \
  --data-urlencode 'query=up{job="node_exporter"}' \
| jq -r '.data.result[] | "up=\(.value[1]) instance=\(.metric.instance)"'

curl -sfG http://127.0.0.1:9090/api/v1/query \
  --data-urlencode 'query=scrape_duration_seconds{job="node_exporter"}' \
| jq -r '.data.result[0].value[1]'

up=1, and a scrape duration well under a second. The lesson’s threshold is that a scrape above one second on a default install means a collector is misbehaving — which is the next task.

Task 5: Price every collector on this host

node_exporter times each collector separately and exposes the result. This is the measurement that replaces opinion about which collectors are expensive:

curl -sfG http://127.0.0.1:9090/api/v1/query \
  --data-urlencode 'query=topk(10, node_scrape_collector_duration_seconds)' \
| jq -r '.data.result[] | "\(.metric.collector) \(.value[1])"' \
| sort -k2 -g -r

Also check that every collector is actually succeeding. A collector that fails on this hardware reports zero duration and no metrics, and looks identical to one that is merely cheap:

curl -sfG http://127.0.0.1:9090/api/v1/query \
  --data-urlencode 'query=node_scrape_collector_success == 0' \
| jq -r '.data.result[] | "FAILING: \(.metric.collector)"' \
| grep . || echo "all collectors succeeding"

On a VM, several hardware collectors will be failing or empty — hwmon, rapl, edac, thermal_zone have nothing to read inside a guest. That is expected and worth seeing: it is why the default set is a starting point rather than an answer.

Now enable the two opt-in collectors the lesson calls out, using a drop-in rather than editing the unit. A drop-in is the production-correct way to add a flag, and it makes the change trivially reversible:

sudo mkdir -p /etc/systemd/system/node_exporter.service.d
sudo tee /etc/systemd/system/node_exporter.service.d/10-optin.conf >/dev/null <<'DROPIN'
[Service]
# An empty ExecStart= resets the list before the new one is appended.
# Without this line systemd refuses the unit: Type=simple takes one ExecStart.
ExecStart=
ExecStart=/usr/local/bin/node_exporter \
  --web.listen-address=127.0.0.1:9100 \
  --collector.systemd \
  --collector.processes \
  --collector.filesystem.mount-points-exclude=^/(sys|proc|dev|run|var/lib/docker/.+)($|/) \
  --collector.netdev.device-exclude=^(veth.*|docker.*|lo)$ \
  --collector.diskstats.device-exclude=^(loop.*|ram.*|dm-.*|sr.*)$ \
  --collector.textfile.directory=/var/lib/node_exporter/textfile_collector
DROPIN

sudo systemctl daemon-reload
sudo systemctl restart node_exporter.service
sleep 45

Re-run the ranking and the scrape duration. Record both numbers:

curl -sfG http://127.0.0.1:9090/api/v1/query \
  --data-urlencode 'query=topk(10, node_scrape_collector_duration_seconds)' \
| jq -r '.data.result[] | "\(.metric.collector) \(.value[1])"' \
| sort -k2 -g -r

curl -sfG http://127.0.0.1:9090/api/v1/query \
  --data-urlencode 'query=scrape_duration_seconds{job="node_exporter"}' \
| jq -r '.data.result[0].value[1]'

Roll the drop-in back before continuing, so the rest of the lab runs against the unit you actually wrote:

sudo rm -f /etc/systemd/system/node_exporter.service.d/10-optin.conf
sudo systemctl daemon-reload
sudo systemctl restart node_exporter.service

Task 6: The failure that does not move `up`

The change is one regex. Find your root filesystem type from pre-lab.txt and add it to the excluded types, exactly as somebody would while trying to quieten a noisy node:

ROOTFS=$(findmnt -no FSTYPE /)
echo "excluding $ROOTFS"

sudo mkdir -p /etc/systemd/system/node_exporter.service.d
sudo tee /etc/systemd/system/node_exporter.service.d/20-break.conf >/dev/null <<DROPIN
[Service]
ExecStart=
ExecStart=/usr/local/bin/node_exporter \\
  --web.listen-address=127.0.0.1:9100 \\
  --collector.filesystem.fs-types-exclude=^(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs|tmpfs|${ROOTFS})\$ \\
  --collector.textfile.directory=/var/lib/node_exporter/textfile_collector
DROPIN

sudo systemctl daemon-reload
sudo systemctl restart node_exporter.service
sleep 45

Now measure. First, the thing that would page you:

curl -sfG http://127.0.0.1:9090/api/v1/query \
  --data-urlencode 'query=up{job="node_exporter"}' \
| jq -r '.data.result[0].value[1]'

curl -sfG http://127.0.0.1:9090/api/v1/query \
  --data-urlencode 'query=node_scrape_collector_success{collector="filesystem"}' \
| jq -r '.data.result[0].value[1]'

Both return 1. The exporter is up, the scrape succeeds, and the filesystem collector reports success — because it did succeed. It walked the mount table, excluded everything it was told to exclude, and returned the empty set it was asked for.

Now the thing nobody is watching:

curl -sfG http://127.0.0.1:9090/api/v1/query \
  --data-urlencode 'query=count(node_filesystem_avail_bytes)' \
| jq -r '.data.result[0].value[1] // "0 — the metric no longer exists"'

The count is gone, or collapsed to a handful of mounts you do not care about. Every disk-space panel, every inode alert, every capacity report for this host now has no data behind it, and there is no error anywhere in the system saying so.

Restore it now and watch the series return. Task 8 re-applies exactly this drop-in, deliberately, to fire an alert on it:

sudo rm -f /etc/systemd/system/node_exporter.service.d/20-break.conf
sudo systemctl daemon-reload
sudo systemctl restart node_exporter.service
sleep 30
curl -sfG http://127.0.0.1:9090/api/v1/query \
  --data-urlencode 'query=count(node_filesystem_avail_bytes)' \
| jq -r '.data.result[0].value[1]'

The series come back. Note how fast: one restart, no data loss for the future, and a permanent hole in the history for the window the exclude was live.

Task 7: Expose something no collector knows about

The textfile collector is the escape hatch for metrics that are not in the kernel. Backup age is the canonical example: no collector can know it, and it is exactly the kind of thing that fails silently for months.

sudo tee /usr/local/bin/backup_age.sh >/dev/null <<'SCRIPT'
#!/bin/bash
# Emits backup_age_seconds to the node_exporter textfile directory.
set -euo pipefail

OUT=/var/lib/node_exporter/textfile_collector
BACKUP=/var/backups/last-good.tar.gz

TMP=$(mktemp "$OUT/backup_age.XXXXXX")
LATEST=$(stat -c %Y "$BACKUP" 2>/dev/null || echo 0)
NOW=$(date +%s)

{
  printf '# HELP backup_age_seconds Seconds since the last good backup.\n'
  printf '# TYPE backup_age_seconds gauge\n'
  printf 'backup_age_seconds %d\n' "$((NOW - LATEST))"
} > "$TMP"

chmod 0644 "$TMP"
mv "$TMP" "$OUT/backup_age.prom"
SCRIPT

sudo chmod 0755 /usr/local/bin/backup_age.sh
sudo touch /var/backups/last-good.tar.gz
sudo -u node_exporter /usr/local/bin/backup_age.sh

curl -sf http://127.0.0.1:9100/metrics | grep '^backup_age_seconds'

The mktemp then mv pattern is not defensive style, it is the requirement. mv within one filesystem is an atomic rename, so node_exporter either reads the whole previous file or the whole new one. Writing directly to backup_age.prom gives the scrape a window in which it can read half a file.

See what a bad file does. Write one that is not valid exposition format and watch the scrape react:

echo 'this is not exposition format' \
| sudo tee /var/lib/node_exporter/textfile_collector/broken.prom >/dev/null
sleep 20

curl -sf http://127.0.0.1:9100/metrics | grep '^node_textfile_scrape_error'
curl -sfG http://127.0.0.1:9090/api/v1/query \
  --data-urlencode 'query=up{job="node_exporter"}' | jq -r '.data.result[0].value[1]'

node_textfile_scrape_error goes to 1 while up stays at 1. This is the same shape as Task 6 — a real fault reported by a metric nobody is alerting on — and it is the reason node_textfile_scrape_error is worth an alert of its own on any host that uses the textfile collector for anything load-bearing.

sudo rm -f /var/lib/node_exporter/textfile_collector/broken.prom

What the textfile collector actually reads

On every scrape the textfile collector lists the configured directory, opens each file ending in .prom, and parses it as Prometheus exposition text. It does not cache: a file removed between scrapes takes its metrics with it, and a file whose writer has stopped running keeps serving the same stale value forever, with no indication that it is stale.

That last property is the trap in every textfile metric. backup_age_seconds computed at write time is a snapshot; if the cron job that writes it dies, the gauge freezes at whatever the backup age was at the moment of the last successful run — a small, healthy-looking number, held constant, while the real backup age grows without bound.

The defence is to make the metric self-invalidating: write the timestamp instead of the age, and compute the age in PromQL. A gauge holding backup_last_success_timestamp_seconds that stops being updated produces a time() - backup_last_success_timestamp_seconds that grows on its own, so the frozen writer and the missed backup produce the same alert.

Task 8: Alert on both failure shapes

Two rules. They look similar and they catch entirely different things. rules/node.yml:

groups:
  - name: node_exporter_health
    interval: 15s
    rules:
      # LOUD: the exporter is gone, the port is blocked, or the host is down.
      # This one alerts on the absence of a successful scrape.
      - alert: NodeExporterDown
        expr: up{job="node_exporter"} == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: 'node_exporter not scrapeable on {{ $labels.instance }}'

      # SILENT: the scrape succeeds and a metric family we depend on is not
      # in it. absent() is the operator that turns "no series" into a value,
      # because a query for a metric that does not exist returns nothing at
      # all, and nothing never crosses a threshold. The count threshold covers
      # the partial case, where the exclude leaves one or two mounts behind;
      # pick it per host class from the count you recorded in Task 3.
      - alert: NodeFilesystemMetricsMissing
        expr: absent(node_filesystem_avail_bytes) or count(node_filesystem_avail_bytes) < 2
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: 'node_filesystem_* is empty while the exporter reports healthy'

curl -sf -X POST http://127.0.0.1:9090/-/reload && echo reloaded
curl -sf http://127.0.0.1:9090/api/v1/rules \
| jq -r '.data.groups[].rules[] | "\(.name) state=\(.state // "n/a")"'

Fire the silent one by re-applying the Task 6 drop-in, waiting out the for, and reading ALERTS:

ROOTFS=$(findmnt -no FSTYPE /)
sudo tee /etc/systemd/system/node_exporter.service.d/20-break.conf >/dev/null <<DROPIN
[Service]
ExecStart=
ExecStart=/usr/local/bin/node_exporter \\
  --web.listen-address=127.0.0.1:9100 \\
  --collector.filesystem.fs-types-exclude=^(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs|tmpfs|${ROOTFS})\$ \\
  --collector.textfile.directory=/var/lib/node_exporter/textfile_collector
DROPIN

sudo systemctl daemon-reload
sudo systemctl restart node_exporter.service
sleep 330

curl -sfG http://127.0.0.1:9090/api/v1/query \
  --data-urlencode 'query=ALERTS{alertstate="firing"}' \
| jq -r '.data.result[] | "\(.metric.alertname) severity=\(.metric.severity)"'

NodeFilesystemMetricsMissing is firing and NodeExporterDown is not. That is the whole lesson of this lab in one command’s output.

Restore, then fire the loud one to see the contrast:

Service impact possiblelab VM— Removes the break, then deliberately stops the exporter for two minutes

$ sudo rm -f /etc/systemd/system/node_exporter.service.d/20-break.conf && sudo systemctl daemon-reload && sudo systemctl restart node_exporter.service && sleep 60 && sudo systemctl stop node_exporter.service

sleep 150
curl -sfG http://127.0.0.1:9090/api/v1/query \
  --data-urlencode 'query=ALERTS{alertstate="firing"}' \
| jq -r '.data.result[] | "\(.metric.alertname) severity=\(.metric.severity)"'

sudo systemctl start node_exporter.service

Validation

1. The unit is running with the hardening applied and the sandbox is real.

systemctl is-active node_exporter.service
systemctl show node_exporter.service \
  -p User -p ProtectSystem -p NoNewPrivileges -p ReadWritePaths

User=node_exporter, ProtectSystem=strict, NoNewPrivileges=yes, and the textfile directory as the only writable path.

2. The exporter is not reachable from off-host. From another machine on the same network, or from the VM’s own non-loopback address:

HOSTIP=$(hostname -I | awk '{print $1}')
curl -sf --max-time 3 "http://$HOSTIP:9100/metrics" >/dev/null \
  && echo "FAIL: reachable on $HOSTIP" \
  || echo "PASS: not reachable on $HOSTIP"

3. All five metric families are back. Re-run the Task 3 loop; every family reports a non-zero series count.

4. The collector timings were recorded twice, with and without the two opt-in collectors, and the two most expensive collectors on your host are named in your notes.

5. Both alerts resolved to inactive after the restores.

curl -sf http://127.0.0.1:9090/api/v1/rules \
| jq -r '.data.groups[].rules[] | select(.type=="alerting")
         | "\(.name) \(.state)"'

Both must read inactive. An alert still firing after the restore means one of the drop-ins is still in place — check ls /etc/systemd/system/node_exporter.service.d/.

6. The textfile metric is present and the error gauge is zero.

curl -sf http://127.0.0.1:9100/metrics \
| grep -E '^(backup_age_seconds|node_textfile_scrape_error)'

Expected Outcome

node_exporter 1.8.2 running under systemd as a dedicated non-login account, bound to loopback, with ProtectSystem=strict and one writable path.
A Prometheus container scraping it, showing up=1 and a sub-second scrape.
A recorded ranking of collector durations on this host, before and after the two opt-in collectors, and the failing collectors on this hardware named.
backup_age_seconds served through the textfile collector by a script using the atomic-rename pattern.
Two alert rules, one fired by each failure shape, with the ALERTS output for each and both back to inactive.
/etc/systemd/system/node_exporter.service.d/ empty or absent.

Troubleshooting

sha256sum -c prints nothing and exits 0. --ignore-missing skipped every line because no filename matched. Check you are in the directory the tarball was downloaded to and that the version in the filename matches NE_VERSION.

The unit fails with status=203/EXEC. The binary is not at /usr/local/bin/node_exporter, or is not executable. ls -l it and re-run the install step.

The unit fails with status=209/STDOUT or a permissions error on the textfile directory. ReadWritePaths names a directory that does not exist. Create it before starting the unit; ProtectSystem=strict makes everything else read-only, so node_exporter cannot create it for itself.

systemctl restart fails after adding a drop-in, complaining about more than one ExecStart. The drop-in is missing the bare ExecStart= reset line. A Type=simple unit takes exactly one, and a drop-in appends rather than replaces unless you clear the list first.

Prometheus reports up=0 with a connection-refused error. The exporter is bound to 127.0.0.1 and Prometheus is not in the host network namespace. Check network_mode: host is present in compose.yaml and that the container was recreated after it was added — docker compose up -d --force-recreate prometheus.

node_filesystem_* is missing and you did not break it. Either the fs-types-exclude regex covers your root filesystem type, or every mount is excluded by mount-points-exclude. Compare findmnt -no FSTYPE,TARGET against both regexes.

node_textfile_scrape_error is 1 and you removed the bad file. Removal takes effect on the next scrape, not immediately. Wait one scrape interval, then re-read. If it persists, another .prom file in the directory is also invalid — ls /var/lib/node_exporter/textfile_collector/.

Several hardware collectors report node_scrape_collector_success == 0. Expected inside a VM: hwmon, rapl, edac and thermal_zone have no underlying device. Disable them explicitly if the noise bothers you, but do not mistake them for a broken install.

Cleanup

This lab installed a binary, created a system account, wrote a systemd unit and a directory under /var/lib, and started a listening service. All five come back out, in that order, and pre-lab.txt says which of them existed before.

Step 1 — read the capture:

cat "$HOME/rb-obs-nodeexporter/pre-lab.txt"

Step 2 — stop the service and remove the unit. Skip this entirely if pre-lab.txt says the unit was already present, because in that case you are looking at somebody else’s node_exporter:

Destructivelab VM— Stops and disables the service, then removes the unit and its drop-in directory

$ sudo systemctl disable --now node_exporter.service && sudo rm -rf /etc/systemd/system/node_exporter.service.d /etc/systemd/system/node_exporter.service && sudo systemctl daemon-reload

Step 3 — remove the binary, the scripts and the state directory:

Destructivelab VM— Removes the exporter binary, the textfile script, and the textfile directory with any .prom files in it

$ sudo rm -rf /usr/local/bin/node_exporter /usr/local/bin/backup_age.sh /var/lib/node_exporter

Step 4 — remove the service account, but only if the lab created it:

Check the capture before running userdel

userdel on an account something else owns removes that thing’s ability to run. pre-lab.txt recorded whether node_exporter existed before you started:

grep 'user node_exporter exists' "$HOME/rb-obs-nodeexporter/pre-lab.txt"

Run the next command only if that line says no.

Destructivelab VM— Deletes the system account created in Task 2

$ sudo userdel node_exporter

Step 5 — remove Prometheus and its volume, then the lab directory:

Data-loss risklab VM— Removes the container and this project's named volume, discarding the TSDB with all of the lab's measurements

$ cd ~/rb-obs-nodeexporter && docker compose down -v

mkdir -p "$HOME/rb-obs-deliverables"
cp -a "$HOME/rb-obs-nodeexporter/pre-lab.txt" \
      "$HOME/rb-obs-nodeexporter/node_exporter.service" \
      "$HOME/rb-obs-nodeexporter/rules/node.yml" \
      "$HOME/rb-obs-deliverables/"

rm -rf "$HOME/rb-obs-nodeexporter"
sudo rm -f /var/backups/last-good.tar.gz

Step 6 — confirm the host is back where it started:

id -u node_exporter 2>/dev/null && echo "account still present" || echo "account gone"
sudo ss -ltnp | grep -E ':9100|:9090' || echo "both ports free"
test -e /usr/local/bin/node_exporter && echo "binary still present" || echo "binary gone"

Production notes

The exclude flags are the change that needs a canary, and the reason is Task 6. Every other line in the unit fails loudly if it is wrong. The exclude regexes fail by producing less telemetry, which no alert on the exporter itself can see. Roll a regex change to one host, run the metric-family count against that host, and only then roll it to the fleet — the same discipline you would apply to a firewall rule, for the same reason: the failure mode is silence.

Alert on metric existence, per host class, not per metric. The NodeFilesystemMetricsMissing rule generalises: for each host class, list the metric families the dashboards and alerts depend on, and alert on absent() for each. It is a dozen lines of YAML per class and it converts the entire category of silent collector failure into a loud one. The rule that catches the disk panel also catches the day someone upgrades node_exporter and a metric is renamed.

A textfile metric with no freshness signal is worse than no metric. Write the timestamp of the last success and derive the age in PromQL, so a dead writer and a real failure look the same to the alert. A frozen gauge that reads healthy is the one shape of monitoring failure that actively misleads the on-call rather than merely failing to inform them.

Bind to the management address and firewall the port anyway. /metrics enumerates mount points, network interfaces and — with the systemd collector — the full unit list. That is a useful inventory for whoever is on the network, and neither of the two controls is sufficient alone: the listen address is undone by one drop-in, and the firewall rule is undone by one merged change to the security group.

Pin the version and treat the upgrade as a change. node_exporter renames and retires metrics between minor releases. The unit above pins 1.8.2; an upgrade goes to one host, gets its metric families counted against the previous list, and only then goes wider. That comparison is the same query you wrote in Task 3, which is the argument for keeping it.

Objectives

Prerequisites

Objective

Architecture

Requirements

Scenario

Tasks

Task 1: Capture the starting state

Task 2: Install the binary, and verify it

Task 3: Write the systemd unit and start it

Task 4: Point Prometheus at it

Task 5: Price every collector on this host

Task 6: The failure that does not move `up`

Task 7: Expose something no collector knows about

Task 8: Alert on both failure shapes

Validation

Expected Outcome

Troubleshooting

Cleanup

Production notes

Deliverables

Verification status

Objectives

Prerequisites

Objective

Architecture

Requirements

Scenario

Tasks

Task 1: Capture the starting state

Task 2: Install the binary, and verify it

Task 3: Write the systemd unit and start it

Task 4: Point Prometheus at it

Task 5: Price every collector on this host

Task 6: The failure that does not move up

Task 7: Expose something no collector knows about

Task 8: Alert on both failure shapes

Validation

Expected Outcome

Troubleshooting

Cleanup

Production notes

Deliverables

Verification status

Task 6: The failure that does not move `up`