Exporter Down

The 02:14 page goes to the platform on-call. The alert name is something harmless like TargetDown; the description says “node-7 has been failing scrapes for 2 minutes”. The on-call engineer runs curl http://node-7:9100/metrics from the Prometheus host and gets connection refused. The node is up; the exporter is not. The first link of the chain has failed. This is the lesson.

Roughly seven in ten missing-metric incidents start here. The diagnosis is fast once the right command is run; the cost is that the wrong command is almost always run first. This lesson is the discipline of running the right command first, every time.

What it is

An exporter is down when the process that owns the metrics endpoint is not running on the target host. The endpoint is either closed at the TCP level (the process is gone) or held open by something that is not the exporter (a previous instance that did not release the port, a debug shell, a copy-paste of the exporter binary). Either way, the scrape fails and up == 0 for the affected target.

The “exporter is down” failure shape is distinct from every other link in the chain because it is observable on the target host itself. The exporter host can answer the question before Prometheus is consulted at all. The four canonical signs:

• The process is absent from the process table.
• The TCP port the exporter binds is not in LISTEN.
• The systemd unit is in inactive or failed state.
• The container is in Exited state, or absent from the container runtime.

A scrape that fails for any other reason (network blocked, TLS failure, HTTP 401, parse error) still leaves the exporter running. The TCP port is still bound. The diagnostic that distinguishes “exporter down” from every other failure is the TCP port check.

Why a sysadmin cares

“Exporter down” is the cheapest missing-metric failure to diagnose and the easiest to prevent. The diagnostic is one command on the exporter host. The prevention is process supervision: a unit that restarts on exit, a Kubernetes deployment with a liveness probe, a Docker Compose service with restart: unless-stopped. A fleet without supervision eventually loses an exporter; a fleet with supervision eventually loses an exporter less often and recovers faster.

Two production pains follow:

1. Silent gaps in metrics. A node_exporter that exited six hours ago and was never restarted is invisible until an alert page opens the dashboard panel and sees an empty graph. The on-call engineer cannot investigate from the dashboard alone; the right move is to check the target host.
2. Stale dashboards. A panel that has been empty for hours is a panel that was not paged on. The on-call shift inherits a quiet gap that takes the same time to diagnose as a fresh one.

How it works

An exporter is a normal process that owns a TCP port and answers HTTP GET requests with a Prometheus exposition. The lifecycle has four states:

   +---------+    +-----------+    +----------+
   | started | -> | listening | -> | scraping |
   +---------+    +-----------+    +----------+
        |               |              |
        v               v              v
    process exit    port collision  OOM kill,
    (clean)         (already bound)  segfault,
                   (host-side)      timeout
        |               |              |
        +---------------+--------------+
                        v
                   +---------+
                   |  down   |  <- the failure shape this lesson covers
                   +---------+

The “started” state covers process initialisation: parsing flags, reading config, opening files. The “listening” state covers binding the TCP port. The “scraping” state covers answering requests. A failure in any of the three produces the “down” state. The exit reason is the diagnostic.

The supervisor’s job is to detect the transition into “down” and restart the process. The supervisor can be systemd, a Kubernetes controller, a Docker restart policy, or a hand-rolled loop. The platform dictates the choice.

Under the hood

How to configure it

The right supervisor depends on the platform. Three examples, each showing the minimum that prevents the silent failure.

systemd unit

# /etc/systemd/system/node_exporter.service
[Unit]
Description=Prometheus node exporter
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=node_exporter
ExecStart=/usr/local/bin/node_exporter \
  --web.listen-address=:9100 \
  --collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host)($|/.*)$
Restart=always
RestartSec=5s
# cap the restart rate to avoid a tight loop filling the journal
StartLimitIntervalSec=60
StartLimitBurst=10

[Install]
WantedBy=multi-user.target

The three lines that matter: Restart=always restarts on any clean exit, RestartSec=5s waits five seconds between restarts to give a transient dependency time to recover, and the StartLimit* pair caps the burst so a tight crash loop is visible rather than silent.

Kubernetes Deployment

# node-exporter.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: node-exporter
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels: { app: node-exporter }
  template:
    metadata:
      labels: { app: node-exporter }
    spec:
      containers:
        - name: node-exporter
          image: prom/node-exporter:v1.8.2
          args:
            - '--web.listen-address=:9100'
          ports:
            - containerPort: 9100
              name: metrics
          # the three lines that keep the exporter alive
          livenessProbe:
            httpGet:
              path: /metrics
              port: 9100
            initialDelaySeconds: 10
            periodSeconds: 15
            failureThreshold: 3
          resources:
            requests: { cpu: 50m, memory: 64Mi }
            limits:   { cpu: 200m, memory: 128Mi }
      # the host filesystem is required for the disk collectors
      hostPID: true
      hostNetwork: true

The liveness probe is the supervisor. Three failures in a row restart the container. The resources.limits.memory is the ceiling that turns a slow leak into an OOM kill rather than a silent degradation.

Docker Compose

# docker-compose.yml
services:
  node-exporter:
    image: prom/node-exporter:v1.8.2
    command:
      - '--web.listen-address=:9100'
    pid: host
    network_mode: host
    restart: unless-stopped
    # cap the restart loop with a logging driver that survives restarts
    logging:
      driver: json-file
      options:
        max-size: 10m
        max-file: 3

The restart: unless-stopped policy covers process exit. The logging block prevents a tight crash loop from filling the disk with journal output.

How to validate it

The diagnostic ladder for “exporter down”. Every command is read-only except where noted.

# Step 1: confirm the symptom from Prometheus
curl -s 'http://prom:9090/api/v1/targets?state=active' \
  | jq '.data.activeTargets[]
        | select(.health=="down")
        | {instance: .labels.instance, lastError: .lastError}'

# Step 2: confirm the process is not running (on the target host)
ssh node-7.internal 'ps -ef | grep -v grep | grep node_exporter'
# expected (healthy): one line for the node_exporter PID
# expected (down):    no output

# Step 3: confirm the port is not bound
ssh node-7.internal 'ss -tlnp | grep :9100'
# expected (healthy): one line with the bound address and PID
# expected (down):    no output

# Step 4: confirm the unit state (systemd hosts)
ssh node-7.internal 'systemctl status node_exporter --no-pager | head -20'
# expected (healthy): Active: active (running)
# expected (down):    Active: inactive (dead) or failed

# Step 5: confirm the container state (Docker hosts)
ssh node-7.internal 'docker ps -a --filter name=node_exporter --format "{{.Names}} {{.Status}}"'
# expected (healthy): node_exporter Up 14 hours
# expected (down):    node_exporter Exited (137) 6 hours ago

A target that fails steps 2 and 3 is “exporter down” with no ambiguity. A target that passes steps 2 and 3 but fails step 1 is not “exporter down”; it is a network, scrape config, or relabel problem (lessons 03, 04, 05).

How it can fail

Six failure shapes appear repeatedly. The first three are the ones to fix; the second three are the ones to prevent.

1. OOM kill. The exporter exceeds its memory limit and is killed by the kernel or the container runtime. Symptom: systemctl status shows Main PID: ... (code=killed, signal=KILL) and the journal contains Out of memory: Killed process; docker ps shows Exited (137); kubectl describe pod shows Reason: OOMKilled. The fix is a higher memory limit after measuring the actual working set, plus a Prometheus alert on the container’s own memory usage.
2. Segfault. The exporter binary crashes for a reason outside the operator’s direct control: a collector that fails on a particular kernel version, a CGO issue, a native dependency mismatch. Symptom: code=exited, status=139/SIGSEGV in systemctl status; Exited (139) in docker ps. The fix is usually a version bump or pinning the kernel; the containment is the restart policy.
3. Liveness probe restart loop. The Kubernetes liveness probe fires because the exporter is slow to respond under load, the probe threshold is too aggressive, or the probe path is wrong. Symptom: kubectl describe pod shows Restart Count: 12 and Last State: Terminated, Reason: Completed. The fix is a longer initialDelaySeconds, a higher failureThreshold, or a less aggressive probe path such as /healthz if the exporter exposes one.
4. Port collision. A second process has bound the exporter’s port (a leftover from a previous instance, a debug session, a copy of the binary in a different path). Symptom: the exporter process is running but ss -tlnp shows a different PID owning the port, or the exporter fails to start with bind: address already in use in the journal. The fix is to find and stop the conflicting process.
5. Missed restart on deploy. A new release deploys the exporter as a new container, but the old container is still terminating. The old container is in Exited; the new one is starting. Symptom: a brief scrape gap during the rollout. The fix is a Kubernetes RollingUpdate with maxUnavailable: 0 and maxSurge: 1, or a Compose stop_grace_period long enough for the exporter to flush.
6. Dependency down. The exporter depends on a filesystem mount, a sysfs path, or a service that has been removed. The exporter cannot start. Symptom: the journal shows repeated start attempts with the same error. The fix is to restore the dependency or change the exporter’s flags to tolerate its absence.

How to troubleshoot it

Security implications

The exporter’s /metrics endpoint exposes operational detail that is sensitive in production: kernel version, mount points, process arguments, file descriptors. The endpoint is read-by-default for any host that can reach it. The lessons:

• Bind the exporter to the network the Prometheus server lives on (--web.listen-address=<prom-net-ip>:9100), not to 0.0.0.0. The default of 0.0.0.0 is convenient for local testing and dangerous in production.
• Enable basic auth on the exporter if the scrape is across an untrusted network (--web.config=/etc/node_exporter/web.yml with a basic_auth_users block).
• Restrict the port at the firewall: only the Prometheus server and the on-call jump box should reach :9100.

Performance implications

The exporter itself is cheap (node_exporter uses roughly 30-60 MiB of RAM and a few percent of CPU at a 15-second scrape interval). The cost of running many exporters on many hosts is dominated by the scrape volume, not the exporter’s own footprint. The performance traps:

• Too many collectors enabled. node_exporter ships with roughly forty collectors; only the filesystem, meminfo, loadavg, and netdev collectors are usually wanted. Disable the rest with --collector.disable-defaults and explicit --collector.<name> flags to keep the scrape small.
• Scrape interval too aggressive. A 5-second scrape interval quadruples the cost of an exporter with 5000 series. 15 seconds is the default; 30 seconds is acceptable for low-priority fleets.
• Memory leak in a custom collector. A custom collector that allocates per-scrape and does not free will OOM the exporter after hours. Profile with --collectors.profile and validate under load.

Production guidance

• Supervise every exporter. systemd, Kubernetes, or Compose. No naked processes.
• Set Restart=always (or the platform equivalent) and a restart interval that absorbs transient failures without hiding a real bug.
• Alert on up == 0 with for: 2m. The persistent outage is the signal; the single dropped packet is not.
• Alert on the restart count: more than three restarts in ten minutes is a crash loop. The diagnostic is Restart Count on Kubernetes, Start Count in systemd, or RestartCount in the Docker API.
• Run a synthetic scrape from a separate host every minute. The synthetic scrape is independent of the production scrape and catches a fleet-wide exporter failure faster than the per-target alerts.
• Document the recovery procedure in the runbook. The on-call engineer at 02:14 should be able to recover an exporter in five minutes from the runbook alone.

Verification

You should now be able to answer:

• What are the four canonical signs that an exporter is down on the target host?
• Which command on the target host is the single highest-signal diagnostic for “exporter down”?
• What is the difference between an OOM kill and a segfault in systemctl status and docker ps output?
• How do you recover an exporter without restarting Prometheus and without losing the scrape timeline?
• Why is restarting Prometheus the wrong first move for an “exporter down” incident?

Quiz