Exporter Performance

A monitoring fleet with 1,200 hosts, each running node_exporter on port 9100, scraped every 15 seconds by two Prometheus replicas. The fleet is quiet. A new SRE adds a community exporter that scans /proc for every process on every host, with a 5-second scrape interval, no caching, and no rate limiting. By day three the monitoring Prometheus is using 80% of its CPU, the SRE’s new exporter is the top contributor to scrape duration, and dashboards across the fleet are reporting “no data” because the scrape timeout is firing.

The right question is not “how do I scale Prometheus”. The right question is “what is the cost of each exporter scrape on the host, and is the scrape interval matched to the rate at which the metric changes”.

What it is

Exporter performance is the discipline of measuring the cost an exporter imposes on the host it runs on, and matching the scrape interval to the metric beat. The cost has three components:

• CPU cost of producing the response. The exporter reads files, queries upstream systems, or recomputes derived values on each scrape. The cost depends on the number of enabled collectors and the upstream’s own latency.
• Memory cost of the response in flight. The serialised response is allocated, written, and freed. With default text/plain, a node_exporter response is ~200 KiB; a misbehaving exporter can produce megabytes.
• Cost of being scraped too often. Each scrape is a fixed overhead plus the variable cost above. Scrape interval dominates the total cost when set lower than the metric changes.

The metric beat is the rate at which the underlying value changes. A counter that increments every request should be scraped often enough that no single scrape loses more than a small fraction of the increments. A gauge that changes once a minute does not need a 5-second scrape interval.

Why a sysadmin cares

The cost of an exporter is paid by the host it runs on and by the Prometheus that scrapes it. Both budgets are finite. The operational consequences of getting the cost wrong are:

• Scrape timeouts. A scrape that takes longer than scrape_timeout is cancelled and recorded as failed. The target’s up gauge flips to 0. The alert fires.
• Host CPU pressure. A heavy collector on a busy host can double the CPU spent in the exporter.
• Prometheus CPU pressure. Serialising and parsing many large responses is the dominant cost in a high-target Prometheus. The TSDB head also costs more as series count grows.
• Hidden cost of expensive collectors. Some collectors (filesystem, processes in node_exporter) walk the whole filesystem or /proc table on every scrape. The cost is proportional to host complexity, not to the number of metrics returned.

The trade-off: every metric you scrape has a cost, and the cost is paid whether or not you look at the metric.

How it works

The mental model is a per-target cost equation:

  total scrape cost per Prometheus
   = N_targets x N_replicas
     x (CPU_per_scrape + Memory_per_response)
     x scrape_rate
   + N_targets x response_size x scrape_rate
                 (network + parse)

Three knobs reduce the cost:

1. Scrape interval. Doubling the interval halves the cost. A 15s interval costs a third of a 5s interval.
2. Response size. Fewer labels, fewer metric families, and tighter label values mean a smaller response. A label of 256 bytes vs 16 bytes is a 16x difference on that one metric.
3. CPU per scrape. Disable collectors you do not need, cache expensive reads, and avoid per-scrape fan-out.

node_exporter collectors (per scrape, approximate):

  cpu            O(N_CPUs)             cheap
  meminfo        O(1)                  cheap
  netdev         O(N_interfaces)       cheap
  filesystem     O(N_mounts x depth)   EXPENSIVE on large hosts
  diskstats      O(N_disks)            cheap
  loadavg        O(1)                  cheap
  processes      O(N_processes)        EXPENSIVE on busy hosts
  systemd        O(N_units)            moderate
  wifi           O(N_interfaces)       cheap

The filesystem and processes collectors are the typical culprits. filesystem walks every mount on every scrape; a host with 30 mounts and 4 levels of nesting pays for 120 directory walks per scrape. processes walks /proc/*/stat for every process; a host with 500 processes pays for 500 reads per scrape.

How to configure it

The configuration is split between the exporter flags and the Prometheus scrape config.

1. Disable collectors you do not need. node_exporter ships with --collector.<name> flags:

# /etc/default/prometheus-node-exporter
ARGS="--web.listen-address=127.0.0.1:9100 \
      --collector.filesystem.mount-points-exclude='^/(sys|proc|dev|run)($|/)' \
      --collector.netclass.ignore-lo=true \
      --collector.wifi=false \
      --collector.powerstats=false \
      --collector.systemd.disable-unit-list-cache=false"

Disabling the wifi collector on a server without Wi-Fi is free. Excluding /sys, /proc, /dev, and /run from the filesystem collector removes four mounts from every scrape.

2. Match the scrape interval to the metric beat. The default is 15 seconds. For metrics that change faster than once a second, you need a smaller interval. For metrics that change once a minute, a longer interval is fine.

# prometheus.yml
scrape_configs:
  - job_name: node
    scrape_interval: 30s           # host metrics beat slowly
    scrape_timeout: 10s
    static_configs:
      - targets: ['10.0.1.4:9100']

  - job_name: api-gateway
    scrape_interval: 15s           # HTTP traffic beats faster
    scrape_timeout: 5s
    static_configs:
      - targets: ['10.0.2.4:8080']

  - job_name: batch-summary
    scrape_interval: 5m            # cron job, one scrape per run is enough
    scrape_timeout: 30s
    static_configs:
      - targets: ['10.0.3.4:9101']

The rule of thumb is scrape_interval = max_rate_metric_beat / 3 so that consecutive scrapes do not skip a beat of the fastest counter you alert on. A counter that increments every request in a service that does 100 req/s should be scraped at most every 3 seconds; a counter that increments every 30s can be scraped every 60s.

3. Singleflight for expensive upstream reads. If your exporter reads from a slow upstream, use singleflight to collapse concurrent scrapes:

import "golang.org/x/sync/singleflight"

var (
  group singleflight.Group
  cache atomic.Pointer[cachedMetrics]
)

func readUpstream(ctx context.Context) (Metrics, error) {
  if c := cache.Load(); c != nil && time.Since(c.ts) < 5*time.Second {
    return c.v, nil
  }
  v, err, _ := group.Do("upstream", func() (interface{}, error) {
    return doReadUpstream(ctx)
  })
  if err != nil {
    return Metrics{}, err
  }
  cache.Store(&cachedMetrics{v: v.(Metrics), ts: time.Now()})
  return v.(Metrics), nil
}

The cache keeps the result for 5 seconds. The singleflight ensures only one in-flight upstream read even if Prometheus scrapes the exporter twice in quick succession.

4. Use the client library’s self-observation metrics. The exporter’s own /metrics reports on its health:

# Total scrapes per second
rate(promhttp_metric_handler_requests_total[1m])

# Scrape latency distribution
histogram_quantile(0.99,
  rate(promhttp_metric_handler_request_duration_seconds_bucket[5m]))

# Goroutine count
go_goroutines

# Process memory
process_resident_memory_bytes

# GC overhead
rate(go_gc_duration_seconds_sum[5m])

A scrape latency p99 above 1 second is a problem. A goroutine count that climbs is a leak. A GC overhead above 25% of CPU is a memory churn problem.

How to validate it

Validate the cost on both sides: the exporter host and the scraper. All commands are READ-ONLY.

# 1. Confirm the exporter's self-observation metrics are present.
curl -sf http://127.0.0.1:9100/metrics | grep -E \
  '^(go_goroutines|process_resident_memory_bytes|process_cpu_seconds_total|promhttp_metric_handler_requests_total)'

# 2. Measure the exporter's own response time from the host.
time curl -sf http://127.0.0.1:9100/metrics > /dev/null

real    0m0.045s
user    0m0.012s
sys     0m0.018s

# 3. Inspect the response size.
curl -sf http://127.0.0.1:9100/metrics | wc -c

198432

# 4. Scrape duration from Prometheus's view (per job).
scrape_duration_seconds{job="node"}

# 5. Scrape latency p99 from the exporter's own histogram.
histogram_quantile(0.99,
  rate(promhttp_metric_handler_request_duration_seconds_bucket[5m]))

# 6. Process CPU from the exporter host's view.
rate(process_cpu_seconds_total[1m])

# 7. Inspect /proc cost on a node_exporter host with strace
#    (READ-ONLY, expensive; do not run in production).
strace -c -p $(pidof node_exporter) 2>&1 | head -20

The outputs confirm: the exporter exposes self-observation, the response is small enough, the scrape is fast enough, and the per-scrape CPU is bounded.

How it can fail

Five specific failure modes:

1. Scrape interval too short. A 5-second interval on a counter that increments every 30 seconds wastes 5x the scrape budget for no information gain. Symptom: scrape_duration_seconds p99 is well below the timeout but the cumulative scrape rate is high; the histogram bucket spread is dominated by a single value.
2. Expensive collector enabled by default. The filesystem or processes collector is on for a host with hundreds of mounts or processes. Symptom: node_exporter is in the top 5 processes by CPU on the host; per-scrape CPU climbs with host complexity.
3. Goroutine leak in a custom exporter. A handler spawns a goroutine per request and never cleans up. Symptom: go_goroutines climbs by the number of scrapes per interval; process_resident_memory_bytes follows.
4. Cache stampede. Many Prometheus replicas scrape the same exporter, each missing the cache and hitting the upstream. Symptom: upstream request rate spikes with Prometheus replica count; the exporter’s upstream_requests_total grows linearly with scrapes, not with upstream changes.
5. Histogram quantile outside the bucket range. A histogram whose buckets stop at 1 second for a metric that takes 5 seconds to respond. Symptom: histogram_quantile returns the bucket boundary for p99; the dashboard shows the upper limit regardless of the actual value.

How to troubleshoot it

Diagnose in this order; it is cheapest to confirm the cost on the host first.

1. What is the exporter’s CPU and memory? top or process_resident_memory_bytes and process_cpu_seconds_total.
2. What is the response size? wc -c on the /metrics body. If it is megabytes, the cardinality is the problem.
3. What is the per-scrape duration? scrape_duration_seconds in Prometheus, or histogram_quantile of promhttp_metric_handler_request_duration_seconds.
4. Which collector is the cost? On node_exporter, enable --collector.filesystem.mount-points-exclude for mounts you do not care about; consider --collectors.disable for wifi, systemd, processes if they are not needed.
5. Are goroutines climbing? go_goroutines over time. A steady climb is a leak; investigate handler code.
6. Is the scrape interval too tight? Compare the rate of change of your slowest counter to the scrape interval. If the counter barely changes between scrapes, the interval is too short.

Security implications

Performance and security interact at one boundary: the exporter’s attack surface scales with its capability. A collector that reads /proc/*/cmdline exposes process command lines, which may include credentials passed on the command line. A collector that reads /proc/*/environ exposes environment variables. The cost of a collector is not only CPU — it is the data the collector emits, which becomes data on the wire, in the TSDB, and in any downstream remote-write receiver.

The discipline is to disable collectors that emit data you do not need, and to redact sensitive label values with metric_relabel_configs.

Performance implications

The performance implications of an exporter are the lesson. Three summary rules:

• Match the scrape interval to the metric beat. A 3-to-1 ratio is a starting point; slower is cheaper.
• Disable collectors you do not use. Every enabled collector has a per-scrape cost.
• Cache and singleflight expensive upstream reads. The scrape interval is the floor for cache TTL.

The trade-off of a tight scrape interval is faster alerting. The cost is a heavier platform. The break-even is the slowest metric you actually alert on.

Production guidance

• Set --web.listen-address to a private interface and --scrape_interval to match the metric beat.
• Disable collectors that are not needed. The defaults are sensible for a developer workstation, not for a server.
• Use the client library’s self-observation metrics. Alert on go_goroutines, process_resident_memory_bytes, and histogram_quantile(0.99, scrape_duration).
• Cache expensive upstream reads with a TTL and use singleflight to avoid stampedes.
• Pre-allocate label values where possible. A 16-byte label value allocates less and serialises faster than a 256-byte one.

Verification

You should now be able to answer:

• What are the three components of scrape cost on an exporter host, and which knob controls each?
• How would you decide whether a 5-second or 30-second scrape interval is right for a given metric?
• Which node_exporter collectors are most expensive on a host with hundreds of mounts or processes, and how would you reduce their cost?
• What is the singleflight pattern, and when does it help?
• Which three self-observation metrics should you alert on for every exporter in the fleet?

Quiz