Exporter Types

The first three weeks of a new monitoring rollout go smoothly: node_exporter reports host metrics, a custom Go binary emits http_requests_total, and the mysql_exporter reports database counters. The fourth week a teammate introduces a community haproxy_exporter that was last updated in 2022, a colleague runs the Pushgateway for a batch job that was easier to push than to expose, and a third team shoves a Prometheus client library into a legacy Java service the team does not own. Three months later nobody knows who maintains what, the Pushgateway is the single source of truth for a cron job that did not run for two days, and the legacy Java service is the only signal that something is up with the order-processing path.

Every exporter adoption decision is permanent in practice. Knowing the categories up front prevents most of this pain.

What it is

The term exporter covers three structurally different kinds of process:

• Dedicated exporter. A standalone process whose only job is to translate an external system’s metrics into the Prometheus contract. node_exporter, mysqld_exporter, snmp_exporter, blackbox_exporter, and most of the prometheus/*_exporter repositories are dedicated exporters. They run alongside the thing they observe, often as a sidecar.
• Instrumentation library. A client library that is embedded in the application code. prometheus/client_golang, prometheus/client_java, prometheus/client_python, and their siblings expose the Prometheus contract from inside the process itself. They use the application’s own data structures and counters.
• Push gateway. A short-lived metrics buffer that accepts pushed samples from processes that cannot or should not be scraped directly (cron jobs, batch workers, AWS Lambda). The Prometheus Pushgateway is the canonical implementation; Open Source alternatives exist but the name “Pushgateway” usually means the official binary.

A fourth category cuts across all three: integration components like Grafana Alloy, the OpenTelemetry Collector, and Telegraf. They are not exporters in the strict sense but they frequently translate other telemetry formats (StatsD, collectd, OTLP) into the Prometheus contract on the way to a scrape.

Why a sysadmin cares

The category you choose determines three operational properties:

• What the exporter can see. A dedicated exporter sees what the external system publishes; an instrumentation library sees what the application itself tracks. The two are not interchangeable.
• Who maintains it. A dedicated exporter is maintained by whoever maintains the upstream system (or by a community volunteer); an instrumentation library is maintained by the Prometheus project and updated in lock-step with the application.
• What survives a restart. A dedicated exporter restarts its own state from scratch; counters that live in the application are gone when the application restarts. Pushgateway survives application restart but accumulates staleness if the producer stops pushing.

Mixing the categories unconsciously — using a dedicated exporter where an instrumentation library was the right choice, or using the Pushgateway for a long-running service — produces silent gaps. A cron job that “exits cleanly” but the Pushgateway never sees because the curl was wrong leaves a metric stuck at the last value forever.

How it works

The mental model is a one-axis spectrum with three regions:

                  Prometheus contract
                          ^
                          |
   Pushgateway  Instrumentation library  Dedicated exporter
   (push)       (in-process, embed)      (sidecar, translate)
        |                |                       |
   cron job          long-running          database, MQ,
   batch worker      HTTP service          load balancer,
   Lambda                                       host metrics

Each region has a stable shape. The shape determines when you reach for it:

+------------------+-------------------+-------------------------+
| Category         | Sees              | Survives restart        |
+------------------+-------------------+-------------------------+
| Dedicated        | External system   | Yes (own state)         |
| Instrumentation  | Application code  | No (process dies)       |
| Pushgateway      | Pushed samples    | Yes until deleted/TTL   |
+------------------+-------------------+-------------------------+

For node_exporter the dedicated-exporter category applies: it reads /proc and /sys on the host and re-exports those numbers as Prometheus metrics. For mysqld_exporter it reads SHOW GLOBAL STATUS from a database it is given credentials for, and translates the result. For blackbox_exporter it probes a target over ICMP, TCP, HTTP, or DNS and reports the result. None of these are libraries; all of them are standalone binaries.

For an HTTP service in Go, the instrumentation library category applies: client_golang is imported into the service’s source, the service exposes /metrics directly, and the library handles the counter-reset detection, label serialisation, and process metrics. There is no sidecar.

For a cron job that runs every five minutes and aggregates a result, the Pushgateway applies: the job pushes one sample, exits, and Prometheus scrapes the Pushgateway on its regular schedule. The Pushgateway is the wrong tool for long-running services because the metrics it holds do not distinguish between “the service is alive but idle” and “the service crashed an hour ago and never pushed again”.

How to configure it

Each category has a different configuration story.

1. Dedicated exporter (node_exporter). Run as a systemd unit, bind to localhost, scrape from the local Prometheus:

# /etc/systemd/system/node_exporter.service
[Unit]
Description=Prometheus node_exporter
After=network-online.target
Wants=network-online.target

[Service]
User=node_exporter
ExecStart=/usr/bin/node_exporter \
  --web.listen-address=127.0.0.1:9100 \
  --collector.filesystem.mount-points-exclude=^/(sys|proc|dev)($|/) \
  --collector.netclass.ignore-lo=true
Restart=on-failure

[Install]
WantedBy=multi-user.target

2. Instrumentation library (Go HTTP service). Embed prometheus/client_golang and expose /metrics:

import (
  "net/http"
  "github.com/prometheus/client_golang/prometheus/promhttp"
)

func main() {
  mux := http.NewServeMux()
  mux.Handle("/metrics", promhttp.Handler())
  mux.HandleFunc("/", rootHandler)
  http.ListenAndServe(":8080", mux)
}

The scrape config points at the application directly, not at a sidecar.

3. Pushgateway (batch job). Run Pushgateway as a sidecar, push from the job, scrape the gateway:

# /etc/default/pushgateway
ARGS="--web.listen-address=127.0.0.1:9091 \
      --push.disable-consistency-check"

# At the end of the batch job
echo "job_runs_total{job=\"nightly-rollup\"} 1" \
  | curl --data-binary @- http://127.0.0.1:9091/metrics/job/nightly-rollup

# prometheus.yml — honour_labels preserves the job label pushed
# by the batch; without it the gateway's job becomes the only job
# in Prometheus.
scrape_configs:
  - job_name: pushgateway
    honor_labels: true
    static_configs:
      - targets: ['127.0.0.1:9091']

4. Blackbox exporter (synthetic probe). Run as a sidecar, scrape with the exporter’s prober config:

# blackbox.yml
modules:
  http_2xx:
    prober: http
    timeout: 5s
    http:
      valid_status_codes: [200, 204]
      preferred_ip_protocol: ip4
      tls_config:
        insecure_skip_verify: false
  icmp:
    prober: icmp
    timeout: 3s

# prometheus.yml
scrape_configs:
  - job_name: blackbox-http
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets: ['https://example.com/healthz']
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9115

The blackbox_exporter is a translator; it does not export business metrics, only the result of the synthetic probe.

5. Idiomatic naming. When introducing a new exporter, the naming convention is <thing>_exporter for dedicated exporters, the application name for in-process instrumentation, and the job name for Pushgateway targets. A redis_exporter is the canonical Redis exporter; mongo_exporter is the canonical MongoDB exporter. Avoid vendor names in the binary name when the community name is stable.

How to validate it

Confirm each category is doing what its category implies. All commands are READ-ONLY.

# 1. Dedicated exporter: confirm it is reachable and serves
#    the contract.
curl -sI http://10.0.1.4:9100/metrics | head -3

# 2. Instrumentation library: confirm the application itself
#    serves /metrics and the library metrics are present.
curl -sf http://10.0.1.4:8080/metrics | grep -E \
  '^(go_goroutines|process_resident_memory_bytes|go_gc_duration_seconds)'

# 3. Pushgateway: confirm the pushed metrics are present and
#    carry the job label the producer set.
curl -sf http://127.0.0.1:9091/metrics | grep nightly_rollup

# 4. Blackbox exporter: confirm the probe is running and
#    reporting probe_success.
probe_success{instance="https://example.com/healthz"}

# 5. In Prometheus: confirm the up gauge per category.
up{job="node"}
up{job="checkout-api"}     # instrumentation library
up{job="pushgateway"}      # Pushgateway
up{job="blackbox-http"}    # blackbox

The outputs confirm the categories are operating as categories — exporter reads from external state, library emits process-state metrics, Pushgateway holds pushed samples, blackbox reports on the probe result.

How it can fail

Five specific failure modes:

1. Pushgateway used for a long-running service. The service pushes its counter once on startup, then stops pushing because it crashed. Prometheus scrapes the gateway, the counter is stuck at the last value, dashboards show a flat line, alerts fire on absence. Symptom: a metric that does not move for days while the upstream process restarts repeatedly.
2. Dedicated exporter runs as root because it needs to read /proc. A misconfiguration lets a remote actor read the host filesystem metrics through the exporter. Symptom: the exporter is reachable on 0.0.0.0:9100 from outside the monitoring subnet.
3. Instrumentation library exposed on the same port as the public API. A user finds /metrics by accident and reads application internals (request paths, user IDs that leaked into labels). Symptom: no specific log entry; observed only via a security review or a real incident.
4. Blackbox exporter probe timeout on every check. The exporter’s timeout is set lower than the target’s actual response time. Symptom: probe_success is 0 for every target; probe_duration_seconds is at or near the timeout for every probe.
5. Community exporter abandoned. The binary is pinned to a version that is no longer maintained. A CVE is published, no fix arrives. Symptom: the exporter version has not changed in 18+ months; the GitHub release page shows no commits; the team’s CVE feed flags a CVE against the version.

How to troubleshoot it

Diagnose in this order; it is cheapest to confirm the category first and the content second.

1. Which category is this? If you do not know the answer, the answer is to find out. The category determines the next four steps.
2. Is the endpoint reachable? For a dedicated exporter or instrumentation library, curl -I first. For the Pushgateway, check the producer’s last push in the gateway’s log.
3. Is the body valid? Pipe to promtool check metrics. The parser reports the exact line and the rule it broke.
4. Is the producer alive? For an instrumentation library, check that the application process is running. For a dedicated exporter, check the sidecar. For Pushgateway, check the producing job’s own log — not the gateway’s.
5. Are the right metrics present? Each category emits a characteristic set: dedicated exporters expose the upstream system’s metrics; instrumentation libraries expose process metrics (go_*, process_*, jvm_*); Pushgateway exposes what was pushed.
6. Is the category wrong? If a counter has not moved in days, suspect Pushgateway. If the metrics are dimensionless and unrelated to the system being monitored, suspect a dedicated exporter for a different upstream.

Security implications

Each category has a different attack surface:

• Dedicated exporter. Reads /proc, /sys, network interfaces, sometimes database credentials. The exporter needs the same privileges as the thing it observes, which is often “almost everything” on the host.
• Instrumentation library. Has the application’s own privileges and can leak application state through label values. The library itself does not add privileges.
• Pushgateway. Accepts unauthenticated pushes by default. A malicious actor on the network can poison any metric the gateway accepts. The Pushgateway’s --web.config.file option enables basic auth and TLS, which most teams do not configure.
• Blackbox exporter. Probes external targets. The probe is read-only; the risk is the target: a blackbox probe to a malicious endpoint can be coerced into probing internal hosts via DNS rebinding or SSRF.

Default configurations often expose more than is appropriate: Prometheus’s API is unprotected by default, the Grafana UI is anonymous-admin by default, and exporters typically bind to 0.0.0.0. The security lesson in this module covers each configuration in depth.

Performance implications

• Dedicated exporter. Cost is a function of the number of collectors enabled. node_exporter with all collectors on a host with 30 filesystems and 100 processes spends ~50 ms per scrape. Disable the collectors you do not need.
• Instrumentation library. Cost is the overhead of the library on the application’s hot path. client_golang counters are atomic increments; expect ~10-20 ns per increment. Histograms are more expensive because of their bucket fan-out.
• Pushgateway. Cost is RAM per pushed metric and the cost of the push itself. A Pushgateway with 1 million pushed series consumes roughly the same RAM as Prometheus would for the same series.
• Blackbox exporter. Cost is the cost of the probe. An ICMP probe to a host is cheap; an HTTPS probe with full certificate validation is more expensive because of the TLS handshake.

The trade-off: each category is cheap when used correctly and expensive when over-used. The Pushgateway is the most common over-use because it is the easiest to drop in.

Production guidance

• Default to instrumentation libraries for services you own and can change. Default to dedicated exporters for systems you cannot modify. Default to Pushgateway only for batch jobs and only with a TTL.
• Pin every exporter version. Track the upstream release cadence. Watch the *_build_info gauge for silent updates.
• Bind exporters to localhost or a private interface. Expose via a sidecar proxy or service mesh when remote scraping is required.
• Use the idiomatic naming convention. A name like redis_exporter is searchable; a name like acme-prom-exporter-v3 is not.
• Audit the Pushgateway quarterly. Anything that should not be a push is a candidate to convert to a pull.

Verification

You should now be able to answer:

• What is the structural difference between a dedicated exporter, an instrumentation library, and the Pushgateway?
• When does each category apply, and what is the symptom of using the wrong one?
• How would you instrument a long-running Go service, a PostgreSQL database you do not own, and a five-minute batch job?
• What is the canonical name for a new Redis exporter, and why does the name matter?
• What characteristic metric set tells you a process is being scraped via instrumentation rather than a dedicated exporter?

Quiz