The Exporter Contract

A 03:00 incident is open. The on-call engineer opens Prometheus and sees that node_cpu_seconds_total for a host has stopped updating. The scraper says up{job="node"} is 1. The host is reachable on port 9100. The engineer curls the exporter and gets back a metric whose value is NaN. The contract was broken; nobody noticed until the page fired.

Every Prometheus exporter, whether node_exporter, the official mysql_exporter, or a 200-line Go program a colleague wrote last quarter, speaks the same dialect. That dialect is the exporter contract. The contract is what lets a single scraper ingest a hundred different producer processes without per-producer code paths. The contract is also what fails silently when one producer drifts: Prometheus keeps scraping, drops or rejects the bad samples, and only the eventual gap in a dashboard tells you.

This lesson states the contract precisely. The next lesson applies it to the taxonomy of exporters. Everything after that assumes you have this contract in your head.

What it is

An exporter is a process that exposes telemetry over HTTP, in a text-based format Prometheus understands, so Prometheus can pull samples on a schedule. The exporter contract is the set of rules that format must obey.

Concretely, the contract is:

1. One HTTP endpoint. Conventionally /metrics on a port the operator chose. The same process usually exposes /-/healthy and /-/ready for orchestrator probes.
2. One text-based, line-oriented exposition format (default content type text/plain; version=0.0.4; charset=utf-8). One line is one metric sample, one HELP comment, or one TYPE comment. UTF-8 throughout. Lines separated by a single LF (\n); the response must end with a newline.
3. HELP and TYPE comments precede the samples they describe. HELP is a free-text description. TYPE is one of counter, gauge, histogram, summary, untyped, gaugehistogram, or (OpenMetrics) info, stateset, unknown.
4. Labels are key/value pairs in curly braces, names match [a-zA-Z_][a-zA-Z0-9_]*, values are arbitrary UTF-8 strings with backslash, double-quote, and newline escaped as \\, \", and \n. Label values may not be empty.
5. Sample values are floats. +Inf, -Inf, and NaN are permitted in the wire format but Prometheus treats them as missing data; emitting them is the most common contract drift.
6. Help, type, and label names live in a per-metric preamble that reappears if and only if the metric family reappears in the body. The preamble is for humans, not for the parser.
7. OpenMetrics negotiation. If the client sends Accept: application/openmetrics-text; version=1.0.0 the exporter MAY reply with OpenMetrics (content type application/openmetrics-text; version=1.0.0; charset=utf-8). The two formats differ: OpenMetrics ends with # EOF, expresses counter and gauge differently, and adds _created series for counters.

That is the whole contract. The strength of the design is that every exporter speaks the same wire format. The weakness is that drift on any single rule is silently absorbed.

Why a sysadmin cares

The contract is what makes the platform pull-based. If the contract breaks, Prometheus stops ingesting data without stopping the scrape, and you discover the break when an alert fires on stale data hours later.

Three production failure shapes appear over and over:

• The single-line NaN. The exporter emits +Inf for a CPU metric because the kernel returned a divide-by-zero. Prometheus drops the sample. The rate query returns no data. The investigation loses hours.
• The metric-without-preamble. A custom exporter omits HELP and TYPE. Grafana auto-completes the metric in queries but cannot infer the unit. A dashboard panel showing mysterious_metric_bytes plots a dimensionless number and the on-call misreads a 1000-fold gap.
• The contract drift on upgrade. A exporter moves from text/plain to OpenMetrics without negotiating. Prometheus parses what it can, drops the rest, and the team gets a partial ingest with no alert on partial.

All three are operationally invisible until a dashboard or alert fails. Catching them is a routine exercise, not a heroic one.

How it works

The mental model is a small DSL over HTTP:

HTTP request: GET /metrics
              Accept: text/plain;version=0.0.4;charset=utf-8
                          -- or --
              Accept: application/openmetrics-text;version=1.0.0

HTTP response:
  Content-Type matches the chosen format
  Body is a UTF-8 byte stream of HELP / TYPE / sample lines
  Terminated by a final newline (text) or "# EOF" (OpenMetrics)

A real scrape of node_exporter looks like this:

# HELP node_cpu_seconds_total Seconds the CPUs spent in each mode.
# TYPE node_cpu_seconds_total counter
node_cpu_seconds_total{cpu="0",mode="idle"} 53281.46
node_cpu_seconds_total{cpu="0",mode="system"} 134.71
node_cpu_seconds_total{cpu="0",mode="user"} 902.18
# HELP node_memory_MemAvailable_bytes Memory information field MemAvailable_bytes.
# TYPE node_memory_MemAvailable_bytes gauge
node_memory_MemAvailable_bytes 1.73832192e+10
# HELP node_exporter_build_info A metric with a constant 1 value labeled by version.
# TYPE node_exporter_build_info gauge
node_exporter_build_info{version="1.8.2",branch="HEAD",revision="abc1234"} 1

Two contracts are visible in 11 lines:

• # HELP and # TYPE introduce each family.
• Each metric line is one sample; the same family can reappear with different label values many times.
• The exporter labels itself with a build-info gauge; that is the recommended way to expose the exporter’s own version.

The exposition parser is in github.com/prometheus/common/expfmt in Go, and the same parser drives the official Python, Java, and Rust client libraries’ text_string_to_metric_families / parse_text / parse helpers. A reference Python parser lives in prometheus_client.parser.

How to configure it

The contract is implemented by the exporter code, but it is configured in three places: the exporter’s own flags, the Prometheus scrape config, and any reverse proxy or service mesh in front of the exporter.

1. Bind interface. Every official exporter exposes a --web.listen-address flag. The default is typically :9100 or :9100/metrics depending on the binary. Bind to a specific address in production:

# /etc/default/prometheus-node-exporter
ARGS="--web.listen-address=127.0.0.1:9100"

The security lesson in this module returns to binding. For now, note that binding to 0.0.0.0 is a frequent, unreviewed default that exposes every node metric to every host on the network.

2. Path and content negotiation. Most exporters expose /metrics (text) and may expose the same path with OpenMetrics content negotiation. A scrape config that wants OpenMetrics sets the Accept header on its own; Prometheus itself does not currently request OpenMetrics by default. A reverse proxy in front of the exporter must not rewrite the content type or collapse the response:

# /etc/nginx/conf.d/exporter.conf — strip Accept rewriting,
# preserve Content-Type, never gzip (the parser wants raw bytes)
location /metrics {
  proxy_pass http://127.0.0.1:9100/metrics;
  proxy_set_header Accept "text/plain;version=0.0.4";
  proxy_pass_request_headers off;
  gzip off;
}

3. Per-scrape limits. Even when the contract is respected, labels can blow up cardinality. The scrape config puts a ceiling on what is accepted:

# prometheus.yml
scrape_configs:
  - job_name: node
    static_configs:
      - targets: ['10.0.1.4:9100', '10.0.1.5:9100']
    sample_limit: 10000           # max samples accepted per scrape
    label_limit: 30               # max labels per metric
    label_name_length_limit: 120
    label_value_length_limit: 512
    scrape_interval: 15s
    scrape_timeout: 10s

A breach fails the scrape loudly (up flips) instead of silently enlarging the TSDB.

How to validate it

Confirm the contract is being held end-to-end. All commands are READ-ONLY.

# 1. Confirm the endpoint serves the format Prometheus expects.
#    Content-Type header is the contract.
curl -sI http://10.0.1.4:9100/metrics | head -5

HTTP/1.1 200 OK
Content-Type: text/plain; version=0.0.4; charset=utf-8
Date: Thu, 13 Aug 2026 19:42:11 GMT

# 2. Confirm the body has HELP and TYPE comments and parses.
curl -sf http://10.0.1.4:9100/metrics | head -25

# 3. Validate against the contract locally with promtool.
#    Pass the live URL via curl, then feed to promtool.
curl -sf http://10.0.1.4:9100/metrics > /tmp/node.prom
promtool check metrics /tmp/node.prom

node.prom: OK

# 4. Confirm Prometheus sees the target as up and is parsing
#    samples (not just connecting).
up{job="node", instance="10.0.1.4:9100"}

# 5. Confirm the family has a HELP / TYPE and the right kind
#    in the parsed body.
count by (__name__, type) ({__name__=~"node_cpu_seconds_total"})

# 6. Confirm OpenMetrics negotiation works if you intend to use it.
curl -sI -H 'Accept: application/openmetrics-text; version=1.0.0' \
  http://10.0.1.4:9100/metrics | head -5

HTTP/1.1 200 OK
Content-Type: application/openmetrics-text; version=1.0.0; charset=utf-8

The outputs confirm: the exporter speaks the right dialect, Prometheus is ingesting it, the metric has a defined type, and any reverse proxy in front of it is not rewriting the response into something the parser will reject.

How it can fail

Six specific failure modes, each with an observable symptom:

1. NaN / Inf in a counter. The exporter emits +Inf because a division ran into a divide-by-zero. Prometheus drops the sample; the counter has no samples at this scrape. Symptom: rate(metric_total[5m]) returns no data and prometheus_target_scrapes_sample_out_of_order_total is unchanged, but you can see the literal NaN on the /metrics endpoint and the metric is “missing” in Grafana. The on-call investigation has no idea why.
2. Counter resets swallowed. The exporter re-emits a counter starting at zero between scrapes because the underlying process restarted but the client library forgot to wire the restart into Prometheus’s counter-reset detection. Symptom: alerts on rate() fire spuriously at every restart; the dashboard shows a saw-tooth pattern; process_start_time_seconds for the underlying service ticks.
3. HELP and TYPE omitted on a custom exporter. The metric appears in Prometheus but Grafana cannot infer the unit. Symptom: the dashboard panel renders but the legend shows raw floats; alerting rules accept it but unit-aware queries (rate() on bytes vs requests) silently produce nonsense.
4. Content-type rewritten by a proxy. A reverse proxy gzips the response or rewrites the Content-Type header to application/octet-stream. Symptom: scrapes succeed in transport but parse errors appear in the Prometheus logs (expfmt: ... parse error); up stays 1, samples are silently dropped.
5. OpenMetrics EOF missing. The exporter speaks OpenMetrics but the proxy strips the trailing # EOF line. Symptom: the parser accepts the partial body but rejects later _created series, so counters emitted as OpenMetrics are partially ingested.
6. Exit code on curl non-zero. An exporter process is alive but its handler panics and returns HTTP 500. Prometheus marks the scrape failed; up flips to 0. Symptom: alert up == 0 fires; the exporter log contains a stack trace around the handler for /metrics.

How to troubleshoot it

Diagnose in this order; it is cheapest to confirm the boundary first and the body second.

1. Is the endpoint reachable? curl -I first. If the network path is broken, no contract parse helps.
2. Is the body valid? Pipe to promtool check metrics. The parser reports the exact line number and the rule it broke.
3. Is Prometheus parsing the body? Look for scrape_errors in the scrape manager log (prometheus_tsdb_compactions_total does not help here — the right counters are prometheus_target_scrapes_exceeded_sample_limit_total and prometheus_target_scrape_pool_exceeded_label_limits_total).
4. Is the metric missing in queries but not in the body? Confirm the HELP/TYPE preamble is present and that the _total suffix convention is honoured if your dashboards rely on it.
5. Is the counter restarting? Compare the value across two scrapes; a drop to zero on a stable host is a contract violation unless the underlying process actually restarted.
6. Is a proxy in the way? Bypass the proxy temporarily (curl 127.0.0.1:9100) and compare the body byte-for-byte against the proxied response.

Security implications

The contract itself is public — there is no signature, no token, no encryption beyond what TLS provides. Two security properties follow:

• Transport is not contract. An exporter listening on HTTP with no auth still satisfies the contract and Prometheus will scrape it. The contract does not say the endpoint should be public; that is a configuration decision the operator must make deliberately. The security lesson in this module covers binding interface, auth, and TLS termination in depth.
• Label values are data. Anything that influences a label value — a header, a path parameter, an unauthenticated query string — flows into the TSDB and any downstream remote-write receiver. The contract permits arbitrary UTF-8 label values; the security boundary is whether an untrusted actor can influence them.

Default configurations often expose more than is appropriate: the Prometheus API is unprotected by default, the Grafana UI is anonymous-admin by default, and exporters typically bind to 0.0.0.0. Each lesson in this module revisits one of these.

Performance implications

Performance implications of the contract are mostly about the size of the response and the rate at which it must be regenerated:

• Response size is the sum of metric name length plus label count plus value length across all families. A healthy node_exporter exposes ~700 distinct metric names and ~5-15k samples per scrape. A misbehaving exporter exposes millions and crushes both the exporter and the scraper.
• Scrape interval is the smallest denominator for collection cost. A 5s interval on 1000 targets is 200 scrapes per second per Prometheus; the scraping budget is the size of the response times the scrape rate times the number of replicas.
• NaN/Inf handling is cheap on the wire but expensive in lost data: a single NaN on a counter breaks rate aggregation for the whole metric.

The trade-off of a permissive contract is that any process can emit Prometheus metrics cheaply. The cost is that every process can emit Prometheus metrics cheaply, including ones that should not.

Production guidance

• Pin the exporter version. Record the version in node_exporter_build_info and alert on unexpected changes.
• Treat the contract as the source of truth. When the body disagrees with the contract, fix the exporter, not the scraper.
• Run promtool check metrics against a live /metrics endpoint at least weekly. The check is cheap and catches drift.
• Confirm the Content-Type header at least once per scrape config. If you negotiated OpenMetrics, your dashboards and alerts depend on it.
• Use sample_limit and label_limit per scrape config. Better a failed scrape than a runaway series count.

Verification

You should now be able to answer:

• What are the seven rules of the text-based exposition format?
• What is the difference between a counter and a gauge in the parser’s view, and what does it cost to omit the TYPE line?
• How does Prometheus react to NaN, +Inf, and -Inf in a sample?
• What is content negotiation, and what changes when an exporter speaks OpenMetrics instead of text?
• What is the cheapest test you can run to confirm a target is satisfying the contract end-to-end?

Quiz