Exporter Auth

A team runs the standard Prometheus stack: a node_exporter on every host, a mysqld_exporter on every database, a blackbox_exporter for synthetic probes. They bind each exporter to the host’s primary interface because that is the default. The hosts sit in a VPC with a permissive security group left over from staging. A vulnerability scanner from a friendly neighbour finds the node_exporter ports, scrapes the node_filesystem_* and node_network_* series, and learns the hostname, IP, and disk layout of every host in the estate. The credentials are not in the exporter metrics, but the attacker’s next step is trivial: the same VPC subnet is where Redis lives without authentication.

Exporters are the most-exposed surface in the Prometheus ecosystem. They are numerous, distributed, and usually left with their default configuration. This lesson is about authentication and TLS on the exporter side, and the right posture per exporter.

What it is

A Prometheus exporter is an HTTP service that exposes /metrics in the Prometheus text exposition format. The exporter speaks plain HTTP (or HTTPS with node_exporter’s web configuration); Prometheus is the only intended caller. The exporter-side authentication and TLS controls the outbound request that Prometheus makes to the exporter.

The four shapes an exporter can take:

1. No auth, plaintext HTTP. The exporter listens on 0.0.0.0:9100 over HTTP. Anyone who can reach the port reads /metrics. This is the default for most exporters in their default configuration.
2. Bind to loopback only. The exporter listens on 127.0.0.1:9100 over HTTP. Only the local host can reach it. Prometheus on the same host scrapes it directly; Prometheus on a peer host needs a port-forward or a sidecar.
3. TLS with server-cert validation. The exporter exposes HTTPS with a server certificate. Prometheus’s tls_config.ca_file validates the certificate. No client auth.
4. mTLS with basic auth. The exporter requires a valid client certificate (mTLS) and may also require basic auth on top. The strongest posture; the most operational cost.

   Prometheus                   exporter
   ---------    plaintext       ---------
     |          127.0.0.1:9100    |
     |--GET /metrics-------------->|
     |<--200 text/plain-----------|
     |                             |
   Prometheus                   exporter (TLS)
   ---------    HTTPS            ---------
     |          10.0.5.17:9100    |
     |--GET /metrics + Validate-->|
     |   server cert               |
     |<--200 text/plain-----------|
     |                             |
   Prometheus                   exporter (mTLS + basic)
   ---------    HTTPS            ---------
     |          10.0.5.17:9100    |
     |--GET /metrics + BasicAuth-->|
     |   + client cert             |
     |<--200 text/plain-----------|

The mental model: most production deployments land on shape 2 (loopback) or shape 3 (private network + TLS). Shape 4 (mTLS) is the right answer for high-trust environments; shape 1 (plaintext on a permissive bind) is the failure shape.

Why a sysadmin cares

The exporter surface is where the asymmetry bites. The Prometheus server is one process with a clear security posture; the exporters are tens or hundreds of processes, one per host or service, with whatever configuration the automation left behind. The most common exporter-side failure shapes in real incidents:

• Exporter bound to 0.0.0.0 on a permissive VPC. Every host on the subnet reads the exporter metrics. For node_exporter, this includes the host’s filesystem, network, and process inventory. For mysqld_exporter, this includes connection counts and replication state. The data is operational, not credentials, but it is a free reconnaissance map of the estate.
• Exporter with no auth behind a public-facing reverse proxy. The team puts nginx in front of node_exporter to share the metrics with a SaaS observability vendor. The proxy exposes /metrics to the internet. The vendor’s customer portal gets breached; the breach includes a list of every host in the team’s estate.
• Default exporter port reachable across an organisational boundary. A team’s Prometheus in AWS scrapes a partner’s exporters through a peering connection. The partner’s network team opens the exporter port for the peering CIDR. The peering expands to a second partner, who is less trusted. The exporter port is now reachable from a less trusted network.

The right answer is to keep exporters on a network that already restricts the audience (private VPC, loopback, a dedicated observability subnet) and to add TLS where the network is shared.

How it works

What exporters support

Most exporters in the Prometheus ecosystem share a common shape: a small Go binary that serves /metrics over HTTP. The exporter’s web configuration is implemented per-binary.

node_exporter is the canonical case. Its web configuration (loaded via --web.config.file) supports:

• basic_auth_users — bcrypt hashes for HTTP basic auth.
• tls_server_config — server certificate, key, optional client CA for mTLS, minimum TLS version.

Other exporters in the Prometheus project that follow the same shape:

• mysqld_exporter, postgres_exporter, redis_exporter, rabbitmq_exporter, nginx_exporter, and most of the “Prometheus project” exporters.
• The blackbox exporter supports the same web config shape.

Exporters that do not support the web configuration (or support only a subset):

• snmp_exporter does not implement --web.config.file in the same way; community modules add basic auth through sidecars.
• jmx_exporter (the Java agent) supports basic auth through a different configuration surface (--web.config.file does work for the standalone Java agent, but the in-process agent is configured via the application’s own YAML).
• Custom exporters built with the Prometheus client library opt into TLS and auth only if the developer added them.

The right production posture:

The asymmetry is intentional. A node_exporter scraping the same host as Prometheus is on the same trust boundary; the bind to loopback is sufficient. A SaaS vendor endpoint is on a different trust boundary; authentication and TLS are required.

The exporter’s web configuration

For exporters that support it (node_exporter and the project exporters), the web configuration is the same shape as the Prometheus web configuration:

# /etc/prometheus/node-exporter.yml
basic_auth_users:
  prometheus: $2y$10$bcrypt-hash-here

tls_server_config:
  cert_file: /etc/prometheus/node-exporter.crt
  key_file: /etc/prometheus/node-exporter.key
  client_ca_file: /etc/prometheus/internal-ca.crt
  client_auth_type: RequireAndVerifyClientCert
  min_version: TLS12

The exporter is started with:

node_exporter \
  --web.listen-address=10.20.5.17:9100 \
  --web.config.file=/etc/prometheus/node-exporter.yml

The Prometheus scrape config then validates the server certificate and presents a client certificate:

scrape_configs:
  - job_name: 'node'
    scheme: https
    static_configs:
      - targets: ['node-1.internal:9100']
    basic_auth:
      username: prometheus
      password_file: /etc/prometheus/secrets/node-exporter.pass
    tls_config:
      ca_file: /etc/prometheus/ca/internal-ca.crt
      cert_file: /etc/prometheus/client.crt
      key_file: /etc/prometheus/client.key
      server_name: node-1.internal
      min_version: TLS12

How to configure it

Bind to loopback (the most common posture)

For exporters that run on the same host as Prometheus or that are scraped through a local sidecar.

node_exporter \
  --web.listen-address=127.0.0.1:9100 \
  --collector.filesystem.mount-points-exclude='^/(sys|proc|dev|host|etc)($$|/)'

No TLS. No auth. The bind address is the only control. It is sufficient when nothing outside the host can reach the port.

Bind to a private address with TLS

For exporters that need to be reachable from peer subnets (dedicated Prometheus hosts scraping a fleet of exporters).

node_exporter \
  --web.listen-address=10.20.5.17:9100 \
  --web.config.file=/etc/prometheus/node-exporter.yml

# /etc/prometheus/node-exporter.yml
tls_server_config:
  cert_file: /etc/prometheus/node-exporter.crt
  key_file: /etc/prometheus/node-exporter.key
  min_version: TLS12

The Prometheus scrape config adds tls_config.ca_file to validate the server certificate (lesson 03).

Add basic auth

For environments where the exporter is on a shared network and there is a need to filter the audience.

htpasswd -nB prometheus
# New password: ********
# Re-type new password: ********
# prometheus:$2y$10$...

# /etc/prometheus/node-exporter.yml
basic_auth_users:
  prometheus: $2y$10$bcrypt-hash-here

tls_server_config:
  cert_file: /etc/prometheus/node-exporter.crt
  key_file: /etc/prometheus/node-exporter.key
  min_version: TLS12

Add mTLS

For high-trust environments where the threat model demands client authentication.

# /etc/prometheus/node-exporter.yml
tls_server_config:
  cert_file: /etc/prometheus/node-exporter.crt
  key_file: /etc/prometheus/node-exporter.key
  client_ca_file: /etc/prometheus/internal-ca.crt
  client_auth_type: RequireAndVerifyClientCert
  min_version: TLS12

The Prometheus scrape config adds cert_file and key_file under tls_config (lesson 03).

Scrape target without exporter-side auth support

For exporters that do not implement --web.config.file (snmp_exporter, some third-party exporters), the right posture is the bind address plus a sidecar that terminates auth:

# The exporter binds to loopback only.
snmp_exporter --web.listen-address=127.0.0.1:9116

# A local Caddy terminates auth in front of the exporter.
# /etc/caddy/caddyfile
:9117 {
  basicauth {
    prometheus $2a$14$bcrypt-hash-here
  }
  reverse_proxy 127.0.0.1:9116
}

The Prometheus scrape config targets localhost:9117 over HTTPS with basic auth. The exporter itself is unreachable without going through Caddy.

How to validate it

# READ-ONLY: confirm the exporter is bound to the expected interface.
sudo ss -tlnp | grep -E ':9100|:9116|:9104'
# LISTEN 0  4096  127.0.0.1:9100  ...  users:(("node_exporter",pid=...))
# A bind on 0.0.0.0:9100 would show 0.0.0.0:9100 or *:9100. That is the wrong answer.

# READ-ONLY: confirm the exporter answers locally.
curl -fsS http://127.0.0.1:9100/metrics | head -3
# # HELP go_gc_duration_seconds A summary of the GC invocation durations.
# # TYPE go_gc_duration_seconds summary
# go_gc_duration_seconds{quantile="0"} 1.23e-05

# READ-ONLY: confirm a peer that should NOT be able to reach the exporter.
curl -s --connect-timeout 3 http://node-1.internal:9100/metrics \
  || echo "OK: peer blocked"
# curl: (7) Failed to connect to node-1.internal port 9100: Connection timed out
# OK: peer blocked

# READ-ONLY: confirm TLS validation on the scrape.
curl -fsS https://node-1.internal:9100/metrics | head -3
# (200 OK if the cert is valid; 403 or TLS error otherwise)

# READ-ONLY: confirm basic auth is required when configured.
curl -s -o /dev/null -w '%{http_code}\n' https://node-1.internal:9100/metrics
# 401

# READ-ONLY: confirm basic auth succeeds.
curl -fsS -u prometheus:$PASS https://node-1.internal:9100/metrics | head -3
# (200 OK)

# READ-ONLY: confirm mTLS handshake succeeds from the Prometheus side.
openssl s_client -connect node-1.internal:9100 \
  -cert /etc/prometheus/client.crt \
  -key /etc/prometheus/client.key \
  -CAfile /etc/prometheus/ca/internal-ca.crt \
  -servername node-1.internal \
  </dev/null 2>&1 | grep -E 'Verify return code|subject='
# subject=CN = node-1.internal
# Verify return code: 0 (ok)

A clean validation: the bind is to loopback or a private address, the unauthorised peer cannot reach the exporter, the TLS handshake returns Verify return code: 0 (ok), and basic auth is enforced when configured.

How it can fail

The five exporter-side failure modes from real incidents.

1. Default 0.0.0.0:9100 left in production. Every host on the subnet reads /metrics. The symptom is nmap -p 9100 <subnet> finding open ports and a follow-up curl /metrics returning host inventory.
2. TLS misconfigured with insecure_skip_verify: true on the Prometheus side. A real certificate rotation produced x509: certificate signed by unknown authority; the team added insecure_skip_verify: true to clear the alert. The connection is now unauthenticated.
3. Basic auth credential shared with the scrape config in Git. The same credential is in prometheus.yml and on the exporter. The team rotates the exporter credential and forgets to update the scrape config. The symptom is last error: 401 Unauthorized on the affected target.
4. mTLS client certificate expires. The Prometheus server presents an expired certificate. The exporter rejects every scrape. The symptom is last error: tls: failed to verify client certificate on every target.
5. Sidecar auth proxy restarted and the upstream 127.0.0.1:<port> is bound to a new process that does not require auth. A Caddy reload left the upstream exporter reachable without basic auth. The symptom is a peer that can reach the proxy and skip the auth check.

How to troubleshoot it

Diagnostic order: what address did the exporter bind, who can reach it, what does the TLS / auth error actually say.

1. Check the bind.

   # Substitute your own value before running: the port the
   # exporter listens on (9100 for node_exporter).
   EXPORTER_PORT=9100
   
   sudo ss -tlnp | grep ":$EXPORTER_PORT"

   127.0.0.1:<port> is the right answer; 0.0.0.0:<port> is too wide.
2. Check the firewall. From a peer that should not be able to reach the exporter, attempt a connection. A successful TCP handshake at all is the problem.
3. Check the scrape error. The targets page shows the exact error from the exporter side: 401 Unauthorized, x509: certificate signed by unknown authority, tls: failed to verify client certificate. Each error has a specific fix.
4. Reproduce with openssl s_client. The TLS handshake chain is visible in the verbose output. The Verify return code line tells you what step failed.
5. Reload carefully. A SIGHUP reloads the web configuration on exporters that support it; otherwise a restart is required. (CONFIGURATION.)

Security implications

The exporter surface has three properties that make authentication harder than it sounds:

• Asymmetric trust. Prometheus is a single process that can be configured carefully; exporters are many processes, often managed by the same automation that deploys the application they monitor. A configuration drift in the automation exposes every exporter.
• Limited auth support. Not every exporter implements --web.config.file. For exporters that do not, the only posture is bind address plus sidecar.
• Reconnaissance value. Exporter metrics are not credentials, but they enumerate the estate: hostnames, ports, filesystems, network interfaces. The value of collecting them is the value of knowing the shape of the target.

The right security posture combines all four controls from this module:

1. Bind address (lesson 01): loopback or private interface.
2. Authentication (lesson 02): basic_auth where the exporter supports it; sidecar where it does not.
3. TLS (lesson 03): server-cert validation; mTLS where the threat model demands it.
4. Admin API (lesson 04): disabled on every Prometheus that scrapes a fleet of exporters, to keep the scrape configuration out of the response of an /api/v1/status/config call.

Verification

You should now be able to answer:

• Why is binding the exporter to loopback often sufficient even without authentication?
• Which exporters implement --web.config.file and which do not?
• What is the right posture for an exporter that does not support basic auth?
• How does mTLS differ from “TLS with basic auth on top”?

Quiz