Exporter Security

A scan of the production network turns up 1,200 hosts on which node_exporter is listening on 0.0.0.0:9100, with no authentication, no TLS, and no rate limiting. Every metric about every host — process command lines, filesystem mount points, environment variables from /proc/<pid>/environ — is readable by anyone who can reach port 9100. The monitoring team thought Prometheus was the only scraper. A bot from a penetration test reached the endpoint in five minutes.

The exporter’s listening interface is a security boundary. The exporter’s endpoint is a credential surface. The exporter’s labels are data. Each of these requires deliberate configuration.

What it is

Exporter security is the discipline of placing the exporter’s HTTP endpoint behind an authenticated, encrypted boundary, restricting the labels that reach storage, and ensuring the exporter does not expose information that should not leave the host.

Three boundaries matter:

• Network boundary. What can reach the exporter on its port. The default bind is 0.0.0.0, which means every host on the local network can scrape the exporter.
• Authentication boundary. What identity the scraper must present. Most exporters do not implement auth natively; the auth happens at a reverse proxy or service mesh.
• Label boundary. What values reach the TSDB. The contract permits any UTF-8 in label values; the security decision is what you put there.

A fourth boundary is implicit: the process privilege boundary, covered in the community-trust lesson in this module. The exporter runs as root on most deployments; that boundary is real but is configured at the systemd unit or container SecurityContext level, not at the HTTP level.

Why a sysadmin cares

The exporter endpoint exposes information about the host that is otherwise protected by the kernel’s permission model. Without a security boundary, the exporter becomes a way for anyone who can reach port 9100 to read /proc, /sys, and the network interfaces of the host.

The failure shapes:

• The exposed metrics. Process command lines that include database passwords. Filesystem mount points that include secret volumes. Network interface names that include internal subnet information.
• The sensitive labels. A custom exporter that emits a metric labelled by customer_email or api_key. The label is replicated into the TSDB, the remote-write receiver, and every backup.
• The topology leak. Labels that include IP addresses or internal host names let an attacker map the network from the metrics alone.

The trade-off: every security control you add is operational friction. The cost is real; the cost of the exposure is larger.

How it works

The mental model is layers of boundary, applied independently:

            +-------------------------+
            |   Production network    |
            +-------------------------+
                       |
                       v
            +-------------------------+
            |  Network ACL / firewall |   <-- only Prometheus subnet
            +-------------------------+
                       |
                       v
            +-------------------------+
            |  Reverse proxy / mesh   |   <-- TLS, basic auth or mTLS
            +-------------------------+
                       |
                       v
            +-------------------------+
            |  Exporter on localhost  |   <-- 127.0.0.1 only
            +-------------------------+
                       |
                       v
            +-------------------------+
            |  metric_relabel_configs |   <-- drop sensitive labels
            +-------------------------+
                       |
                       v
                  Prometheus TSDB

Each layer is independent. Removing the network ACL but keeping the reverse proxy still works; removing the reverse proxy but keeping the bind-to-localhost still works; removing the relabel rules but keeping the bind-to-localhost still works. The strongest posture has all three.

The reverse proxy layer is the most flexible. A nginx-ingress-controller in Kubernetes, an Envoy sidecar in a service mesh, or a simple nginx in front of a bare-metal exporter all provide TLS termination, basic auth, and request limiting. The exporter itself stays simple; the policy lives at the proxy.

The relabel layer is in the Prometheus scrape config. A metric_relabel_configs block drops or rewrites labels after parsing but before storage. Sensitive labels are removed here; topology labels that change routing are removed here; labels that bloat cardinality can also be removed here, though cardinality limits in sample_limit are the primary control.

How to configure it

Each boundary has a separate configuration. The three blocks below cover a production setup.

1. Bind to localhost on the exporter.

# /etc/default/prometheus-node-exporter
ARGS="--web.listen-address=127.0.0.1:9100 \
      --web.config.file=/etc/prometheus/exporter-web.yml"

The exporter now listens on loopback only. Anything that needs to scrape it must be on the same host, or must go through a reverse proxy that is.

2. Reverse proxy with TLS and basic auth.

# /etc/nginx/conf.d/exporter.conf
server {
  listen 9100 ssl;
  server_name _;

  ssl_certificate     /etc/nginx/tls/exporter.crt;
  ssl_certificate_key /etc/nginx/tls/exporter.key;
  ssl_protocols       TLSv1.2 TLSv1.3;
  ssl_ciphers         ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256;

  # Bind to localhost on the upstream side
  proxy_pass http://127.0.0.1:9100/metrics;

  # Basic auth via htpasswd
  auth_basic "Prometheus";
  auth_basic_user_file /etc/nginx/htpasswd/exporter.htpasswd;

  # Limit request rate
  limit_req zone=exporter burst=10 nodelay;

  access_log /var/log/nginx/exporter.access.log;
}

limit_req_zone $binary_remote_addr zone=exporter:10m rate=10r/s;

The proxy terminates TLS, requires basic auth, limits request rate, and forwards to the exporter on localhost. The exporter itself stays simple.

The Prometheus scrape config:

# prometheus.yml
scrape_configs:
  - job_name: node
    scheme: https
    basic_auth:
      username: prometheus
      password_file: /etc/prometheus/exporter-htpasswd
    tls_config:
      ca_file: /etc/prometheus/ca-bundle.crt
      server_name: node-exporter.internal
    static_configs:
      - targets:
          - node-01.internal:9100
          - node-02.internal:9100
    metric_relabel_configs:
      # Drop labels that may leak sensitive data
      - source_labels: [__name__]
        regex: 'node_exporter_build_info'
        action: drop
        # Build info is useful in dashboards but only the version
        # label is needed; the others are noise.
      - source_labels: [labelname]
        regex: 'cmdline|environ|secret|token|api_key|password'
        action: labeldrop
        # Any label matching the pattern is dropped before storage.
      - source_labels: [instance]
        regex: '(.*)\.internal'
        replacement: '$1'
        target_label: instance_short
        # Create a short instance label without the internal suffix.

The labeldrop action takes a regex and drops any label name that matches. Be careful: cmdline matches cmdline_arguments, kernel_cmdline, and any other label ending in cmdline if you do not anchor the regex with ^.

3. Service mesh with mTLS termination. In a Kubernetes cluster with a service mesh (Istio, Linkerd), the sidecar proxy terminates mTLS automatically. The exporter binds to localhost; the sidecar proxies traffic on the pod IP.

# exporter-pod.yaml (excerpt)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: node-exporter
spec:
  template:
    metadata:
      labels:
        app: node-exporter
      annotations:
        # Linkerd injects the proxy sidecar
        linkerd.io/inject: enabled
    spec:
      containers:
        - name: node-exporter
          image: prom/node-exporter:v1.8.2
          args:
            - --web.listen-address=127.0.0.1:9100
          ports:
            - containerPort: 9100
              name: metrics
          securityContext:
            runAsNonRoot: false   # node_exporter needs root for /proc
            readOnlyRootFilesystem: true
            allowPrivilegeEscalation: false

# ServiceMonitor for Prometheus Operator
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: node-exporter
  labels:
    app: node-exporter
spec:
  selector:
    matchLabels:
      app: node-exporter
  endpoints:
    - port: metrics
      scheme: https
      tlsConfig:
        # The mesh terminates mTLS; Prometheus talks plain HTTP
        # to the sidecar which proxies to the exporter.
        insecureSkipVerify: true
      metricRelabelings:
        - source_labels: [labelname]
          regex: 'cmdline|environ|secret|token|api_key|password'
          action: labeldrop

The mesh handles authentication between Prometheus and the exporter; the relabel rules drop sensitive labels.

4. Topology labels that change routing. Some labels reveal internal network topology that should not be in metrics: IP addresses, internal host names, or labels whose values change under autoscaling. Drop them at relabel time:

metric_relabel_configs:
  # Drop instance IPs; keep instance names.
  - source_labels: [instance]
    regex: '(\d+\.\d+\.\d+\.\d+):\d+'
    action: labeldrop
  # Drop labels that change with every pod restart.
  - regex: 'pod_ip|pod_uid|container_id'
    action: labeldrop

The trade-off: dropping these labels means you cannot pivot to that dimension. The discipline is to drop only what you would not want an attacker to see, not everything.

How to validate it

Validate each boundary independently. All commands are READ-ONLY.

# 1. Confirm the exporter is bound to localhost only.
ss -tlnp | grep 9100

LISTEN 0 128 127.0.0.1:9100 0.0.0.0:* users:(("node_exporter",pid=1234,fd=6))

Anything other than 127.0.0.1:9100 in the local address is a misconfiguration.

# 2. Confirm TLS is terminated at the proxy.
curl -sI https://node-01.internal:9100/metrics | head -5

HTTP/1.1 200 OK
Server: nginx
Content-Type: text/plain; version=0.0.4; charset=utf-8

# 3. Confirm auth is enforced.
curl -sI https://node-01.internal:9100/metrics | head -3

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Basic realm="Prometheus"

# 4. Confirm the contract still holds with TLS + auth.
# The password is whatever the scrape config's password_file holds.
EXPORTER_PASSWORD=$(sudo cat /etc/prometheus/exporter-htpasswd)

curl -sf -u "prometheus:$EXPORTER_PASSWORD" \
  https://node-01.internal:9100/metrics > /tmp/node.prom
promtool check metrics /tmp/node.prom

node.prom: OK

# 5. Confirm sensitive labels are dropped in Prometheus.
#    The cmdline label, if exposed by an exporter, should not
#    appear here.
{labelname=~".*cmdline.*"}

# 6. Confirm topology labels are dropped.
{labelname=~".*pod_ip.*"}

The outputs confirm: the exporter is loopback-only, TLS is terminated, auth is enforced, the contract holds, and sensitive labels are gone.

How it can fail

Five specific failure modes:

1. Exporter bound to 0.0.0.0 by default. A new deployment uses the package default instead of --web.listen-address. Symptom: ss -tlnp shows 0.0.0.0:9100; an external host can scrape /metrics without auth.
2. No TLS termination. Prometheus scrapes the exporter in plaintext over the internal network. Symptom: a packet capture on the path shows the metric body; the metrics are readable by anyone with network access to the route.
3. Sensitive label in storage. A custom exporter emits a customer_email label; the relabel rule was forgotten or matched the wrong name. Symptom: the label appears in PromQL autocomplete; a Grafana dashboard panel reveals the email address.
4. Topology label leaks routing. A label like pod_ip is on every series; an attacker maps the network from the metrics alone. Symptom: the label appears in metrics and the dashboard; the metric body reveals an internal subnet.
5. Auth credential rotated in the wrong place. Prometheus’s basic_auth.password_file is updated, but the htpasswd file on the proxy is not. Symptom: Prometheus scrape starts failing with 401 Unauthorized; up flips to 0.

How to troubleshoot it

Diagnose in this order; it is cheapest to confirm the network boundary first.

1. What address is the exporter bound to? ss -tlnp on the host. If the answer is 0.0.0.0, that is the bug.
2. What auth is the proxy enforcing? curl -I without credentials. If the response is 200 OK, auth is not configured.
3. What does the body contain? curl -sf with creds, grep for sensitive label names. If they appear, the relabel rule is missing or wrong.
4. Are labels surviving into the TSDB that should not? count by (__name__) ({labelname=~".*secret.*"}) in PromQL. A non-zero result is a leak.
5. Is TLS terminated correctly? openssl s_client -connect the endpoint; check the certificate chain and the cipher in use.

Security implications

The exporter’s security boundary is the most operationally important part of the metrics stack. The discipline:

• Bind to localhost by default. The exporter’s own process is the only thing that should reach /metrics directly.
• Terminate TLS at a proxy. TLS at the exporter is fine; TLS at a proxy gives you more control.
• Require basic auth or mTLS. The proxy is the right place for auth; the exporter should not implement its own.
• Drop sensitive labels at relabel time. Anything in a label value is replicated into the TSDB, remote-write receivers, and backups.
• Restrict outbound network. A compromised exporter should not phone home; NetworkPolicy or egress firewalling.
• Restrict process privileges. Run with minimum privileges where possible; node_exporter-class exporters need root because of /proc access.

Performance implications

Security and performance interact at the proxy. A reverse proxy in front of the exporter adds a small amount of latency per scrape (typically sub-millisecond) and uses CPU for TLS termination. The cost is small relative to the cost of the exporter itself; the operational benefit is large.

The trade-off: a service mesh adds a sidecar proxy to every pod. The proxy uses CPU and memory; in return, mTLS is automatic and the bind-address problem disappears.

Production guidance

• Bind every exporter to 127.0.0.1 or a private interface.
• Terminate TLS at a reverse proxy or service mesh. The exporter stays simple.
• Require basic auth or mTLS. The Prometheus basic_auth block in the scrape config uses password_file to keep the credential off the command line.
• Use metric_relabel_configs to drop sensitive labels before storage. Document the dropped labels in the instrumentation guide.
• Audit the bind address, TLS, and relabel rules on every exporter quarterly. The defaults shift on package upgrades.

Verification

You should now be able to answer:

• What are the four security boundaries around an exporter, and how are they configured independently?
• Why is 0.0.0.0 the wrong default bind address for a production exporter, and how do you fix it?
• How does metric_relabel_configs differ from the scrape relabel_configs, and when does each apply?
• What is the right pattern for integrating an exporter with a service mesh that terminates mTLS automatically?
• Which label names should always be dropped at relabel time, and what is the regex pattern?

Quiz