Prometheus Network Exposure

A team runs a small Prometheus on a single VM to watch a few services. They bind Prometheus to 0.0.0.0:9090 because that is the default. The host sits in a VPC with a permissive security group left over from staging. Six weeks later, a vulnerability scanner from a friendly neighbour finds a Grafana CVE and an attacker pivots onto the same VPC subnet. They curl http://10.0.4.17:9090/api/v1/targets, see every scrape target, read internal service names, and from there reach Redis without authentication. Prometheus itself was not the vulnerable thing. The exposure was.

This lesson is about the network surface Prometheus presents and how to make that surface as small as the operational reality permits. The right bind address is the first decision; it sets the posture for every other security control in this module.

What it is

Prometheus 2.55.x exposes two HTTP listeners by default. Both present the same web UI, the same /api/v1/* query API, the same /metrics endpoint for self-monitoring, and the same /-/reload endpoint (when --web.enable-lifecycle is set). The two listeners are:

• --web.listen-address — the address and port Prometheus binds its own listener on. Default 0.0.0.0:9090. This is the process-level socket.
• --web.external-url — the URL Prometheus uses when it generates self-references in the UI, in alert template variables, and in the redirect that follows a successful reload. Default is the URL derived from --web.listen-address on the inbound request.

These are not the same setting. The first is where Prometheus binds. The second is what Prometheus believes its public URL is. They interact when a reverse proxy sits in front: Prometheus must be told that the public URL is https://prometheus.example.com even though it is bound to 127.0.0.1:9090.

                    Internet
                       |
                       v
                 +-----------+
                 |   nginx   |      TLS terminates here
                 +-----+-----+      :80 redirects to :443
                       |
                       v
                 +-----------+
                 | prometheus|      --web.listen-address=127.0.0.1:9090
                 +-----------+      --web.external-url=https://prometheus.example.com
                       ^
                       |
                  VPC firewall
                  restricts 9090 to peer subnets

The mental model: Prometheus binds to the loopback or to a private address, the firewall lets only the right peers talk to it, and any public exposure lives behind a TLS-terminating reverse proxy that Prometheus knows about via --web.external-url.

Why a sysadmin cares

The network surface decides four things:

• Who can read metrics. /api/v1/query_range returns every series Prometheus has stored. Some of those series contain instance IDs, internal service names, sometimes user IDs as labels. A Prometheus reachable from the internet is a search engine for the internal service map.
• Who can write metrics. Prometheus accepts remote write from any caller that can reach --web.listen-address and present the right credentials (see lesson 02). An exposed Prometheus is a remote-write target by default.
• Who can reload or stop it. --web.enable-lifecycle exposes /-/reload and /-/quit. Lesson 04 covers the controls, but the bind address decides the audience for them.
• What the URL Prometheus itself emits looks like. A Prometheus with --web.external-url=http://prom-internal:9090 behind a TLS proxy produces mixed-content warnings in the alert template variables and breaks webhook receivers that expect absolute URLs.

A wrong bind address is not an “exposure in theory” problem. It is the most common root cause of accidental Prometheus data disclosure in incident reports.

How it works

The Prometheus process opens a single TCP listener at --web.listen-address and serves the API, the UI, the self-/metrics endpoint, and the lifecycle endpoints from that socket. The bind happens during early start; a port already in use produces listen tcp 0.0.0.0:9090: bind: address already in use and the process exits 1.

For the UI and the API to produce absolute URLs that match the user’s browser, Prometheus needs to know the public URL. There are three inputs and they resolve in a documented order:

1. --web.external-url=<URL> if set. Used verbatim.
2. X-Forwarded-Proto and X-Forwarded-Host request headers, if the request comes from a peer in the configured --web.trusted-proxies CIDR list.
3. The request’s own scheme and Host header.

If the request reaches Prometheus directly (no proxy) and --web.external-url is unset, Prometheus answers with whatever URL the client asked for. This is fine inside a private VPC. It is a recipe for mixed-content warnings when a reverse proxy is in front and --web.external-url was forgotten.

How to configure it

The first decision is where Prometheus binds. Three reasonable production shapes.

Option 1: loopback only, reverse proxy in front

The most defensible posture. Prometheus listens on the loopback interface; nginx, Caddy, or HAProxy terminates TLS and forwards requests. Nothing on the host can reach Prometheus without talking to the proxy.

# /etc/default/prometheus  (systemd EnvironmentFile)
ARGS="--config.file=/etc/prometheus/prometheus.yml \
      --storage.tsdb.path=/var/lib/prometheus \
      --web.listen-address=127.0.0.1:9090 \
      --web.external-url=https://prometheus.example.com \
      --web.console.libraries=/usr/share/prometheus/console_libraries \
      --web.console.templates=/usr/share/prometheus/consoles"

The systemd unit then passes ARGS to the binary. The bind on 127.0.0.1 means nothing on the LAN can reach Prometheus even if a host firewall misconfiguration would otherwise allow it.

Option 2: private VPC interface, firewall restricts peers

When Prometheus needs to be reachable from peer subnets (a Grafana in a different subnet, a remote-write receiver in another VPC) without going through a reverse proxy, bind to a private interface and rely on the network controls.

ARGS="--config.file=/etc/prometheus/prometheus.yml \
      --storage.tsdb.path=/var/lib/prometheus \
      --web.listen-address=10.20.5.17:9090 \
      --web.external-url=https://prometheus.internal.example.com"

The host firewall or the VPC security group limits the source addresses that can reach 10.20.5.17:9090. Common shape: allow 10.20.0.0/16 (the Grafana subnet) and a remote-write receiver CIDR, deny everything else. The --web.external-url is still needed so absolute URLs are correct for in-cluster users.

Option 3: sidecar / containerised with a published port

In a container deployment, the bind address is the container address. The host port mapping in Docker or Kubernetes decides who can reach the process. In Kubernetes this typically becomes a ClusterIP service, not a NodePort; in Compose it is a `ports:

• “127.0.0.1:9090:9090”` mapping that keeps the service off the host network.

# docker-compose.yml  (excerpt)
services:
  prometheus:
    image: prom/prometheus:v2.55.1
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.path=/prometheus
      - --web.listen-address=0.0.0.0:9090
      - --web.external-url=https://prometheus.example.com
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus-data:/prometheus
    ports:
      - "127.0.0.1:9090:9090"   # bound to loopback on the host
    restart: unless-stopped

The 0.0.0.0:9090 inside the container binds to all container interfaces; the 127.0.0.1:9090:9090 mapping on the host keeps it off the LAN.

The --web.console.libraries interaction

Console templates (legacy HTML dashboards) generate absolute URLs from --web.external-url. If the operator still uses consoles (most production installs do not, but a few keep them for Solaris-style status pages), setting the wrong external URL produces broken console_public_url links.

# Required for the consoles to render useful URLs.
--web.external-url=https://prometheus.example.com
--web.console.libraries=/usr/share/prometheus/console_libraries
--web.console.templates=/usr/share/prometheus/consoles

How to validate it

Three read-only checks confirm the bind address is correct.

# READ-ONLY: confirm Prometheus is listening on the expected interface.
sudo ss -tlnp | grep -E ':9090|prometheus'
# LISTEN 0  4096  127.0.0.1:9090  ...  users:(("prometheus",pid=1821,...))
# A bind on 0.0.0.0 would show 0.0.0.0:9090 or *:9090. That is the wrong answer.

# READ-ONLY: confirm the external URL Prometheus uses internally.
curl -s http://127.0.0.1:9090/api/v1/status/runtimeinfo | jq
# {
#   "data": {
#     "GOGC": "100",
#     "GOMAXPROCS": "4",
#     "storageRetention": "15d",
#     ...
#   }
# }
# Note: the API does not echo --web.external-url directly. Check the
# rendered HTML for the base href, or look at a UI-relative redirect.
curl -sI http://127.0.0.1:9090/ | grep -i location
# (no redirect for the root path; this is informational only)

# READ-ONLY: from a peer that should NOT be able to reach it.
curl -s --connect-timeout 3 http://10.20.5.17:9090/api/v1/targets \
  || echo "OK: peer blocked"
# curl: (7) Failed to connect to 10.20.5.17 port 9090: Connection timed out
# OK: peer blocked

# READ-ONLY: from a peer that SHOULD be able to reach it (the proxy).
curl -fsS http://prometheus.internal.example.com/api/v1/targets | jq '.data.activeTargets | length'
# 47

A clean validation: the ss output shows 127.0.0.1:9090 or a private IP, never 0.0.0.0:9090; the unauthorised peer cannot connect; the authorised peer returns the right data.

How it can fail

Six failure modes from real Prometheus installs.

1. Default 0.0.0.0:9090 left in production. The bind happens. Every host on the subnet can reach Prometheus. The symptom is nmap -p 9090 <subnet> returning one or more open ports and a follow-up curl /api/v1/targets returning service inventory.
2. --web.external-url not set behind a reverse proxy. The Prometheus UI loads over TLS but every link in the UI points at http://prometheus.internal:9090/.... The symptom is mixed-content warnings in the browser console and broken links in alert template variables and email receivers.
3. --web.external-url set to the loopback URL. A copy-paste from --web.listen-address leaves the external URL as http://127.0.0.1:9090. The symptom is that external receivers (Slack webhook URLs in alert templates) point at the wrong host entirely.
4. Bind to a public IP. Either through --web.listen-address= <public-ip>:9090 or by accident via a misconfigured cloud security group. The symptom is that an external port scan discovers Prometheus and an unauthenticated caller can read /api/v1/targets.
5. Container port published on 0.0.0.0. The bind inside the container is 0.0.0.0:9090, the Docker Compose `ports:
   • “9090:9090”mapping is on0.0.0.0`. The host now publishes Prometheus on every interface. The symptom is the same as failure mode 1, but in a container context.
6. X-Forwarded-* headers trusted from any peer. When --web.trusted-proxies is unset or set to 0.0.0.0/0, Prometheus accepts the forwarded headers from anyone. The symptom is --web.external-url getting silently overridden on every request by the client’s chosen values.

How to troubleshoot it

The diagnostic order: what address did Prometheus actually bind, who can reach it, what URL is Prometheus telling its clients.

1. Check the bind. sudo ss -tlnp | grep 9090. If the output shows 0.0.0.0:9090 or *:9090, the bind is too wide. Restart with --web.listen-address set to a specific address. (SERVICE-IMPACT.)
2. Check the firewall. From a peer that should not be able to reach Prometheus, attempt a connection. A successful TCP handshake at all is the problem. Audit the security group / iptables / nftables rules. (READ-ONLY.)
3. Check the external URL. Open /api/v1/status/config from a trusted peer; the response includes the --web.config.file path if set, but not --web.external-url directly. Instead, check the rendered HTML:

   curl -s http://127.0.0.1:9090/graph | grep -oE 'href="[^"]+"' | head -5

   If the URLs reference 127.0.0.1:9090 and you reached the service via a proxy, --web.external-url is unset or wrong.
4. Check the proxy trust. A request reaching Prometheus through the proxy with --web.external-url correctly set still produces the wrong URL? Inspect the proxy’s X-Forwarded-Proto and X-Forwarded-Host headers. If --web.trusted-proxies is empty, Prometheus ignores them.
5. Check from outside. From a host on the same subnet that should not be able to reach Prometheus:

   # Substitute your own value before running: the address of
   # the host Prometheus runs on.
   PROM_HOST=192.0.2.20
   
   nmap -p 9090 "$PROM_HOST"

   If 9090/tcp open appears, the bind is too wide.

Security implications

The bind address is the floor of the Prometheus security model. Every control in the rest of this module (lesson 02 auth, lesson 03 TLS, lesson 04 admin API hardening) assumes an attacker cannot simply walk up to the port. If the port is on a public interface, none of those controls matter for an external adversary.

Two specific attack classes:

• Service inventory disclosure. /api/v1/targets and /api/v1/label/instance/values enumerate every endpoint Prometheus scrapes, by hostname, port, and labels. For a motivated attacker this is a free reconnaissance map of the internal estate.
• Relabelling as a write primitive. Prometheus’s relabel rules and metric relabel rules are visible at /api/v1/status/config. A leaked Prometheus is a leaked view of how every metric is shaped and dropped.

Performance implications

The bind address itself does not change performance. The trade-offs come from what is reachable:

• Loopback only. The fastest path. No network stack overhead, no firewall rules, no proxy. The cost is that everything that talks to Prometheus must live on the same host or reach the proxy.
• Private VPC interface. Network round trips for every API call. Acceptable for a Grafana in the same VPC, expensive for a remote operator every time the dashboard refreshes.
• Behind a reverse proxy. Adds a TLS termination and a hop. For Prometheus’s own /api/v1/query_range traffic (which can be heavy during a long time-range dashboard render), the proxy becomes a capacity-planning concern. See the nginx lesson for the relevant sizing notes.

The bind address is a posture decision, not a performance decision. Pick it on security grounds; size the proxy on performance grounds.

Verification

You should now be able to answer:

• What does --web.listen-address control, and what is its default?
• What does --web.external-url control, and why does it matter when a reverse proxy is in front?
• Why is a Prometheus bound to 0.0.0.0 in a permissive VPC a reconnaissance map for an attacker?
• What command shows the actual address Prometheus is bound to?

Quiz