Network Exposure

The node_exporter on the database host is reachable from every host on the production network. The Loki ingester is reachable from every host on the management network. The Prometheus admin API is reachable from every host on the monitoring VLAN. The Grafana is reachable from every host that has a route to the reverse proxy. The “everything can reach everything” posture is the default of a stack that was assembled from docker-compose examples and systemd unit files. The default works until a single host is compromised. Then every endpoint in the stack is one curl away.

This is what the word exposure means in an observability context: the set of network paths that can reach a component. The lesson is about the boundary that decides what each component listens on, and what reaches it.

What it is

Network exposure is the set of (interface, port, protocol) tuples that a component is reachable on. The exposure surface for the reference architecture in this course is roughly:

   Component            | Default bind        | Production bind
   ---------------------+---------------------+---------------------------------
   Prometheus           | 0.0.0.0:9090        | <monitoring-iface>:9090
   Alertmanager         | 0.0.0.0:9093        | <monitoring-iface>:9093
   node_exporter        | 0.0.0.0:9100        | <monitoring-iface>:9100
   Pushgateway          | 0.0.0.0:9091        | 127.0.0.1:9091 or <vpn>:9091
   Loki (all-in-one)    | 0.0.0.0:3100        | <monitoring-iface>:3100
   Loki querier/ingester| 0.0.0.0:3100,9096   | <monitoring-iface> per role
   Tempo                | 0.0.0.0:3200,4317   | <monitoring-iface>:3200
   Tempo OTLP           | 0.0.0.0:4317,4318   | <collector-iface>:4317
   OTel Collector       | 0.0.0.0:4317,4318   | <collector-iface>:4317
   Grafana Alloy        | 0.0.0.0:12345       | 127.0.0.1:12345
   Grafana              | 0.0.0.0:3000        | 127.0.0.1:3000
   Grafana (via proxy)  | 0.0.0.0:443 (nginx) | public interface

The default bind for every component is 0.0.0.0 (or [::] for the IPv6 side). The default is wrong. The right bind is the specific interface that the component should be reachable on; the rest is firewall.

Why a sysadmin cares

Three production failure modes map directly to permissive network exposure.

1. The exporter on the wrong interface. A node_exporter that binds 0.0.0.0:9100 exposes host metrics, including filesystem mounts and process names, to every host on the same network. The blast radius is the host enumeration of every database and every application server that the observability stack observes.
2. The data-plane API on the public network. A Loki ingester or a Tempo OTLP receiver reachable from outside the monitoring network accepts writes from anyone. The blast radius is the storage budget and the alert noise.
3. The admin API on the same interface as the read API. A Prometheus that exposes /api/v1/admin/* on the same interface as /api/v1/query lets any caller snapshot the TSDB or delete series. The blast radius is the observability data of every service.

How it works

The exposure surface is shaped by three layers, in order.

   Layer                | What it does                     | What fails when wrong
   ---------------------+----------------------------------+-------------------------
   1. Component bind    | Restrict the listening interface | All interfaces reachable
   2. Host firewall     | Drop traffic not on the bind     | Bypass via shared net
   3. Network segment   | Isolate the monitoring VLAN      | Cross-segment traffic

Layer 1 is the cheapest. Layer 2 is the canonical defence. Layer 3 is the architectural control. A production stack has all three in place; the absence of any one of them is a finding.

                         Public Internet
                                |
                                v
                          +-----------+
                          |   nginx   |   :443
                          +-----+-----+
                                |
                          ------+------ monitoring VLAN (10.0.10.0/24)
                                |
       +----------------+-------+-------+----------------+
       |                |               |                |
   Prometheus       Loki ingester    Tempo OTLP       Grafana Alloy
   10.0.10.5:9090   10.0.10.6:3100   10.0.10.7:4317   10.0.10.x:12345
       |                |               |                |
       +-------+--------+-------+-------+--------+-------+
               |                |                |
          app network      app network      app network
          10.0.20.0/24     10.0.20.0/24     10.0.20.0/24
               |                |                |
               v                v                v
          scrape targets   log push targets  trace push targets
          (loopback)       (loopback)        (loopback)

The monitoring VLAN is reachable from the application network only on the data-plane ports (9090 for Prometheus, 3100 for Loki, 4317 for Tempo). The application network never sees the admin API, the storage backends, or the secrets manager.

How to configure it

Prometheus

# /etc/prometheus/prometheus.yml
# The web.listen-address flag in the systemd unit does the bind.
# /etc/systemd/system/prometheus.service
[Service]
ExecStart=/usr/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --web.listen-address=10.0.10.5:9090 \
  --web.external-url=https://prometheus.internal.example.com

node_exporter

# /etc/systemd/system/node_exporter.service
[Service]
ExecStart=/usr/bin/node_exporter \
  --web.listen-address=10.0.10.5:9100 \
  --collector.filesystem.mount-points-exclude=^/(sys|proc|dev)($|/)

For the database host, the bind is the monitoring VLAN interface, not the database network interface. The exporter never reaches the database network; the database network reaches the exporter on the monitoring interface.

Loki

# /etc/loki/loki-config.yaml
server:
  http_listen_address: 10.0.10.6:3100
  grpc_listen_address: 10.0.10.6:9096
  log_level: info

In microservices mode, the querier, ingester, distributor, and compactor all have their own http_listen_address; the bind is per component.

Tempo

# /etc/tempo/tempo.yaml
server:
  http_listen_address: 10.0.10.7:3200
  grpc_listen_address: 10.0.10.7:9095

# The OTLP receiver binds a separate address.
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 10.0.10.7:4317
      http:
        endpoint: 10.0.10.7:4318

OpenTelemetry Collector

# /etc/otelcol/config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 10.0.10.20:4317
      http:
        endpoint: 10.0.10.20:4318

exporters:
  prometheusremotewrite:
    endpoint: http://10.0.10.5:9090/api/v1/write

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlp/tempo]

Grafana Alloy

# /etc/alloy/config.alloy
http_listen_addr = "127.0.0.1:12345"

prometheus.scrape "default" {
  targets = [{
    __address__ = "10.0.10.5:9100",
    job         = "node",
  }]
  forward_to = [prometheus.remote_write.default.receiver]
}

prometheus.remote_write "default" {
  endpoint {
    url = "http://10.0.10.5:9090/api/v1/write"
  }
}

The Alloy HTTP listener is for the debug UI and the /metrics endpoint; bind to loopback.

Grafana

# /etc/grafana/grafana.ini
[server]
http_addr = 127.0.0.1
http_port = 3000

Grafana is reachable on the loopback only. The reverse proxy speaks to it on 127.0.0.1:3000; nothing on the network reaches it directly.

Firewall rules

# CONFIGURATION: iptables baseline on the monitoring host.
# Only the monitoring VLAN may reach the data-plane ports.
iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -p tcp -s 10.0.10.0/24 --dport 9090 -j ACCEPT
iptables -A INPUT -p tcp -s 10.0.10.0/24 --dport 3100 -j ACCEPT
iptables -A INPUT -p tcp -s 10.0.10.0/24 --dport 4317 -j ACCEPT
iptables -A INPUT -p tcp -s 10.0.20.0/24 --dport 9090 -j ACCEPT
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -j DROP

The application network reaches Prometheus on :9090 for scrapes; the monitoring VLAN reaches every component for operator access; everything else is dropped.

How to validate it

# Substitute your own values before running:
PUBLIC_ADDR=203.0.113.10      # the address on this host's public interface
APP_HOST=192.0.2.20           # a host on the application network
MONITORING_HOST=192.0.2.10    # the monitoring host being scanned

# READ-ONLY: confirm every component binds the expected interface.
ss -tlnp | grep -E ':9090|:9091|:9093|:9100|:3100|:9095|:9096|:3200|:4317|:4318|:12345|:3000'
# LISTEN 0  4096  10.0.10.5:9090   prometheus
# LISTEN 0  4096  10.0.10.5:9100   node_exporter
# LISTEN 0  4096  10.0.10.6:3100   loki
# LISTEN 0  4096  10.0.10.7:3200   tempo
# LISTEN 0  4096  10.0.10.7:4317   tempo
# LISTEN 0  4096  127.0.0.1:12345  alloy
# LISTEN 0  4096  127.0.0.1:3000   grafana

# READ-ONLY: confirm Prometheus is not on the public network.
curl -fsS --max-time 3 "http://$PUBLIC_ADDR:9090/api/v1/query?query=up"
# curl: (7) Failed to connect

# READ-ONLY: confirm Loki is not on the public network.
curl -fsS --max-time 3 "http://$PUBLIC_ADDR:3100/ready"
# curl: (7) Failed to connect

# READ-ONLY: scan the public network for the data-plane ports.
nmap -p 9090,9100,3100,3200,4317,4318,12345 "$PUBLIC_ADDR"
# Not shown: 9090, 9100, 3100, 3200, 4317, 4318 closed

# READ-ONLY: confirm the loopback bind is on Grafana.
ss -tlnp | grep ':3000'
# LISTEN 0  4096  127.0.0.1:3000   grafana

# CONFIGURATION: scan from outside the monitoring VLAN.
ssh "$APP_HOST" "nmap -p 9090,3100,4317 $MONITORING_HOST"
# 9090/tcp  open   http        # expected: app host scrapes Prometheus
# 3100/tcp  closed
# 4317/tcp  closed

A clean validation: every component binds the expected interface, the data-plane ports are closed on the public interface, the Grafana is loopback-only, and the application network can scrape Prometheus but cannot reach Loki or Tempo directly.

How it can fail

The high-frequency network-exposure failure modes from real incidents.

1. Docker default 0.0.0.0 publish. A docker-compose file uses "3100:3100" instead of "10.0.10.6:3100:3100". The Loki ingester is now reachable on every interface of the host. The visible symptom is ss -tlnp on the Loki host showing [::]:3100 instead of 10.0.10.6:3100.
2. node_exporter copied from a public gist. The systemd unit uses --web.listen-address=:9100. The exporter is reachable on every interface of every host that runs it. The visible symptom is node_network_* metrics in the production Prometheus that show traffic from networks that are not part of the architecture.
3. Kubernetes Service of type LoadBalancer for Loki. The cloud-provider load balancer exposes the Loki ingester on a public IP. The visible symptom is a Shodan alert for :3100 on the cluster’s egress IP.
4. IPv6 wildcard missed. An operator binds the IPv4 interface (--web.listen-address=10.0.10.5:9090) but the component also listens on [::]:9090 by default on the IPv6 stack. The visible symptom is ss -tlnp showing both 10.0.10.5:9090 and [::]:9090; the wildcard IPv6 listener is reachable from any host with IPv6 routing.
5. Firewall rule on the wrong interface. An iptables rule accepts traffic from 10.0.10.0/24 to :9090 but does not limit the inbound interface. A compromised host on a different VLAN spoofs the source address. The visible symptom is journalctl -k showing traffic from 10.0.10.x that did not arrive on the monitoring interface.
6. Pushgateway on 0.0.0.0:9091. The pushgateway is the only Prometheus component designed to receive writes, and a permissive bind turns it into a metrics-injection surface. The visible symptom is up{job="pushgateway"} == 1 for a job label that no internal service owns.

How to troubleshoot it

The diagnostic order is “what is the component listening on?”, “what is the host firewall doing?”, “what can the network reach?”.

1. Re-read ss -tlnp for every component on every host. The wildcard binds (0.0.0.0, [::], [::]:port) are findings.
2. Re-read iptables -L INPUT -n -v on every host. The default policy should be DROP; the explicit accepts should match the bind addresses.
3. Re-read docker ps and the compose files. A published port without a host interface publishes to 0.0.0.0.
4. Scan from the application network with nmap. The application network should reach only the data-plane ports it has a reason to reach.
5. Scan from the public network with nmap. No observability component should respond.
6. Inspect the cloud-provider load balancers for LoadBalancer-type services. A Loki or Tempo service of this type is a finding.

Security implications

• Bind is the cheapest control. A bind is a one-line change in the systemd unit, the docker-compose file, or the Helm chart. It costs nothing at runtime.
• Bind is not sufficient. A bind does not authenticate; it does not authorise; it does not encrypt. The bind is the first layer of defence-in-depth, not the only layer.
• Firewall rules are the canonical control. A host firewall that drops everything by default and accepts only the expected source/destination pairs is the production baseline.
• Network segmentation is the architectural control. A monitoring VLAN that is reachable only from the application network and the operator bastion is the long-term answer.

Performance implications

• Bind has no performance cost. The bind is a kernel filter that rejects traffic before the application sees it.
• Firewall has a small cost. iptables on a modern kernel with connection tracking is microseconds per packet; the cost is invisible at typical observability traffic volumes.
• Network segmentation may add latency. A monitoring VLAN that traverses a router or a VPC peering has the latency of the routing hop. The hop is small (sub-millisecond on a managed VPC peering) and stable.

Production guidance

• Bind every component to the specific interface that should reach it. The default 0.0.0.0 is for the developer trying the component for the first time; production has a different answer.
• Drop everything by default on the host firewall. Accept only the source/destination pairs the architecture requires.
• Segment the monitoring VLAN from the application network and the public network. Reachability is the architectural decision; the firewall is the enforcement.
• Audit the bind, the firewall, and the segmentation at least once per quarter. A stack that was correctly configured at install time is not necessarily correctly configured after three application-team handoffs.
• Pay attention to IPv6. The default IPv6 wildcard bind is the same exposure as the default IPv4 wildcard bind.

Verification

You should now be able to answer:

• What is the default listening interface for Prometheus, Loki, Tempo and Grafana, and what is the right production bind for each?
• Why is a node_exporter on 0.0.0.0 the highest-volume network exposure in a typical install?
• What is the difference between the bind, the firewall, and the network segmentation, and which layer is the cheapest?
• Why does a docker-compose file with ports: "3100:3100" expose Loki on every interface, and what is the right alternative?
• Why is IPv6 wildcard bind a finding even when the IPv4 bind is restricted?

Quiz