Network Block

A platform team rolls out a new VPC peering connection between the observability subnet and the production subnet. The peering is correct, the route tables look right, and the security groups are unchanged. Within fifteen minutes every Prometheus target in production is up == 0 with lastError: context deadline exceeded. The exporter hosts are healthy; the path between Prometheus and the exporters is silently dropping SYNs. The team spent an hour checking the exporters before someone ran mtr from the Prometheus host and saw the packets die at a transit gateway.

The network block is the second-most-common missing-metric cause. It is more expensive to diagnose than “exporter down” because the evidence is in the path, not the endpoint, and the path is usually owned by a different team. The discipline is the same: walk the chain, identify the link, and fix the link without touching the others.

What it is

A network block is the condition where the TCP connection between Prometheus and the exporter cannot complete, despite both endpoints being healthy in isolation. The block can occur at any hop between the Prometheus host and the exporter host: the local iptables, the security group, the route table, the VPC peering, the NAT gateway, the on-prem firewall, the kernel conntrack table.

Two distinct failure signatures appear in the scrape log:

• Connection refused. The TCP SYN received a RST in response. The exporter host is reachable; the port is not open (or not bound by the expected process). The kernel knows the host is up and rejects the connect.
• Connection timeout. The TCP SYN did not receive a response within scrape_timeout. The host is either unreachable (no route) or the path is silently dropping packets (firewall, SG with no rule, conntrack exhaustion). The kernel never receives a response.

The two signatures are different diagnostics. Connection refused points at the exporter host; connection timeout points at the path. Treating one as the other is the most common diagnostic mistake.

Why a sysadmin cares

The Prometheus scrape path is a network path that crosses team boundaries: the platform team owns the Prometheus server, the application team owns the exporter host, the network team owns the routing and the firewall. A block that the platform team cannot resolve alone is a block that takes an hour of cross-team paging to clear. The discipline that prevents the hour is the diagnostic that names the failure as “path” rather than “endpoint” before any team is paged.

Three production pains follow:

1. Cross-team time-to-diagnose. A network block is often diagnosed by the wrong team first, because the symptom looks like an exporter failure to the platform team and like a peering failure to the network team. The discipline that names the failure correctly is the discipline that pages the right team first.
2. Flapping connectivity. A NetworkPolicy that allows the path most of the time but blocks it during a particular kubectl rollout produces flapping up values. The investigation reads as random; the cause is the policy’s interaction with the rollout.
3. Conntrack exhaustion. A high-fanout scrape across many short-lived connections exhausts the conntrack table on a busy firewall. Symptom: a fraction of the fleet is up == 0, not the whole fleet. The fix is connection-tracking tuning or persistent connections.

How it works

A TCP connection between Prometheus and the exporter traverses several hops. Each hop can accept, reject, or silently drop the SYN. The observable signature differs by hop behaviour:

  prom-host:9090  ---(SYN)--->  hop-1 (SG / firewall)
                                     |
                              accept | reject | drop
                                     |
                                     v
                                  hop-2 (router / VPC)
                                     |
                              accept | reject | drop
                                     |
                                     v
                                  hop-3 (NAT / proxy)
                                     |
                                     v
                               exporter-host:9100
                                     |
                              accept | reject | drop
                                     |
                                     v
                                  LISTEN / RST / silent

• Accept at every hop produces a connection. The exporter responds.
• Reject at any hop produces a RST. The Prometheus host sees connection refused.
• Drop at any hop produces silence. The Prometheus host sees context deadline exceeded after scrape_timeout seconds.

A firewall that drops packets is more dangerous than a firewall that rejects them: the drop is invisible to the operator without a probe. The probe is mtr or tcpdump, neither of which is in the default Prometheus diagnostic.

Under the hood

How to configure it

The right approach is to configure the path from both ends and verify it from a third vantage point. Three examples.

AWS security groups

# terraform - the exporter security group
resource "aws_security_group_rule" "prom_to_exporter" {
  type                     = "ingress"
  security_group_id        = aws_security_group.exporter.id
  source_security_group_id = aws_security_group.prometheus.id
  from_port                = 9100
  to_port                  = 9100
  protocol                 = "tcp"
  description              = "Scrape from Prometheus only"
}

The source_security_group_id field is the rule that allows the Prometheus host’s SG. Cross-VPC peering requires the peer SG ID; cross-account requires aws_security_group_rule with a cidr_blocks field pointing at the peered VPC’s CIDR.

Kubernetes NetworkPolicy

# exporter-netpol.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-prom-scrape
  namespace: monitoring
spec:
  podSelector:
    matchLabels: { app: node-exporter }
  policyTypes: [Ingress]
  ingress:
    - from:
        - podSelector:
            matchLabels: { app: prometheus }
      ports:
        - protocol: TCP
          port: 9100

The podSelector on the exporter matches the pods; the from.podSelector matches the Prometheus pod. The policy allows the scrape path and nothing else. Any pod not matched by the from selector is denied; any pod not matched by the podSelector on the policy itself is unaffected.

Linux iptables on the Prometheus host

The Prometheus host usually allows egress by default. The rules to watch are the ones on the exporter host and any transit host (NAT gateway, firewall VM). The diagnostic is to inspect, not to add rules. Adding egress rules on the Prometheus host is rarely the fix; the fix is usually on the exporter side.

# READ-ONLY: confirm the rule that allows the scrape
iptables -L INPUT -n -v --line-numbers | grep 9100
# expected (healthy): a rule accepting tcp dpt:9100 from
#   the Prometheus subnet (or "anywhere" with rate limiting)

# READ-ONLY: confirm no rule is dropping the path
iptables -L INPUT -n -v --line-numbers | grep -E 'DROP|REJECT'

How to validate it

The diagnostic ladder for a network block. Each step is read-only.

# Step 1: confirm the signature from Prometheus
curl -s 'http://prom:9090/api/v1/targets?state=active' \
  | jq '.data.activeTargets[]
        | select(.health=="down")
        | {instance: .labels.instance, lastError: .lastError}'
# connection refused       -> exporter host port is closed
#                             (link 1, not this lesson)
# context deadline exceeded -> path is dropping packets
# no such host              -> DNS is failing (different lesson)

If the lastError is context deadline exceeded, continue:

# Step 2: confirm the exporter host is reachable from Prometheus
curl --connect-timeout 5 -v http://node-7.internal:9100/metrics 2>&1 | head
# expected (healthy): TCP connect completes, HTTP 200 returned
# expected (blocked): curl hangs at "Trying X.X.X.X..." and times out

# Step 3: confirm the TCP port specifically (faster than curl)
nc -vz -w 5 node-7.internal 9100
# expected (healthy): Connection to node-7.internal 9100 port [tcp/*] succeeded
# expected (blocked): nc: connect to node-7.internal port 9100 (tcp) timed out

# Step 4: trace the path
mtr -rwc 20 node-7.internal
# expected (healthy): 0% loss at every hop
# expected (blocked): loss starts at the hop that drops the SYN

# Step 5: confirm the firewall state on the exporter host
ssh node-7.internal 'iptables -L INPUT -n -v | head -30'
ssh node-7.internal 'ss -tlnp | grep :9100'
# the port must be in LISTEN; the firewall must accept the
# Prometheus source

A scrape that times out at step 2 and produces 0% loss at step 4 to the exporter IP but 100% loss to the exporter port is a security group issue. A scrape that times out at step 2 and produces loss at step 4 is a routing issue. Walk the ladder.

How it can fail

Six failure shapes appear repeatedly:

1. Firewall silent drop. The firewall has no rule allowing the scrape path and the default policy is DROP. Symptom: context deadline exceeded; the exporter host is healthy; nc -vz from the Prometheus host times out; mtr shows loss at the firewall hop. The fix is to add an allow rule from the Prometheus source to the exporter port.
2. Security group too narrow. The SG allows the scrape port but not from the Prometheus source (the source is 0.0.0.0/0 and the SG was tightened, or the source is a peered VPC’s CIDR and the peering was removed). Symptom: identical to a firewall silent drop from the Prometheus host’s perspective. The fix is to update the SG source.
3. Missing NAT. The exporter is on a private subnet whose egress is supposed to be NAT’d; the NAT rule was removed or the route table was updated to bypass it. Symptom: the Prometheus host can reach the exporter IP on some ports but not on the scrape port. The fix is to restore the NAT rule or to add a dedicated scrape entry point.
4. Kubernetes NetworkPolicy denial. A NetworkPolicy in the exporter namespace restricts ingress to a specific set of pod selectors, and the Prometheus pod is not matched (label drift, namespace change, deployment rename). Symptom: a subset of the fleet is up == 0 corresponding to the affected namespace. The fix is to update the policy’s from.podSelector.
5. MTU mismatch. A GRE or IPSec tunnel between the Prometheus subnet and the exporter subnet has an MTU of 1500 but the path requires 1500 minus encapsulation overhead. The SYN at MSS 1460 is fragmented and dropped by a df-bit rule somewhere in the path. Symptom: the connection fails after the SYN-ACK; tcpdump shows large packets being dropped. The fix is to lower the MTU on the Prometheus host’s interface or to enable PMTUD.
6. Conntrack table exhaustion. A high-fanout scrape exhausts the firewall’s conntrack table. Symptom: a fraction of the fleet is up == 0, randomly distributed; conntrack -L | wc -l is at the table max. The fix is to tune nf_conntrack_max, to use persistent connections (Prometheus 2.55 supports keepalive scrapes via the OpenMetrics scraper), or to spread the scrape across multiple Prometheus replicas.

How to troubleshoot it

Security implications

The Prometheus scrape path is read access to operational data that is sensitive in production: kernel version, mount points, process arguments, file descriptors. The path should be restricted to the Prometheus host and the on-call jump box.

• Bind the exporter to the Prometheus network, not 0.0.0.0 (lesson 02 covers this).
• Restrict the path at the firewall: only the Prometheus source subnet, only the scrape port, only the protocol.
• Audit the SG and NetworkPolicy changes; a permissive SG is the most common attack surface for an exporter endpoint.
• Use mTLS between Prometheus and the exporter (lesson 04 covers the configuration). The block is then defense in depth: even a compromised Prometheus host cannot scrape exporters in a network it is not authenticated for.

Performance implications

The network block does not affect Prometheus performance directly; it removes the data. The performance traps are elsewhere:

• Conntrack table size. A scrape fleet larger than the conntrack table produces random blocks. Tune the table or spread the scrape.
• Persistent connections. Prometheus 2.55 keeps the underlying TCP connection open with HTTP keep-alive. A scrape that closes the connection (an exporter behind a proxy that closes after every response) defeats the optimisation and inflates the conntrack cost.
• Cross-region latency. A Prometheus server scraping across a region boundary sees latencies in the hundreds of milliseconds; scrape_timeout of 10 seconds leaves headroom but produces slow scrapes under packet loss. Local Prometheus per region, then federation, is the right architecture for multi-region fleets.

Production guidance

• Co-locate Prometheus and the exporter fleet in the same subnet (or VPC, or cluster) to avoid cross-team network dependencies.
• Run a synthetic scrape from a separate vantage point (a jump box, a CI runner) every minute. The synthetic scrape catches a network block before the per-target alerts do.
• Page on up == 0 for: 2m with a team: network label when the lastError is context deadline exceeded. The label routes the page to the right team.
• Document the scrape path in the runbook: which SG, which NetworkPolicy, which firewall rule. The on-call engineer at 02:14 should be able to find the rule without grepping the whole repo.
• Use OpenMetrics with persistent connections on bandwidth-constrained paths; the negotiation cost is paid once per connection, not once per scrape.
• Test firewall and SG changes in a staging environment with the same Prometheus scrape configuration. The block appears at the same place it will in production.

Verification

You should now be able to answer:

• What is the difference between connection refused and context deadline exceeded in the scrape log, and what does each one point at?
• Which two read-only commands from the Prometheus host are the highest-signal diagnostics for a network block?
• What is the difference between an AWS security group rule with cidr_blocks and one with source_security_group_id, and when does each apply?
• How do you recognise a conntrack exhaustion block, and what is the fix?
• Why is the change log the fastest first diagnostic when a block starts at a known time?

Quiz