Break/Fix: Prometheus Cannot Scrape Target

Reported symptoms

At 09:26 the on-call phone gets nine TargetDown alerts in one notification group. All nine are instances of the node job. The job has 41 targets; the other 32 are fine.

The first four things anyone checks all come back reassuring, which is what makes the next twenty minutes expensive:

• Are the hosts down? No. All nine are serving production traffic. Their application error rates are flat, their own logs show nothing at 09:22, and somebody is logged into two of them doing unrelated work.
• Is Prometheus broken? No. Readiness returns 200, prometheus_config_last_reload_successful is 1, and git log on the scrape configuration shows nothing for six weeks.
• Is the network down? No. ping from the Prometheus host reaches all nine with no loss.
• Is the data actually missing? This is where the call goes sideways. Somebody opens the fleet dashboard, sees CPU, memory and disk for all 41 hosts with no gap at 09:22, and says the alert is wrong.

Then the lastError column makes it worse. Six of the nine say context deadline exceeded. Three say connect: connection refused. Those are different failures with different causes - the course says so plainly: refused points at the endpoint, timeout points at the path. The call splits into two workstreams and starts paging two teams.

The change calendar has nothing in the hour the alerts fired. The only entry in 24 hours is a Prometheus hardware migration that completed, was verified, and was signed off at 18:10 the previous evening.

Evidence provided

$ curl -s 'http://prom-a.obs.example.com:9090/api/v1/targets?state=active' | jq -r '.data.activeTargets[] | select(.health=="down") | [.labels.instance, .lastError] | @tsv'

app-21.rz.example.com:9100   Get "http://app-21.rz.example.com:9100/metrics": context deadline exceeded
app-22.rz.example.com:9100   Get "http://app-22.rz.example.com:9100/metrics": context deadline exceeded
app-23.rz.example.com:9100   Get "http://app-23.rz.example.com:9100/metrics": context deadline exceeded
app-24.rz.example.com:9100   Get "http://app-24.rz.example.com:9100/metrics": context deadline exceeded
app-25.rz.example.com:9100   Get "http://app-25.rz.example.com:9100/metrics": context deadline exceeded
app-26.rz.example.com:9100   Get "http://app-26.rz.example.com:9100/metrics": context deadline exceeded
app-27.rz.example.com:9100   dial tcp 203.0.113.27:9100: connect: connection refused
app-28.rz.example.com:9100   dial tcp 203.0.113.28:9100: connect: connection refused
app-29.rz.example.com:9100   dial tcp 203.0.113.29:9100: connect: connection refused

Ask each replica on its own, rather than asking the query layer in front of them:

# The two Prometheus replicas of the HA pair, queried directly.
for H in prom-a.obs.example.com prom-b.obs.example.com; do
  printf '%s ' "$H"
  curl -s -G "http://$H:9090/api/v1/query" \
    --data-urlencode 'query=count(up{job="node"} == 0)' \
    | jq -r '.data.result[0].value[1] // "0"'
done

Replica A answers 9. Replica B answers 0. Both servers have the same scrape configuration and the same target list.

$ curl -s -G 'http://prom-a.obs.example.com:9090/api/v1/query' --data-urlencode 'query=scrape_duration_seconds{job="node", instance=~"app-2[1-6].*"}' | jq -r '.data.result[] | [.metric.instance, .value[1]] | @tsv'

app-21.rz.example.com:9100   5
app-22.rz.example.com:9100   5
app-23.rz.example.com:9100   5
app-24.rz.example.com:9100   5
app-25.rz.example.com:9100   5
app-26.rz.example.com:9100   5

On the target hosts themselves, the exporter is fine:

$ ssh app-27.rz.example.com 'ss -tlnp | grep :9100; curl -s http://127.0.0.1:9100/metrics | head -2'

LISTEN 0  4096  *:9100  *:*  users:(("node_exporter",pid=1187,fd=3))
# HELP go_gc_duration_seconds A summary of the wall-time pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary

The same TCP probe, run from each replica in turn:

# Run this from prom-a, then from prom-b, against one host of each shape.
for T in app-21 app-27; do
  nc -vz -w 5 "$T.rz.example.com" 9100
done

From replica A, app-21 times out and app-27 is refused. From replica B, both succeed. The difference is not the target. The difference is who is asking.

$ git -C /srv/netops grep -nF '192.0.2.4' -- zones/restricted/

zones/restricted/objects.yml:14:      - 192.0.2.40   # prom-a, scrape source
zones/restricted/objects.yml:15:      - 192.0.2.41   # prom-b, scrape source

$ git -C /srv/platform log -p --since=2.days -- hosts/prom-a.obs.example.com/network.yml

commit 4c1e9a2  2026-08-12 18:04  migrate prom-a to new chassis
+  address: 198.51.100.40/24
 secondary:
+    - 192.0.2.40/24     # keep during cutover window

commit 9b73d05  2026-08-13 09:22  prom-a migration: release old address
 secondary:
-    - 192.0.2.40/24     # keep during cutover window

Work the evidence before reading on

Every host is up. Every exporter is listening. The network carries ICMP. Prometheus has not been reconfigured. And nine targets are down.

1. Replica A sees nine failures and replica B sees none, from identical configuration against identical targets. What is different about the two servers, and what does that difference have to do with a firewall?
2. You have two error strings and you are about to open two workstreams. Before you do: what does each string tell you about the device that produced it, and what does either one tell you about the cause?
3. The dashboard has data and the alert says the targets are down. Exactly one of those is lying. Which, and what mechanism makes the other one true?
4. The change calendar is empty for 09:22. The migration record is not.

Before continuing: if you had been asked at 09:30 to choose between opening the restricted zone by hand and waiting four hours for a change window, what would you have needed to know to make that call safely - and do you have it?

Root cause

1. The nine targets are the only ones with a source-address policy

The 41 targets in the node job are not policed the same way. Thirty-two of them sit in racks whose firewalls allow the whole site prefix to reach 9100, so the address a scrape arrives from is not part of their policy and cannot break. The nine that failed are the entire membership of the restricted zone, which allows a named list of scrape sources instead.

That list is a single shared address object rendered to both rack firewalls. It names 192.0.2.40 and 192.0.2.41. Replica A has been scraping from 198.51.100.40 since the previous evening.

This is why the nine hosts looked like they had nothing in common. They have nothing in common at the application layer, and they were not chosen by anything the platform team owns. They are a network policy group, and the only place that grouping is visible is in a repository the platform team does not read.

2. The change succeeded fourteen hours before it failed

The cutover was done carefully, which is what hid it. Replica A moved to new hardware and a new address at 18:04, and the change deliberately kept 192.0.2.40 bound as a secondary through the cutover window so that anything still expecting the old address would keep working. It did. Every target stayed green, the migration was verified, and the change was signed off.

At 09:22 the next morning the final task of that same change released the secondary address. From the next scrape onwards, replica A arrived at the restricted zone from a source nobody had allowed.

The change calendar was empty at 09:22 because the change had been closed the night before. The safety mechanism that made the cutover smooth is the same mechanism that moved the failure fourteen hours away from anything a responder would think to correlate with it.

3. One cause, two error strings

Six timeouts and three refusals is the signature that split the call into two workstreams. It should not have.

The restricted zone is fronted by two firewalls, one per rack, because the zone spans two racks and the platform replacement is half finished. Both render their scrape allowlist from the same address object, so both stopped allowing replica A at the same moment. They differ only in what they do with a packet they will not allow:

• The rack-A firewall is the older platform. Its default action is to drop silently. Replica A sends a SYN, nothing comes back, and after scrape_timeout seconds Prometheus records context deadline exceeded.
• The rack-B firewall was replaced last quarter with a ruleset whose house standard ends in an explicit reject. It answers the SYN with an ICMP port-unreachable, the kernel on replica A turns that into ECONNREFUSED, and Prometheus records connect: connection refused.

The error string names the behaviour of the device that stopped the packet. It does not name the cause. Reading it as “endpoint versus path” is correct as a first cut and wrong here, and the thing that resolves the ambiguity is cheap: the same probe from a second source. Replica B reached all nine. A failure that depends on who is asking is a policy failure, not an endpoint failure, whatever the error string says.

Resolution

1. Decide hold or fix, explicitly, and say the decision out loud on the call. Replica B is scraping all nine targets, so no data is being lost and no dashboard is wrong. What you have lost is redundancy on nine targets and the ability to trust replica A alerts. That is a real degradation, and it is also a defensible reason to hold rather than to open a restricted zone by hand at 09:30.
2. If you hold, hold properly: name an owner, set an end time, and state what changes the decision. "Hold until the 14:00 change window, owner is the platform on-call, escalate immediately if replica B degrades" is a hold. "We will look at it later" is not, and it is how nine targets stay single-homed for a fortnight.
3. Fix at the source of truth, not at the devices. Add replica A at 198.51.100.40 to the shared address object in zones/restricted/, and let both rack firewalls re-render from it. One edit, one review, two devices converge.
4. Do not fix the nine hosts or the two firewalls individually. You will fix the six that were paging loudest, miss the three that fail differently, and leave the source of truth still wrong so the next render undoes your work.
5. Do not widen the rule to the whole 198.51.100.0/24 prefix to make the page stop. That deletes the control the restricted zone exists to enforce, it is invisible in the diff to anyone who does not know what the zone is for, and nobody ever comes back to narrow it.
6. Do not re-add 192.0.2.40 to replica A. It makes the symptom vanish in one minute, resurrects a decommissioned address, and guarantees this same incident happens again the next time somebody tidies up - with the added difficulty that the address will by then be documented as released.
7. Close the loop on the migration change rather than filing a new one. The defect is not in the firewall; it is in a migration runbook that released an address without checking who was allowing it. Reopen that change record so the fix lands where the next migration will read it.

Verification

1. Ask both replicas directly. count(up{job="node"} == 0) must return zero on replica A and on replica B. Do not verify through the Thanos Querier: the deduplicating layer is the thing that hid the failure, and it will happily report success while replica A is still blind.
2. Require the zero to survive two scrape intervals, not one. A single successful scrape after a firewall change can be a rule that has not fully propagated to both devices.
3. Read lastError for all nine targets and require it to be empty, rather than reading health and requiring it to be up. An empty error is a stronger statement than a healthy status, and it is the field that will name a partial fix.
4. Confirm scrape_duration_seconds for the six formerly-timing-out targets has come off the 5s ceiling. A duration that has returned to normal proves the path is open; a status that flipped to up proves only that one scrape got through.
5. Prove the check can fail before you trust it passing. You already have the negative control - the same query returned 9 during the incident. Capture that value in the incident record. If you did not capture it, produce one deliberately against a scratch target rather than closing on an untested green.
6. Trigger a fresh render of both rack firewalls and re-check everything above. A rule that exists on a device but not in the repository is erased by the next convergence run, and that run will happen at a time nobody is watching.
7. Confirm the restricted zone still denies what it is supposed to deny. Probe 9100 on one of the nine hosts from a source that is not an allowed scrape source and require it to fail. Adding an allow rule under time pressure is exactly when a prefix gets widened by accident.

Prevention

• Enumerate every place a Prometheus server is named as a source before you move it: firewall address objects, security groups, network policies, exporter allowlists, proxy configurations. The scrape source address is a dependency of the server in the same way its storage volume is, and it is the one nobody inventories.
• Test from the new address during the cutover window, while the old one is still bound. That window exists precisely so that a mistake is recoverable, and this incident wasted it by treating “everything is still green” as evidence rather than as the expected consequence of the old address still being live.
• Make releasing the old address a separate, gated step with somebody watching, not the last unattended task of a change that has already been declared successful. The riskiest moment of that migration happened fourteen hours after anyone was paying attention to it.
• Build platform-health views on per-replica data. Give the dashboard a replica template variable, or pin it to a single server. Deduplication is correct for application dashboards and actively harmful for the dashboard that is supposed to tell you whether the platform is degraded.
• Alert on up == 0 per replica, and add absent(up{job="node"}) next to it. The failure one step sideways from this one - discovery or a relabel rule removing the targets entirely - produces no up series at all, and therefore no TargetDown alert, and therefore no page.
• Keep one source of truth for the scrape allowlist. This incident was survivable in ten minutes because there was exactly one object to change. The same estate with nine hand-maintained rulesets is a day of work and a near-certainty that one host is missed.