Skip to main content
RunBook Academy

← All break/fix scenarios in Observability

intermediateprometheus-scrape~20 min

Break/Fix: Prometheus Cannot Scrape Target

Reported symptoms

  • ●Prometheus /-/ready returns 200
  • ●Target HTTP is refused or times out

Evidence

  • · /api/v1/targets shows the job as DOWN
  • · lastError is \"connection refused\" or \"context deadline exceeded\"
Diagnosis and resolutionclick to reveal

Root cause

A target is not reachable from Prometheus due to network or configuration issue.

Remediation

1. Confirm the target host is up. 2. Confirm /metrics endpoint listens. 3. Add the missing allow rule. 4. Restore scrape.

Verification

Target returns `up` == 1; /metrics appears in queries.

Prevention

Verify all external scrape targets are documented; review quarterly.

Scenario

You are operating a production observability platform. The following symptoms appear:

  • Prometheus /-/ready returns 200
  • Target HTTP is refused or times out

Available evidence:

  • /api/v1/targets shows the job as DOWN
  • lastError is “connection refused” or “context deadline exceeded”

Your task

Determine the cause, recover, document, and validate.

Investigation

The investigation follows the discipline taught in Part XCVIII:

  1. Form hypothesis, find evidence, test, validate.
  2. Use the available evidence above to bound the search.
  3. Reach one of the likely root causes.

Recovery procedure

(Do not reveal until you have reasoned through the problem.)

  1. Identify the failing component.
  2. Apply the remediation pathway.
  3. Validate with the verification step.
  4. Document the incident.

Remediation

  1. Confirm the target host is up. 2. Confirm /metrics endpoint listens. 3. Add the missing allow rule. 4. Restore scrape.

Verification

Target returns up == 1; /metrics appears in queries.

Rollback

Revert the network change in the controlled manner; document the rule.

Prevention

Verify all external scrape targets are documented; review quarterly.