Skip to main content
RunBook Academy

← All break/fix scenarios in Observability

intermediateprometheus-scrape~20 min

Break/Fix: Relabel Drops Targets

Reported symptoms

  • ●Prometheus is running
  • ●Relabeling rules in configuration

Evidence

  • · /api/v1/targets shows the targets list as empty for the job
  • · No `lastError` because targets are not even being attempted
Diagnosis and resolutionclick to reveal

Root cause

A relabeling action removes the targets before scraping.

Remediation

1. Inspect /api/v1/targets. 2. Identify the relabeling rule. 3. Remove the `drop` or fix the regex. 4. Restore.

Verification

Targets reappear; metrics are scraped.

Prevention

Review the relabeling rules and how they were applied.

Scenario

You are operating a production observability platform. The following symptoms appear:

  • Prometheus is running
  • Relabeling rules in configuration

Available evidence:

  • /api/v1/targets shows the targets list as empty for the job
  • No lastError because targets are not even being attempted

Your task

Determine the cause, recover, document, and validate.

Investigation

The investigation follows the discipline taught in Part XCVIII:

  1. Form hypothesis, find evidence, test, validate.
  2. Use the available evidence above to bound the search.
  3. Reach one of the likely root causes.

Recovery procedure

(Do not reveal until you have reasoned through the problem.)

  1. Identify the failing component.
  2. Apply the remediation pathway.
  3. Validate with the verification step.
  4. Document the incident.

Remediation

  1. Inspect /api/v1/targets. 2. Identify the relabeling rule. 3. Remove the drop or fix the regex. 4. Restore.

Verification

Targets reappear; metrics are scraped.

Rollback

Revert configuration changes

Prevention

Review the relabeling rules and how they were applied.