← All runbooks in Observability
Runbook: Investigate a Missing Prometheus Target
1 · Prerequisites
Confirm every item is in place before any state change.
- Prometheus server
- Target that should be scraping
2 · Pre-checks
Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.
- · Confirm the target exists
- · Check the recent change log
- · Verify the target is reachable from Prometheus
3 · Procedure
Execute each step in order. Verify the expected output of a step before moving to the next.
- 1Inspect /api/v1/targets for the affected job
- 2Look for lastError
- 3If lastError is empty: check scrape config and reload Prometheus
- 4If lastError shows connection refused: check target is running
- 5If lastError shows timeout: check network and firewall
- 6If lastError shows no such host: check DNS
4 · Verification
Confirm the procedure actually fixed the problem.
- ✓up{job=...} == 1
- ✓The target shows lastScrape and lastError is empty
- ✓The metric from the target appears in queries
5 · Rollback
If verification fails, undo the procedure in reverse order.
- ↶Revert the configuration change
- ↶Restore the target host if it was decommissioned
6 · Escalation
When the runbook isn't enough, contact:
- · Escalate if the target is critical infrastructure
- · Coordinate with the team owning the target
Purpose
Investigate a Missing Prometheus Target
When to use this runbook
Use this runbook when the operator needs a guided procedure to handle the situation described above.
Pre-checks
Before starting the procedure, confirm the prerequisites and pre-checks are met. The structured lists are rendered from the frontmatter by the page layout.
Procedure
Follow the steps from the frontmatter procedure steps. The page layout renders the steps as a checklist with copy-to-clipboard affordances.
Verification
After the procedure, the structured verification items from the frontmatter are rendered as a checklist.
Rollback
If the procedure fails or makes things worse, follow the structured rollback steps from the frontmatter.
Escalation
The structured escalation path is rendered from the frontmatter. Use it if the operator cannot complete the procedure safely.