Skip to main content
RunBook Academy

← All break/fix scenarios in Observability

intermediateprometheus-rules~20 min

Break/Fix: PromQL Returns No Data

Reported symptoms

  • ●Prometheus is running
  • ●Metric should exist

Evidence

  • · Query returns empty result
  • · Metric is present in /api/v1/series
Diagnosis and resolutionclick to reveal

Root cause

A label matcher is wrong or the query is in the wrong form.

Remediation

1. Reproduce the query in the expression browser. 2. Compare to /api/v1/series. 3. Fix the matcher.

Verification

Query returns expected results.

Prevention

Test queries in a non-production environment.

Scenario

You are operating a production observability platform. The following symptoms appear:

  • Prometheus is running
  • Metric should exist

Available evidence:

  • Query returns empty result
  • Metric is present in /api/v1/series

Your task

Determine the cause, recover, document, and validate.

Investigation

The investigation follows the discipline taught in Part XCVIII:

  1. Form hypothesis, find evidence, test, validate.
  2. Use the available evidence above to bound the search.
  3. Reach one of the likely root causes.

Recovery procedure

(Do not reveal until you have reasoned through the problem.)

  1. Identify the failing component.
  2. Apply the remediation pathway.
  3. Validate with the verification step.
  4. Document the incident.

Remediation

  1. Reproduce the query in the expression browser. 2. Compare to /api/v1/series. 3. Fix the matcher.

Verification

Query returns expected results.

Rollback

Revert the query change.

Prevention

Test queries in a non-production environment.