Skip to main content
RunBook Academy

← All break/fix scenarios in Observability

intermediateprometheus-rules~25 min

Break/Fix: Counter Reset Causes Misleading Query

Reported symptoms

  • ●Counter with a reset occurred
  • ●rate() used at a window that straddles the reset

Evidence

  • · rate() returns a sudden spike
  • · Counter restarted during the window
Diagnosis and resolutionclick to reveal

Root cause

A counter reset during a rate window creates a misleading spike.

Remediation

1. Identify the metric. 2. Increase counter or use reset-aware rate. 3. Validate.

Verification

Rate returns expected values across the reset.

Prevention

Reset-aware rate settings are part of the rule conventions.

Scenario

You are operating a production observability platform. The following symptoms appear:

  • Counter with a reset occurred
  • rate() used at a window that straddles the reset

Available evidence:

  • rate() returns a sudden spike
  • Counter restarted during the window

Your task

Determine the cause, recover, document, and validate.

Investigation

The investigation follows the discipline taught in Part XCVIII:

  1. Form hypothesis, find evidence, test, validate.
  2. Use the available evidence above to bound the search.
  3. Reach one of the likely root causes.

Recovery procedure

(Do not reveal until you have reasoned through the problem.)

  1. Identify the failing component.
  2. Apply the remediation pathway.
  3. Validate with the verification step.
  4. Document the incident.

Remediation

  1. Identify the metric. 2. Increase counter or use reset-aware rate. 3. Validate.

Verification

Rate returns expected values across the reset.

Rollback

Revert rate change

Prevention

Reset-aware rate settings are part of the rule conventions.