Skip to main content
RunBook Academy

← All break/fix scenarios in Observability

advancedobservability-of-observability~30 min

Break/Fix: Observability Stack Loses a Node

Reported symptoms

  • ●One Prometheus / Loki / Tempo node is down

Evidence

  • · Scrape fails for that node
  • · Cluster metrics show the down state
Diagnosis and resolutionclick to reveal

Root cause

A node has failed.

Remediation

1. Verify node is down. 2. Investigate. 3. Restart on a healthy host.

Verification

Stack is back to nominal.

Prevention

HA from the start

Scenario

You are operating a production observability platform. The following symptoms appear:

  • One Prometheus / Loki / Tempo node is down

Available evidence:

  • Scrape fails for that node
  • Cluster metrics show the down state

Your task

Determine the cause, recover, document, and validate.

Investigation

The investigation follows the discipline taught in Part XCVIII:

  1. Form hypothesis, find evidence, test, validate.
  2. Use the available evidence above to bound the search.
  3. Reach one of the likely root causes.

Recovery procedure

(Do not reveal until you have reasoned through the problem.)

  1. Identify the failing component.
  2. Apply the remediation pathway.
  3. Validate with the verification step.
  4. Document the incident.

Remediation

  1. Verify node is down. 2. Investigate. 3. Restart on a healthy host.

Verification

Stack is back to nominal.

Rollback

Revert

Prevention

HA from the start