Skip to main content
RunBook Academy

← All break/fix scenarios in Observability

advancedtime-skew~30 min

Break/Fix: Time Skew Corrupts Correlation

Reported symptoms

  • ●Logs from different services have wrong ordering

Evidence

  • · Trace spans appear out of order
Diagnosis and resolutionclick to reveal

Root cause

Clocks between hosts have drifted.

Remediation

1. Verify NTP. 2. Force a chrony step. 3. Validate.

Verification

Clocks are in sync; spans order correctly.

Prevention

chrony / systemd-timesyncd alerts on NTP offset

Scenario

You are operating a production observability platform. The following symptoms appear:

  • Logs from different services have wrong ordering

Available evidence:

  • Trace spans appear out of order

Your task

Determine the cause, recover, document, and validate.

Investigation

The investigation follows the discipline taught in Part XCVIII:

  1. Form hypothesis, find evidence, test, validate.
  2. Use the available evidence above to bound the search.
  3. Reach one of the likely root causes.

Recovery procedure

(Do not reveal until you have reasoned through the problem.)

  1. Identify the failing component.
  2. Apply the remediation pathway.
  3. Validate with the verification step.
  4. Document the incident.

Remediation

  1. Verify NTP. 2. Force a chrony step. 3. Validate.

Verification

Clocks are in sync; spans order correctly.

Rollback

Revert

Prevention

chrony / systemd-timesyncd alerts on NTP offset