Skip to main content
RunBook Academy

← All break/fix scenarios in Observability

intermediateloki-ingestion~20 min

Break/Fix: Log Timestamp Problem

Reported symptoms

  • ●Loki queries return logs from wrong time window

Evidence

  • · Logs have wrong timestamps
Diagnosis and resolutionclick to reveal

Root cause

A service emits logs with local time or with wrong format.

Remediation

1. Identify the format. 2. Fix or use Loki pipeline_stages to parse.

Verification

Log timestamps align with the event time.

Prevention

Standardise: UTC, ISO 8601

Scenario

You are operating a production observability platform. The following symptoms appear:

  • Loki queries return logs from wrong time window

Available evidence:

  • Logs have wrong timestamps

Your task

Determine the cause, recover, document, and validate.

Investigation

The investigation follows the discipline taught in Part XCVIII:

  1. Form hypothesis, find evidence, test, validate.
  2. Use the available evidence above to bound the search.
  3. Reach one of the likely root causes.

Recovery procedure

(Do not reveal until you have reasoned through the problem.)

  1. Identify the failing component.
  2. Apply the remediation pathway.
  3. Validate with the verification step.
  4. Document the incident.

Remediation

  1. Identify the format. 2. Fix or use Loki pipeline_stages to parse.

Verification

Log timestamps align with the event time.

Rollback

Revert

Prevention

Standardise: UTC, ISO 8601