Reconstructing the timeline - evidence over recollection · Linux

The timeline is the deliverable that everything else in the review is derived from. Get it right and the trigger is obvious, the contributing factors fall out of the gaps, and disagreements about what happened resolve themselves by pointing at a line. Get it wrong and the review produces a story that several people remember differently and nobody can check.

The single most important property of a timeline is that every line has a source. Not “we think the deploy went out around two” - a log line, a commit, a ticket transition, a message with a timestamp.

Memory is not evidence

Human recollection of an incident is systematically distorted in ways that all point the same direction.

Compression. Forty minutes of confusion is remembered as ten. Ask anybody how long they spent before escalating and the answer is roughly half the real figure.
Reordering. People remember the order in which they understood things, not the order in which they happened. The hypothesis that turned out to be right migrates earlier.
Anchoring on the outcome. Once you know the cause, every earlier observation seems to have pointed at it, including the ones that pointed elsewhere at the time.
Loss of the discarded. The three hypotheses that were tested and rejected vanish from the account entirely - and those are exactly what tell you whether the evidence available was sufficient.

None of this is carelessness; it is how recall works under stress. It is also why the timeline is built from artefacts first, and interviews used only to explain the artefacts.

The sources

Sweep all of them. An incident that touched several layers has evidence in several places, and the interesting lines are usually in the source nobody thought to check.

The journal on every affected host, at sub-second precision
Application logs, which are frequently not in the journal and frequently in a different timezone
Package and configuration history: dpkg.log, dnf history, the /etc git log, configuration management run reports
Deployment and CI systems: what was released, to where, at what time
Monitoring and alerting: when each alert fired, when it was acknowledged, when it resolved
The chat channel and the incident ticket, which timestamp the human decisions
Load balancer, proxy and firewall logs, which often show the user-visible start before any host does
Cloud provider or hypervisor event logs: live migrations, host maintenance, storage events
The status page, which records what was said and when

Normalise the clocks first

Merging timestamped lines from several sources is only valid if the timestamps are comparable. Two failures make them not comparable, and both are common.

Read-only / Safewas this host telling the truth about the time— Run this on every host contributing evidence, and run it during the review rather than trusting that time sync was healthy. A host that was 4 seconds off during the incident will reorder events across hosts, and the resulting story will be confidently wrong.

$ timedatectl; chronyc tracking | head -6

               Local time: Tue 2026-08-11 15:52:03 UTC
         Universal time: Tue 2026-08-11 15:52:03 UTC
               RTC time: Tue 2026-08-11 15:52:02
              Time zone: Etc/UTC (UTC, +0000)
System clock synchronized: yes
            NTP service: active
Reference ID    : 0A000201 (ntp1.example.com)
Stratum         : 3
Ref time (UTC)  : Tue Aug 11 15:47:24 2026
System time     : 0.000318215 seconds slow of NTP time
Last offset     : -0.000091447 seconds
RMS offset      : 0.000412008 seconds

Illustrative output

Timezone. The journal displays in the local timezone by default, application logs may use another, and a cloud console often shows a third. Force everything to UTC before you compare:

journalctl --utc --since '2026-08-11 08:55' --until '2026-08-11 09:30' -o short-precise

Skew. A host whose clock was wrong during the incident produces lines that interleave incorrectly with everything else. The journal records this: a clock jump appears as a discontinuity you can search for.

journalctl -b --no-pager | grep -iE 'time jumped|System clock (wrong|time changed)|Clock (change|jump)'

If a host was skewed, either correct its lines by the known offset and say so explicitly in the timeline, or mark that host as providing ordering within itself only. Silently merging skewed lines is worse than leaving them out, because the result looks authoritative.

A four-second skew invents a causal relationship

The reason clock skew matters more in RCA than anywhere else is that the timeline is used to infer causation from order. If db01 is four seconds ahead, its error at what it calls 09:14:06 was really at 09:14:02 - before app01 logged its failure at 09:14:04 rather than after it.

Read in the wrong order, the story is “the application failed and the database noticed”. Read correctly, it is “the database failed and the application noticed”. Those two stories produce entirely different remediation work, and nothing in the merged log file indicates which one you are looking at.

Check chronyc tracking or timedatectl on every contributing host before you merge anything, and record the check in the review. It costs one command per host and it is the difference between a timeline and a coincidence.

Two timelines, tracked separately

Keep the system timeline and the response timeline in separate columns. They answer different questions, and merging them hides the most valuable measurement in the review.

System timeline: what the system did. Trigger, first degradation, first user-visible impact, cascading failures, recovery.

Response timeline: what people did. First alert, first human acknowledgement, incident declared, first mitigation attempted, service restored, all-clear.

The intervals between the two are the numbers that matter:

09:14:02  SYSTEM    upstream TLS verification begins failing
09:14:30  SYSTEM    circuit breaker opens; 100% of API requests fail
09:17:00  SYSTEM    first customer-visible errors on the status path
09:31:11  RESPONSE  first alert fires (synthetic check, 15 min interval)
09:33:40  RESPONSE  on-call acknowledges
09:38:02  RESPONSE  incident declared SEV1
09:41:15  RESPONSE  hypothesis: expired certificate
09:46:30  RESPONSE  new certificate issued and deployed
09:47:52  SYSTEM    error rate returns to baseline

  time to detect    09:14:02 -> 09:31:11 = 17 m 09 s
  time to diagnose  09:33:40 -> 09:41:15 =  7 m 35 s
  time to restore   09:41:15 -> 09:47:52 =  6 m 37 s
  total duration    09:14:02 -> 09:47:52 = 33 m 50 s

Time to detect is seventeen minutes of a thirty-four minute incident. That single number is usually the largest and cheapest improvement available, and it is invisible in a merged timeline because the alert line sits between two system lines and looks like part of the story rather than a delay in it.

Note also what the decomposition exposes: the synthetic check runs every fifteen minutes, so detection latency was bounded by the check interval rather than by anyone being slow. That is a contributing factor with an obvious action, and it came out of the arithmetic rather than out of anybody’s opinion.

Building it

Read-only / Safemerge across hosts— short-iso-precise puts the whole timestamp in field 1, so a plain sort on that field interleaves the hosts correctly; short-precise splits it across three fields and sorts by month name. The hostname is already field 2 of every line, so the merged file says which machine spoke.

$ for h in app01 app02 lb01 db01; do ssh "$h" "journalctl --utc --since '2026-08-11 09:00' --until '2026-08-11 10:00' -o short-iso-precise --no-pager"; done | sort -k1,1 > /var/tmp/incident-merged.txt; wc -l /var/tmp/incident-merged.txt

4187 /var/tmp/incident-merged.txt

Illustrative output

Four thousand lines is not a timeline; it is the raw material. Reduce it by pulling out only the lines that change your understanding, and give each one a source reference so a reader can go back to it.

TIME (UTC)    TYPE      HOST   EVENT                                  SOURCE
09:14:02.771  SYSTEM    app01  x509 expired on upstream api           journal:myapp
09:14:30.114  SYSTEM    app01  circuit breaker opened                 journal:myapp
09:31:11.000  RESPONSE  -      synthetic check alert fired            alertmanager
09:33:40.000  RESPONSE  -      on-call acknowledged                   pagerduty
09:41:15.000  RESPONSE  -      hypothesis raised in channel           slack #inc-482
09:46:30.902  SYSTEM    lb01   new certificate loaded, reload ok      journal:nginx
09:47:52.418  SYSTEM    app01  error rate at baseline                 prometheus

Seven lines, each with a source. That is a timeline. A reviewer who was not there can check every line, and disagreements are settled by looking rather than by remembering.

Gaps are findings

When the timeline is assembled, look at what is missing. Three patterns are worth a contributing factor each.

A long interval with no lines at all. Nothing was logged and nobody wrote anything down. Usually this is the period when everyone was busy, and it is where the response is least understood. Ask what was happening and record it, even approximately, marked as reconstructed rather than evidenced.

Impact before detection. Any interval between the first user-visible symptom and the first alert is a detection gap with a size attached. It converts directly into an action item.

An observation that does not fit. A line that your explanation does not account for. This is the most valuable thing in the timeline and the most tempting to leave out, because it complicates a story that was nearly finished.

Every observation must be explained, including the inconvenient ones

The stop-rule for the troubleshooting loop applies here with more force, because there is no time pressure to excuse skipping it: you are finished when you can explain every observation in the evidence, including the ones that did not fit your first story.

An unexplained line means one of three things, and all three matter: the story is incomplete, there is a second problem running concurrently, or a clock is wrong. Each changes what the remediation should be.

The practical failure is subtle. The unexplained observation is rarely disputed - it is simply not mentioned in the write-up, because it was hard to fit and the deadline was Friday. The review then produces an action that prevents the wrong thing, and the same incident recurs with a slightly different shape.