Evidence Preservation

The incident is closed at 16:42. The channel is renamed to closed-2026-08-13-checkout. The IC moves to the next Sev1. The post-incident review is scheduled for the following Monday. At the review, the room is asked: “at what time did the rollback fire?”. Three engineers give three different answers. The Incident Commander is not in the room. The Slack channel has been deleted by Slack’s retention policy. The decision log was never written. The dashboard URL is in three Slack tabs and two inboxes.

This is the failure mode evidence preservation exists to prevent. The evidence is the artefact the post-incident review reasons over. Without the evidence, the review is a forum, not an investigation. The discipline is the timeline being written before the incident is closed.

What it is

Evidence preservation is the discipline of capturing the incident timeline in a form that survives the incident. The captured artefacts are:

• The incident channel. The Slack or Teams channel that hosted the response. The channel is exported to a private Git repository at the close of the incident.
• The decision log. A structured, timestamped record of who decided what, when, and on what evidence. The log is written in the channel by the scribe.
• The dashboard snapshot. The pinned Grafana dashboard exported as JSON, with the time range and template variables captured.
• The query archive. The set of PromQL, LogQL, and TraceQL queries the analyst ran. The archive is the shareable artefact for the post-incident review.

The evidence is not the dashboard. The dashboard is a view into the telemetry; the evidence is the artefact that proves what the dashboard showed at the time. The discipline is proving what the team saw, not what the team remembers.

Why a sysadmin cares

The post-incident review is the feedback loop that improves the platform. The improvement is grounded in evidence: the review reads the channel, the decision log, the dashboard, and the queries. Without the evidence, the review is a recollection exercise. Recollection is biased toward the most recent and most stressful events; the contributing factors that preceded the trigger are forgotten.

The Etsy Debriefing Facilitation Guide and the Atlassian Incident Handbook both emphasise that the quality of the review is bounded by the quality of the evidence. A team that preserves the evidence values the review; a team that does not, declines over time.

How it works

The evidence is captured by four parallel pipelines, each firing on a different cadence.

   Evidence Pipeline
   ==================

   +-----------------+   +-----------------+   +-----------------+
   | Incident        |   | Decision log    |   | Dashboard       |
   | channel         |   | (scribe writes) |   | snapshot        |
   |                 |   |                 |   |                 |
   | every message   |   | every decision  |   | at close        |
   +--------+--------+   +--------+--------+   +--------+--------+
            |                     |                     |
            | hourly export       | live in channel     | at close
            v                     v                     |
   +-----------------+   +-----------------+            |
   | Git commit      |   | Slack thread    |            |
   | (per incident)  |   | (decision log)  |            |
   +-----------------+   +-----------------+            |
                                                       |
   +-----------------+                                  |
   | Query archive   |  -------------------------------+
   | (analyst)       |   save dashboard JSON
   |                 |   on disk
   +--------+--------+
            |
            v
   +-----------------+
   | Git commit      |
   | (per query)     |
   +-----------------+

The four pipelines are independent. The channel export is the primary record. The decision log is the structured overlay. The dashboard snapshot is the view at the moment of mitigation. The query archive is the analyst’s working notes.

The four are joined by a single identifier: the incident ID. The ID appears in the channel name, in the commit messages, in the dashboard title, and in the query file names. The post-incident review joins the artefacts by the ID.

How to configure it

The configuration is the channel export job, the decision log template, the dashboard snapshot script, and the Git repository.

The channel export job is a cron or a CI job that runs every hour during an incident.

# SEVERITY: READ-ONLY
INCIDENT_ID="${INCIDENT_ID:?set incident id}"
INCIDENT_CHANNEL="${INCIDENT_CHANNEL:?set channel id}"
INCIDENT_START_EPOCH="${INCIDENT_START_EPOCH:?set start epoch}"

ARCHIVE_DIR="/var/lib/incident-archive/${INCIDENT_ID}"
mkdir -p "${ARCHIVE_DIR}"

slack-cli --token "${SLACK_BOT_TOKEN}" \
  conversations.history \
  --channel "${INCIDENT_CHANNEL}" \
  --oldest "${INCIDENT_START_EPOCH}" \
  --limit 1000 \
  | jq '.messages' > "${ARCHIVE_DIR}/channel-${INCIDENT_ID}.json"

git -C "${ARCHIVE_DIR}" add channel-${INCIDENT_ID}.json
git -C "${ARCHIVE_DIR}" commit -m "incident ${INCIDENT_ID}: channel snapshot"

The job is run by the on-call bot when the incident is declared. The job continues until the incident is closed.

The decision log is a thread in the incident channel with a structured template. The first message in the thread is the template; subsequent replies are the entries.

# Decision log template

[hh:mm] DECISION: <one-line description>
HYPOTHESIS: <what the team believed at the time>
EVIDENCE: <which panel, log query, or trace>
ALTERNATIVE: <the option that was rejected>
DECIDED BY: <name>
COUNTER: <alice +1, bob -1, ...>

The decision log is written by the scribe. The thread is pinned in the channel. The IC calls out the decision in the main channel; the scribe writes the structured entry in the thread.

The dashboard snapshot is a script that runs at the moment of mitigation.

# SEVERITY: READ-ONLY
INCIDENT_ID="${INCIDENT_ID:?set incident id}"
DASHBOARD_UID="${DASHBOARD_UID:-sev1-main}"

curl -s -u "${GRAFANA_USER}:${GRAFANA_API_KEY}" \
  "https://grafana.example.com/api/dashboards/uid/${DASHBOARD_UID}" \
  | jq '.dashboard' \
  > "/var/lib/incident-archive/${INCIDENT_ID}/dashboard-${DASHBOARD_UID}.json"

git -C "/var/lib/incident-archive/${INCIDENT_ID}" \
  add dashboard-${DASHBOARD_UID}.json
git -C "/var/lib/incident-archive/${INCIDENT_ID}" \
  commit -m "incident ${INCIDENT_ID}: dashboard snapshot at mitigation"

The query archive is a small directory of .promql, .logql, and .traceql files committed to the archive repository. The analyst writes the query as a file rather than pasting into a chat.

# SEVERITY: READ-ONLY
INCIDENT_ID="${INCIDENT_ID:?set incident id}"
QUERY_DIR="/var/lib/incident-archive/${INCIDENT_ID}/queries"

mkdir -p "${QUERY_DIR}"
cat > "${QUERY_DIR}/01-checkout-error-rate.promql" <<'EOF'
# hh:mm — checked error rate per service for the last 5m
sum by (service) (
  rate(http_requests_total{status=~"5.."}[5m])
)
/ sum by (service) (
  rate(http_requests_total[5m])
)
EOF

git -C "/var/lib/incident-archive/${INCIDENT_ID}" add queries/
git -C "/var/lib/incident-archive/${INCIDENT_ID}" \
  commit -m "incident ${INCIDENT_ID}: archived checkout-error-rate query"

The archive is a working directory, not a database. The discipline is that the analyst writes the query as a file because the file is committed to Git and the Git history is the audit trail.

How to validate it

Five checks before the incident is closed.

1. The channel export is current.

# SEVERITY: READ-ONLY
INCIDENT_ID="${INCIDENT_ID:?set incident id}"
ARCHIVE_DIR="/var/lib/incident-archive/${INCIDENT_ID}"

jq -r '.[].ts' "${ARCHIVE_DIR}/channel-${INCIDENT_ID}.json" \
  | sort -n | tail -1

The most recent timestamp in the export should be within the last ten minutes. A timestamp older than the last hour means the export is stale.

2. The decision log thread exists.

# SEVERITY: READ-ONLY
slack-cli --token "${SLACK_BOT_TOKEN}" \
  conversations.replies --channel "${INCIDENT_CHANNEL}" \
  --ts "${DECISION_THREAD_TS}" \
  | jq '.messages | length'

A count greater than zero means the decision log is populated. A count of zero means the scribe has not written any entries.

3. The dashboard snapshot is on disk.

# SEVERITY: READ-ONLY
ls -la "/var/lib/incident-archive/${INCIDENT_ID}/dashboard-"*.json

A non-empty list means the snapshot was taken. An empty list means the snapshot script was not run.

4. The query archive is committed.

# SEVERITY: READ-ONLY
git -C "/var/lib/incident-archive/${INCIDENT_ID}" \
  log --oneline -- queries/

A commit history with one entry per query file is the audit trail. An empty log means the analyst did not commit any queries.

5. The archive repository is private.

# SEVERITY: READ-ONLY
gh repo view "${ARCHIVE_REPO}" --json visibility \
  | jq -r '.visibility'

Expected output:

PRIVATE

A public or internal repository leaks the incident timeline. The incident archive must be private.

How it can fail

Five failure modes recur in incident evidence preservation.

1. Channel deleted before export. The channel is deleted at incident closure by an over-eager admin. Symptom: the post-incident review has no conversation history. The review is a forum, not an investigation.
2. Edited messages. A participant edits a message to remove a misleading claim. Symptom: the edited message is part of the audit; the edit is not visible. The post-incident review misses the correction.
3. Untimestamped evidence. A screenshot is pasted into the channel without a timestamp. Symptom: the screenshot is a moment in time, but the review cannot place it in the timeline.
4. Screenshots instead of URLs. The analyst screenshots the dashboard instead of pasting the URL. Symptom: the screenshot is a frozen view; the review cannot re-render the dashboard with the same template variables.
5. Clock skew. The dashboard timestamp is in container time; the channel timestamp is in wall-clock time. Symptom: the timeline is offset by the clock skew; the review cannot correlate the channel message with the panel change.

How to troubleshoot it

When the evidence is incomplete at the close of the incident, the order is:

1. Stop the channel deletion. The IC prevents the channel from being deleted until the export is confirmed.
2. Run the export. The on-call bot runs the channel export one last time. The export is the primary record.
3. Capture the decision log. The scribe writes the decision log thread if it does not exist. The thread is the structured overlay.
4. Snapshot the dashboard. The dashboard snapshot script is run. The snapshot is the view at the moment of mitigation.
5. Commit the queries. The analyst commits the queries as files. The commit is the audit trail.
6. Lock the archive. The archive repository is marked read-only. The seal is the post-incident review’s read-only input.

Security implications

The incident archive is a sensitive artefact. It contains internal service names, customer-impact language, and often references to unpublished security vulnerabilities. The archive must be in a private repository.

The channel export must redact credentials. The SLACK_BOT_TOKEN in the export command is a credential; the export script must not write the token to the JSON. The dashboard snapshot may contain user PII via the labels; the snapshot must be filtered at the dashboard level before the export.

The decision log may contain names of customers who were affected. The log must be stored in a private repository and the post-incident review must anonymise the customer references before the review document is published.

The archive repository access list must be the on-call rotation plus the incident participants, not the whole engineering organisation. The audit trail of who read the archive is a separate concern; the archive itself is the incident’s heart.

Performance implications

The channel export job runs every hour during an incident. A channel with thousands of messages returns within seconds; a channel with hundreds of thousands of messages is a slow export. The mitigation is to limit the export to the incident’s time window, not the full channel history.

The dashboard snapshot is a one-off operation. The snapshot is a JSON document with the panel definitions and the templating variables; the document is small (a few hundred kilobytes). The export is bounded by the dashboard size, not the data source.

The query archive is a working directory. The directory grows with the number of queries the analyst writes. The discipline is to commit the queries as files, not to paste them into the channel. The commit history is the audit trail.

Production guidance

• The evidence is captured during the incident, not at the end. The hourly export is the discipline.
• The decision log is the structured overlay. The thread is pinned in the channel; the scribe writes the entries as the decisions are made.
• The dashboard snapshot is the view at the moment of mitigation. The snapshot is taken twice: at the rollback, and at the closure.
• The query archive is the audit trail of the investigation. The analyst writes the query as a file and commits it; the diff is the investigation.
• The archive repository is private. The visibility is a CI check; the access list is a quarterly review.

Verification

You should now be able to answer:

• What four artefacts make up the incident evidence, and which one is the primary record?
• Why is the channel export run during the incident and not at the end?
• What is the decision log, where is it stored, and what structured fields does each entry carry?
• What five failure modes recur in evidence preservation, and how do you diagnose each one?
• Why is the archive repository a private repository, and what is the access-list discipline?

Quiz