Post-mortem and runbook exercises — the artefact that pays down the lesson

The post-mortem and runbook exercises are the artefact that pays down the lesson. The post-mortem records what happened; the exercises turn that record into a procedure the team can execute under pressure. A cluster without post-mortems is a cluster that repeats its failures.

The post-mortem

The post-mortem is the artefact that pays down the lesson. The post-mortem is the document that records the truth of the incident: what happened, why, what was the impact, what was the recovery, what was the lesson.

flowchart TD
    A[etcd incident] --> B[Post-mortem]
    B --> C[Timeline]
    B --> D[Impact]
    B --> E[Root cause]
    B --> F[Contributing factors]
    B --> G[Lessons]
    B --> H[Action items]

The post-mortem is the artefact that pays down the lesson.

The post-mortem template

The post-mortem template for an etcd incident:

• Summary. A one-paragraph description of the incident.
• Timeline. A minute-by-minute log of the incident.
• Impact. The customer-facing impact, the internal impact, the financial impact.
• Root cause. The technical chain of events.
• Contributing factors. The systemic factors that allowed the root cause.
• Detection. How the incident was detected.
• Response. How the incident was responded to.
• Recovery. How the etcd was restored.
• Lessons. What the team learned.
• Action items. The list of follow-up tasks.

The runbook exercises

The runbook exercises are the prevention strategy. The exercises are the rehearsals that the team runs to prepare for the next incident.

# Exercise 1: Restore the etcd from a snapshot
etcdutl snapshot restore /var/backups/etcd/snapshot.db

# Exercise 2: Replace a failed member
# MEMBER_ID from the `etcdctl member list` output; NEW_MEMBER is the
# name the replacement will run under.
MEMBER_ID=8e9e05c52164694d
NEW_MEMBER=etcd-03
ETCDCTL_API=3 etcdctl member remove "$MEMBER_ID"
ETCDCTL_API=3 etcdctl member add "$NEW_MEMBER"

# Exercise 3: Compact, defrag, and disarm the alarm
# REVISION from the `Revision:` field of `etcdctl endpoint status`.
REVISION=1348572
ETCDCTL_API=3 etcdctl compaction "$REVISION"
ETCDCTL_API=3 etcdctl defrag
ETCDCTL_API=3 etcdctl alarm disarm

# Exercise 4: Recover from a quorum loss
# (see Part CXXVI-02)

The runbook exercises are the cluster’s preparedness.

The action items

The action items are the deliverables. The action items are the follow-up tasks:

• “Add a fsync latency alert.”
• “Test the snapshot restoration in staging.”
• “Update the runbook with the failure mode.”
• “Tighten the gate so the change is harder to ship.”

Each action item has an owner. Each action item has a due date. The action items are tracked in the team’s backlog.

The schedule

The post-mortem and runbook exercises are scheduled:

• Post-mortem. Within 24 hours of the incident.
• Action items. Within 7 days of the post-mortem.
• Runbook exercises. Quarterly.

The schedule is the discipline.

Production discipline

The post-mortem and runbook exercises are the cluster’s hypothesis. The discipline is the same scale-free: every etcd incident gets a post-mortem and runbook exercises. The action items are the deliverables; the schedule is the discipline.

• Schedule the post-mortem within 24 hours. The post-mortem is the artefact.
• Run the runbook exercises quarterly. The exercises are the rehearsals.

Quiz