Leading incident response: when you are the on-call

The moment the pager goes off

Your phone buzzes. The text says “Cluster quorum lost - PVE-NYC-PROD”. You have 90 seconds to decide what to do. That decision-making, more than the technical fix, is what separates a good on-call from a great one.

This lesson covers incident leadership: how to coordinate the response, communicate clearly, and learn from the incident after.

flowchart LR
  A[Alert fires] --> B[Phase 1<br/>Acknowledge &amp; assess]
  B --> C[Phase 2<br/>Coordinate response<br/>IC + SME + Comms]
  C --> D[Phase 3<br/>Communicate<br/>15-min cadence]
  D --> E[Phase 4<br/>Resolve &amp; verify<br/>+10 min monitoring]
  E --> F[Phase 5<br/>Post-mortem<br/>within 48h]
  F --> G[Action items<br/>tracked to completion]

Phase 1: Acknowledge and assess (first 5 minutes)

When the alert fires:

1. Acknowledge — silence the pager, even if you haven’t fixed anything. Acknowledging prevents escalation.
2. Read the alert carefully — what system, what metric, what threshold
3. Form a hypothesis — what’s likely happening?
4. Open the incident channel — Slack #incident-2026-08-08-001 or equivalent
5. Post your first status update — even if it’s just “I’m on it, investigating”

[15:04:00] ALERT: Cluster quorum lost - PVE-NYC-PROD
[15:04:30] @on-call acknowledged. Investigating.
[15:05:30] Hypothesis: corosync link between pve1 and pve2 lost.
          Checking pvecm status on both nodes.
[15:07:00] Confirmed: pve2 cannot reach pve1. Corosync still up on pve1.
          Likely network switch issue. Checking switch logs.

Phase 2: Coordinate the response

Once you’ve assessed the situation, you need to coordinate. Three roles to fill:

Incident Commander (IC)

• Runs the response
• Decides when to declare, when to escalate, when to resolve
• Owns the timeline and the communications
• Does NOT do the technical work — their job is to coordinate, not to fix

Subject Matter Expert (SME)

• Does the actual technical investigation and fix
• Reports findings to the IC
• Can be multiple people (e.g., network SME + storage SME)

Communications Lead

• Owns stakeholder updates
• Posts to status pages, customer comms, executive updates
• Translates technical jargon into business impact

For a small team, one person can wear all three hats — but the disciplines remain. The IC discipline is most important: stop trying to fix things yourself and let others contribute.

Phase 3: Communicate

A good incident update has:

• What we know — facts, not speculation
• What we’re doing — current action
• What we don’t know — acknowledged unknowns
• Next update — when to expect the next message

[15:15:00] INCIDENT UPDATE
- Cluster quorum lost between pve1 and pve2
- Customer impact: 5% of VMs (web tier) experiencing errors
- Current action: investigating switch pve-sw-01
- Unknown: whether switch hardware failed or just rebooted
- Next update: 15:25 or sooner if state changes

Update frequency:

• Critical / customer-facing: every 15 minutes minimum
• Severe: every 30 minutes
• Degraded: every 1-2 hours

If nothing has changed, say so. “Still investigating, no change” is reassuring.

Phase 4: Resolve and verify

The moment the incident is resolved:

1. Verify the fix actually worked — don’t trust success messages, run the smoke tests
2. Monitor for 10-15 minutes to confirm the system is stable
3. Declare resolved in the incident channel
4. Schedule the post-mortem within 24-48 hours

[15:42:00] RESOLVED
- Switch rebooted; cluster recovered automatically
- All VMs back to healthy state
- Customer impact ended at 15:42
- Post-mortem scheduled: tomorrow 10:00 UTC in #incident-2026-08-08-001

Phase 5: Post-mortem (blameless)

A post-mortem is not about blame. It’s about understanding why the system failed and how to prevent it from happening again.

Format

Most post-mortems follow this template:

# Post-mortem: Cluster quorum loss on PVE-NYC-PROD

## Summary
- What happened (one sentence)
- When (UTC timestamps)
- Duration
- Customer impact

## Timeline
- 15:04 Alert fired
- 15:04 On-call acknowledged
- 15:07 Root cause identified (switch rebooted)
- 15:42 Resolved

## Root cause
The switch pve-sw-01 rebooted unexpectedly due to a firmware bug triggered by a specific SNMP poll pattern.

## Contributing factors
- Single switch for the cluster link (no redundancy)
- Switch firmware was 2 versions behind

## What went well
- IC discipline held; on-call stayed on comms
- Customer impact was limited to one tier
- Recovery was fast once root cause was found

## What went poorly
- Took 10 minutes to identify the switch as the cause
- Status page update was 5 minutes late

## Action items
- [ ] Add a second switch for cluster redundancy (owner: alice, due: 2026-09-15)
- [ ] Update switch firmware to latest (owner: bob, due: 2026-08-15)
- [ ] Improve alert to point at switch health (owner: alice, due: 2026-08-30)

The blameless discipline

The blameless rule: assume everyone acted with the information they had at the time. Don’t write “Alice made a mistake”. Write “The system did not provide Alice with the information she needed to make the right call”.

Why?

• People don’t take risks if they fear blame
• Blame hides systemic issues
• The goal is to improve the system, not punish the person

Common incident leadership mistakes

“I’ll just fix it myself”

You become the bottleneck. Even if you’re the only person on the team, you should still follow the discipline: IC role + SME role, even if both are you.

“I don’t want to interrupt people”

Most engineers want to be looped in during incidents. Bringing in help early reduces total time-to-resolve.

“I can’t post updates without a fix”

Updates without fixes are still valuable. “Still investigating” beats silence.

“We’ll figure out the root cause later”

If you don’t capture the timeline and root cause during the incident, you’ll lose them. Take notes throughout.

“Post-mortem is for the next sprint”

Schedule the post-mortem within 48 hours. Memory fades fast. Also, action items lose urgency if the post-mortem is delayed.

Building incident response muscle

A team that handles incidents well has:

• Drills — quarterly simulated incidents (one lesson covers this)
• Defined roles — every team member knows what IC / SME / Comms means
• Templates — incident channel template, post-mortem template
• Onboarding — new joiners run a drill in their first month
• Metrics — MTTA (mean time to acknowledge), MTTR (mean time to resolve), % of incidents with a post-mortem

Key takeaways

• Acknowledge fast, even before you know the cause
• IC discipline means coordinating, not fixing
• Update every 15 minutes during a critical incident
• Resolve means verified fix + 10-15 minutes of monitoring
• Post-mortem within 48 hours, blameless, action-oriented

Knowledge check