Incident roles and communication - the team coordination

An incident response team needs clear roles and communication. This lesson covers the standard roles, the communication patterns, and the post-incident review.

Roles

In a typical incident:

• Incident Commander (IC): leads the response. Makes decisions. Coordinates the team.
• Subject Matter Expert (SME): knows the system. Does the investigation.
• Communications Lead: handles internal and external communications.
• Scribe: takes notes. Records decisions and timeline.

The IC is the single point of decision. The SME is the single point of investigation. The Communications Lead manages messaging. The Scribe captures the story.

Communication channels

A typical incident uses:

• War room (video call): the IC, SMEs, comms, scribe. The synchronous channel.
• Chat (Slack, Teams): the team and stakeholders. Asynchronous updates.
• Status page: customer-facing.
• Incident ticket: the system of record.

Channels serve different purposes:

• War room: real-time coordination.
• Chat: status updates and questions.
• Status page: customer communication.
• Ticket: documentation.

Effective incident calls

A good incident call:

• One person speaks at a time.
• Updates are concise (30 seconds to 1 minute).
• Decisions are recorded (by the scribe).
• Time is tracked.

A bad incident call:

• Multiple people talk at once.
• Long, unfocused updates.
• No decisions.

Status updates

Status updates include:

• What happened (the symptom).
• What is the impact.
• What is being done.
• When the next update is.

The cadence depends on severity.

Severity levels

Severity is decided from observable impact, not from how alarming the alert looked or how hard the fix will be. Declare it in the first five minutes, write it in the channel topic, and re-declare it out loud whenever it changes — every other decision in the incident hangs off this one number.

Two rules keep this honest:

• Lost redundancy is a SEV2, not a SEV3. Nothing is broken yet and everything looks green from the outside, which is exactly why it gets under-declared. A two-node cluster running on one node is one failure away from an outage, and that failure will happen at 03:00 on a Sunday.
• Severity goes up easily and down slowly. Over-declaring costs you an apology; under-declaring costs you the hour it takes to assemble the response you should have started with. If the two of you disagree, take the higher one and downgrade once you have evidence.

Post-incident review

A post-incident review (PIR) is a blameless review of what happened:

• What was the incident?
• What was the impact — who was affected, for how long, measured from the first customer-visible symptom rather than from the first alert?
• What was the trigger: the specific change or event that started it (a deploy, a certificate expiry, a disk filling)?
• What were the contributing factors: the conditions that let the trigger cause an outage instead of a shrug — missing monitoring, a single point of failure, a runbook that was wrong?
• What was the root cause: the decision or gap that put those conditions in place?
• How was it detected, and how long did detection take? If a customer told you first, that is its own action item.
• What did we do well?
• What did we do poorly?
• What should we change?

Separating trigger from contributing factors from root cause is what stops the review from producing a single action item that says “be more careful”. The trigger is usually unremarkable and will recur; the contributing factors are what you can actually remove.

The output is a list of action items. The review is blameless: the focus is the system, not the people.

Knowledge check