Communication — roles, cadence, and the channel as the record

An incident with good communication and mediocre debugging usually ends better than the reverse. The reason is arithmetic: the outage lasts as long as the slowest step, and in most incidents the slowest step is somebody waiting for information, duplicating work, or undoing what a colleague just did.

This lesson is about the structure that prevents that. It is not soft-skills material — it is the part of the response that determines whether five people are one team or five simultaneous investigations.

Roles, and the one separation that matters

On a team of three these collapse. One person can be commander and scribe. One separation must survive every collapse: the incident commander is not the person with their hands in the terminal.

The reason is attention. Debugging is deep focus; commanding is breadth — noticing that thirty minutes have passed, that the second theory has not been tested, that nobody has told support. A person doing both does the debugging and silently stops doing the other, and nobody notices until the incident review.

If you are alone, be explicit about switching hats. Set a timer for fifteen minutes; when it fires, stop typing and ask the commander questions out loud: what is the impact now, has it changed, who has not been told, what is the next thing to try if this one fails.

Cadence beats detail

Pick an interval by severity and hold it, including when there is nothing to report.

“No change, still investigating, next update at 10:45” is a complete and useful update. It tells everyone the incident is still owned, and it buys thirty minutes of uninterrupted work — which is the actual purpose. A team that updates unpredictably gets asked for updates constantly, and every one of those questions costs the responder more than the update would have.

What an update contains

10:15 UTC — INC-4471 — checkout API errors

IMPACT   Roughly 40% of checkout requests returning 500 since 09:52 UTC.
       Browsing and login unaffected. No data loss identified.
STATUS   Traffic removed from host-b at 10:04; error rate down to ~8%.
       Investigating why the api container on host-b is unhealthy.
NEXT     Capturing container state, then restarting the container on host-b.
UPDATE   10:45 UTC.
IC       @nadia    OPS @tomas    COMMS @priya

Four elements, in that order, every time. Impact first because it is what everyone is actually asking. Next-update time last because it is what stops them asking again.

The external version is shorter, in user language, and says less:

Investigating — Some customers may see errors when completing a purchase.
Browsing and account access are working normally. We are investigating
and will update by 10:45 UTC.

The channel is the timeline

The postmortem needs a minute-by-minute account. Reconstructing one from memory two days later produces a document that is confidently wrong about the order of events — and the order of events is usually where the lesson is.

The cheap fix is to paste the actual commands and their actual output into the channel as you run them.

$ docker inspect proj-api-1 --format 'status={{.State.Status}} health={{.State.Health.Status}} restarts={{.RestartCount}}'

status=running health=unhealthy restarts=0

That message is simultaneously an update, a piece of evidence, and a timestamped timeline entry. It costs nothing extra — you ran the command anyway — and it removes the “what time did we notice it was unhealthy” question from the review entirely.

Docker gives you a second, independent timeline for free:

docker events \
--since '2026-08-11T09:30:00Z' \
--until '2026-08-11T10:30:00Z' \
--format '{{.Time}} {{.Type}} {{.Action}} {{.Actor.Attributes.name}}'

The daemon records every start, stop, die, kill, health status change and network connect with a timestamp. Export that window into the incident record. It is authoritative about what happened to containers in a way that no human recollection is, and it frequently contradicts the story everyone remembers.

Handoff

Incidents outlive shifts. A handoff that is a summary loses the detail; a handoff that is a transcript loses the shape. Do both:

1. State the current impact in one sentence, in user terms, with a number.
2. State what is confirmed, and separately what is suspected. Keep the two lists distinct.
3. List every change made so far and whether it has been reversed — containment actions especially, since an un-reversed docker update --restart=no will confuse the next person badly.
4. Name the next action and the reason it is next.
5. Hand over the roles explicitly and by name, in the channel, and wait for the incoming person to acknowledge in writing.
6. Stay for ten minutes after the handoff. Most of what the outgoing person knows turns out to be un-written and surfaces in questions.

Sanity check