Escalation and handover - moving an incident between people

Escalation is not asking for help, and it is not admitting that you could not solve something. It is the transfer or widening of decision authority, and framing it as a competence question is the single biggest reason it happens too late.

The engineer who spends ninety minutes on a problem rather than escalating at thirty is not being diligent. They are holding a decision that belongs to someone else, and the incident is longer because of it.

Escalate on triggers, not on feelings

“I will escalate if I get stuck” fails because being stuck is not a state anyone recognises from the inside. Define the triggers in advance and treat them as automatic:

Adding people is not escalating

A call with eleven people on it and no change in decision authority is not an escalated incident. It is the same incident, now with worse communication and a commander who has to manage an audience.

Escalation is specific: it names a person, a role, and what decision they are being asked to take.

ESCALATION - INC-4820
To:        s.okonkwo (on-call engineering lead)
Reason:    Time box - 30 minutes at SEV2 with no change in error rate
Asking for:
  (a) Authority to fail over to the secondary site. The failover
      procedure has not been tested since March and may fail.
  (b) A decision on whether to wake the database on-call.
State:     Impact and current mitigations are in INC-4820. Summary below.
Impact:    Checkout unavailable, EU only, 41 min, no data loss confirmed
           on the payment path; order-history write path unchecked.
Tried:     Rolled back deploy 4471 (no effect). Drained node2 (no effect).
           Rate limit on /api/search (reduced 5xx from 74% to 61%).
Believe:   Something upstream of the application tier - the pattern is
           identical across all three app nodes.
Do not know: Whether the order-history writes are completing.

The Do not know line is the one that makes an escalation useful. A summary containing only what you have established invites the escalation target to start over; naming the open question lets them direct effort immediately.

Handover between shifts

Long incidents outlive the people running them. A handover is a transfer of the commander role and everything attached to it, and it is done explicitly or not at all.

1. Write the packet before the conversation. Talking through an incident from memory takes twenty minutes and loses things; reading a document together takes five.
2. Cover: current impact and when it was last measured, current severity, the mitigations register with what is still open, what has been tried and ruled out, the current hypothesis, the open questions, who is engaged including vendors and case numbers, and the next scheduled communication.
3. State the transfer out loud, in the channel: "Sofia is now incident commander as of 07:00Z." Not in a direct message - the whole incident needs to know who to address.
4. Have the incoming commander say it back. Not "understood" - an actual summary of the impact, the current hypothesis and the next action. Mismatches surface here or during the next hour.
5. The outgoing person stays reachable for 15 minutes, and then genuinely leaves. Half-handed-over incidents produce two people making decisions.

Escalating to a vendor

Vendor escalation has the longest latency in the whole process, so it starts earliest, and it fails for predictable reasons: the case is opened at the wrong severity, or without the data the vendor needs, and the first response is a request for that data — costing you a whole response cycle.

Collect the diagnostic bundle before opening the case. Most enterprise Linux vendors want an sos report:

# Generates a diagnostic archive under /var/tmp. Interactive by
# default; --batch skips the prompts.
sudo sos report --batch

# Substitute your own values before running:
CASE=01234567

# Label the archive with the case number so it is identifiable
sudo sos report --batch --case-id "$CASE"

The case itself needs four things, and omitting any one of them adds a response cycle:

1. The business impact in their severity language, not yours. A vendor SEV1 usually requires the words "production down" and a statement of who is affected; a case opened as "high" gets a next-business-day queue.
2. The exact versions: distribution, kernel, and the package in question. Not "the latest".
3. What changed, and when. Vendors ask this first, every time.
4. The diagnostic bundle attached at case creation, plus the specific log excerpt showing the failure, with timestamps in UTC.

Then track the vendor case in your own incident: a case number sitting in someone’s email with nobody watching it is a dependency you have stopped managing.

Knowledge check