Recovery and post-mortem — the security incident's closure

Containment stops the bleeding; it does not end the incident. Until the malicious workload, the credential it used, and the image it came from are all gone, the cluster is one restart away from the same compromise — and rolling back to the last known-good manifest will happily reinstate the vulnerable version. This lesson covers eradication and the return to a known-good state, then the post-mortem that converts the incident into action items with owners and dates.

The recovery

The recovery is the cluster’s security closure. The recovery is the cluster’s return to a known-good state.

flowchart TD
    A[Security incident] --> B[Containment]
    B --> C[Eradication]
    C --> D[Recovery]
    D --> E[Post-mortem]

The recovery is the cluster’s security closure.

The eradication

The eradication is the cluster’s removal of the threat. The eradication is the cluster’s deletion of the malicious binary, the malicious deployment, the malicious credential.

# Substitute your own values before running:
NS=production
MALICIOUS_DEPLOYMENT=xmrig-worker
COMPROMISED_SECRET=payments-sa-token-8fq2d
APP_DEPLOYMENT=payments-api
CONTAINER=api                                       # container inside $APP_DEPLOYMENT
PATCHED_IMAGE=registry.example.com/acme/api:1.4.2

# Delete the malicious deployment
kubectl delete deployment "$MALICIOUS_DEPLOYMENT" -n "$NS"

# Rotate the malicious credential
kubectl delete secret "$COMPROMISED_SECRET" -n "$NS"

# Patch the malicious binary
kubectl set image deployment "$APP_DEPLOYMENT" -n "$NS" "$CONTAINER=$PATCHED_IMAGE"

The eradication is the cluster’s removal of the threat.

The post-mortem

The post-mortem is the artefact that pays down the lesson. The post-mortem is the document that records the truth of the incident: what happened, why, what was the impact, what was the recovery, what was the lesson.

flowchart TD
    A[Post-mortem] --> B[Timeline]
    A --> C[Impact]
    A --> D[Root cause]
    A --> E[Contributing factors]
    A --> F[Lessons]
    A --> G[Action items]

The post-mortem is the artefact that pays down the lesson.

The post-mortem template

The post-mortem template for a security incident:

• Summary. A one-paragraph description of the incident.
• Timeline. A minute-by-minute log of the incident.
• Impact. The customer-facing impact, the internal impact, the financial impact.
• Root cause. The technical chain of events.
• Contributing factors. The systemic factors that allowed the root cause.
• Detection. How the incident was detected.
• Response. How the incident was responded to.
• Recovery. How the cluster was restored.
• Lessons. What the team learned.
• Action items. The list of follow-up tasks.

The action items

The action items are the deliverables. The action items are the follow-up tasks:

• “Add a Falco rule for the malicious binary.”
• “Rotate the credential automatically.”
• “Update the runbook with the failure mode.”
• “Tighten the gate so the change is harder to ship.”

Each action item has an owner. Each action item has a due date. The action items are tracked in the team’s backlog.

The remediation

The remediation depends on the failure:

# Option 1: Apply the post-mortem recommendations
# (in the runbook)

# Option 2: Implement the action items
# (in the team's backlog)

# Option 3: Schedule the runbook exercises
# (quarterly)

# Option 4: Communicate the lesson
# (in the team's knowledge base)

The remediation is the closure.

Production discipline

A security incident recovery is the cluster’s hypothesis. The discipline is to walk the 11-step methodology applied to the security layer, identify the cause, apply the remediation. The security is the cluster’s protection; the remediation is the closure.

• Schedule the post-mortem within 24 hours. The post-mortem is the artefact.
• Track the action items. The action items are the deliverables; each one has an owner and a due date.
• Run the runbook exercises. The exercises are the rehearsals.
• Update the runbook. The runbook is the team’s reference.

Quiz