Post-Incident Cost Review

The Loki capacity incident closed at 16:20 on a Friday. The ingester memory returned to baseline. The bucket fill level dropped below 80 percent after the offending tenant was capped. The on-call wrote the timeline; the platform team breathed out; the next incident is, of course, already in motion somewhere.

Without a post-incident cost review, the incident is over. With one, the incident becomes the cheapest control in the platform. The review is the difference between a team that pays for the same incident twice and a team that pays for it once.

What it is

A post-incident cost review is a structured meeting held within five business days of an incident closing, with the goal of quantifying the cost of the incident, allocating that cost to a root cause and an owning team, and defining the guard rails that prevent the next incident of the same shape. The review is distinct from a postmortem in that the focus is the cost and the action items, not the timeline. A postmortem without a cost review is a narrative; a cost review without a postmortem is a number; the two together are the discipline.

The format is fixed. The same five sections, in the same order, every time. The format is the artefact the platform team points to when the action items are reviewed.

Why a sysadmin cares

The cost review is the discipline that turns an incident into a guard rail. Three operational consequences recur when the review is skipped:

• The same incident recurs. Without an action item that becomes a control, the next rotation pays for the same shape. The second incident costs as much as the first; the second incident is paid in full.
• The cost is not allocated. Without a documented allocation, the platform budget absorbs the cost; the team whose service caused the incident does not see a bill. The incentive to prevent the next incident disappears.
• The lessons are not propagated. Without a written artefact, the lessons live in the heads of the engineers who were on call that day. When those engineers rotate, the lessons rotate with them.

The cost review is not blame. It is the discipline that makes the incident cheap. The incident is paid; the review is the part that prevents the next one.

How it works

The mechanism is a fixed-format document plus a meeting plus a tracking system. The three together are the review.

Incident closes
      |
      v
Within 24h: on-call writes the timeline + captures the data
      |
      v
Within 5 business days: review meeting (30-60 minutes)
      |
      v
Review produces:
   1. Cost summary (storage, egress, compute, operator time)
   2. Root cause (with evidence)
   3. Allocation (which team, which budget)
   4. Action items (as guard rails, not tasks)
      |
      v
Action items are tracked in the platform backlog
      |
      v
Unclosed items appear in the next monthly review

The five sections of the document are not optional. A review that omits the cost summary is a postmortem; a review that omits the allocation is a complaint; a review that omits the action items is a record.

How to configure it

The configuration has three parts: the document template, the alert that schedules the review, and the tracking system for the action items.

The document template is the artefact. It is stored in the repository; it is filled in by the on-call; it is reviewed in the meeting.

# Post-Incident Cost Review: <incident-id>

## 1. Summary

- **Incident ID:** <id>
- **Service:** <service-name>
- **Incident commander:** <name>
- **Started:** <timestamp>
- **Closed:** <timestamp>
- **Duration:** <duration>
- **Severity:** <P1/P2/P3>

## 2. Cost

| Category | Quantity | Unit cost | Subtotal |
| --- | --- | --- | --- |
| Storage | <GB> | <per-GB-month> | <subtotal> |
| Egress | <GB> | <per-GB> | <subtotal> |
| Compute | <pod-hours> | <per-pod-hour> | <subtotal> |
| Operator time | <hours> | <loaded-rate> | <subtotal> |
| **Total** | | | **<total>** |

## 3. Root cause

<one paragraph, with evidence links to the triage queries>

## 4. Allocation

- **Owning team:** <team>
- **Budget line:** <budget>
- **Cost absorption:** <platform / team / shared>

## 5. Action items

| ID | Description | Type | Owner | Due | Status |
| --- | --- | --- | --- | --- | --- |
| AI-1 | <guard rail> | rate-limit / alert / CI / runbook | <owner> | <date> | open / in-progress / closed |
| AI-2 | <guard rail> | rate-limit / alert / CI / runbook | <owner> | <date> | open / in-progress / closed |

## 6. References

- timeline link
- triage query screenshots
- related incidents

The alert that schedules the review is a Prometheus rule that fires five business days after a P1 or P2 closes. The alert posts to the platform team channel.

# /etc/prometheus/rules/post_incident.yaml
groups:
  - name: post_incident_review
    rules:
      # A Loki P1 or P2 incident closed; remind the platform
      # team to schedule the cost review.
      - alert: PostIncidentReviewDue
        expr: |
          time() - max(loki_incident_closed_timestamp_seconds)
            > 5 * 24 * 3600
            and
          max(loki_incident_severity) <= 2
        for: 1h
        labels:
          severity: warning
        annotations:
          summary: 'Cost review due for incident {{ $labels.incident_id }}'
          runbook: 'https://runbooks/loki/post-incident-review'

The tracking system is the platform backlog. The action items are filed as guard rails (rate limits, alerts, CI checks, runbook entries), not as tasks. A guard rail is a control that prevents the incident shape from recurring; a task is a work item that may or may not be done.

How to validate it

The validation is four queries. The first confirms the incident data is captured; the second confirms the cost is allocated; the third confirms the action items exist; the fourth confirms the action items closed in the next cycle.

# 1. The incident data is captured. The cost summary should
# exist in the repository.
# Severity: READ-ONLY
gh issue list --label incident-cost-review \
  --state all --limit 20 \
  --json number,title,state,createdAt,closedAt \
  | jq '.[] | {number, title, state, createdAt, closedAt}'

# 2. The cost is allocated. The action item should reference
# a team and a budget line.
# Severity: READ-ONLY
gh issue list --label incident-action-item \
  --state all --limit 50 \
  --json number,title,body \
  | jq '.[] | select(.body | contains("budget"))'

# 3. The action items exist. Every cost review should produce
# at least one guard rail.
# Severity: READ-ONLY
gh issue list --label incident-action-item \
  --state all --limit 50 \
  --json number,title,labels \
  | jq '[.[] | select(.labels | any(.name | startswith("guard-")))] | length'

# 4. The action items closed. The platform team should track
# the closure rate over time.
# Severity: READ-ONLY
gh issue list --label incident-action-item \
  --state closed --limit 100 \
  --json number,closedAt,createdAt \
  | jq '[.[] | (.closedAt | fromdate) - (.createdAt | fromdate)] | add / length'

Expected: a positive closure rate (action items closed); a reasonable closure latency (median under 30 days). A high open rate or a long closure latency means the discipline is slipping.

How it can fail

Five failure shapes recur at the cost review.

1. The skipped review. The team is too busy; the review does not happen; the lessons live in heads. The first indicator is the cost-review GitHub issues accumulating without meetings.
2. The narrative-only review. The team writes a postmortem without numbers. The first indicator is the cost section is empty or vague.
3. The blame-only review. The team allocates the cost to a person rather than a system. The first indicator is the allocation section names an individual rather than a team.
4. The task-only action items. The team writes action items as tasks (“investigate X”). The first indicator is the action items do not reference rate limits, alerts, CI checks, or runbook entries.
5. The untracked closure. The team writes action items but does not track their closure. The first indicator is the open rate climbing.

How to troubleshoot it

1. Confirm the review happened (query 1)
        |
        v
2. Confirm the cost is allocated (query 2)
        |
        v
3. Confirm the action items are guard rails (query 3)
        |
        v
4. Confirm the action items closed (query 4)
        |
        v
5. For each failure shape, apply the right response:
        |
        +----> skipped? -> the alert should have fired; the
        |                  problem is the alert route, not
        |                  the team
        |
        +----> narrative-only? -> the template must include a
        |                        cost table; the discipline is
        |                        the template
        |
        +----> blame-only? -> the template must require a
        |                    team name, not a person; the
        |                    discipline is the template
        |
        +----> task-only? -> the review must include a
        |                  sign-off from the platform lead
        |                  who checks the action items are
        |                  guard rails
        |
        +----> untracked? -> the action items must be in the
        |                  platform backlog, not in the
        |                  postmortem document

Security implications

The cost review is not a security event by default, but two shapes have a security dimension. A log loop that amplified PII is a data incident; the cost review should include the compliance cost of the exposure in addition to the platform cost. A misconfigured tenant id that billed one tenant’s logs to another is a cross-tenant leak; the cost review should include the legal cost of the leak in addition to the storage cost. The discipline is to include the compliance and legal costs in the cost summary, not to scope them out.

Performance implications

The performance cost of the review is one to two hours per incident (the on-call writes the timeline; the meeting is 30 to 60 minutes; the action items are tracked in the backlog). The cost is fixed; the benefit is the prevention of the next incident. The ratio is favourable for any incident whose recurrence would cost more than the review.

Verification

You should now be able to answer:

• What are the four cost categories that a Loki cost review must quantify?
• What is the difference between a postmortem and a cost review, and why are both needed?
• What makes an action item a guard rail rather than a task, and why does the distinction matter?
• What is the escalation path for an action item that does not close within 60 days?

Quiz