Investigation Time Budget

A primary on-call engineer joins a P1 channel at 03:14: a checkout SLO is burning at 14x baseline. Twenty minutes later the on-call manager asks for an update. The engineer’s Grafana window contains four dashboards and the change log. No hypothesis is on paper. They have spent the budget on dashboard exploration and have not written a single sentence.

A second engineer took a different shape of the same incident, three weeks earlier. They wrote a one-sentence hypothesis at 03:14, confirmed it at 03:21, mitigated at 03:29. The SLO burn stopped in fifteen minutes. The team difference is not talent. It is time discipline.

This lesson is the time discipline. The methodology (lesson 01) is the shape. The time budget is the metronome that keeps the shape on tempo. Without it, the loop drifts; with it, the loop produces a result that can be measured.

What a time budget is

A time budget is a wallclock limit, set per phase, per tier, that bounds how long the on-call engineer can spend before one of three actions is mandatory:

1. Escalate to a more senior engineer or to a domain specialist.
2. Defer the investigation to the next time window (often the next business day).
3. Commit to a hypothesis: the budget is over; the engineer’s best hypothesis is the one being mitigated, with the unconfirmed risk recorded in the runbook entry.

The budget is not a target. The goal is to finish earlier, not later. Engineers who finish phase 4 in four minutes when the budget was fifteen minutes are doing the right thing. The budget is the upper bound, not the lower bound.

Phase 1 - 2  --->  Phase 3 - 4  --->  Phase 5  --->  Phase 6
   budget          budget          decision       record
   |               |               |              |
   | if exceeded:  | if exceeded:  | decision:    | artifact:
   | stop and      | refine or     | confirmed /  | runbook entry
   | escalate      | replace       | refuted      |
   |               | hypothesis    | / unde-      |
   |               |               | cided        |
   |               | if 3x over:   |              |
   |               | escalate      |              |

Three properties the budget enforces:

• Bound the search. Every minute spent in phase 4 beyond the budget has a smaller marginal probability of producing new evidence than the first minute. The budget stops the engineer from chasing diminishing returns.
• Force hypothesis replacement. Three iterations of hypothesis-evidence is roughly the practical maximum before the engineer is “shopping”. The four-iteration rule is a consensus from incident postmortems across three SRE teams.
• Make time-to-mitigate measurable. Without a budget, MTTM is whatever the engineer happened to spend. With a budget, MTTM is bounded and can be improved against a target.

Per-tier budgets

The platform usually defines four tiers. The most common shape across SRE rotations is below; tune the numbers to your SLO and on-call rota, but keep the ratios.

The “Stop-and-escalate at” column is the most overlooked. It is the wallclock at which escalation becomes mandatory even without a confirmed hypothesis. The P0 stops at 15 minutes; the P1 at 45 minutes; the P2 at 6 hours; the P3 at 2 weeks.

Why P1 stops at 45 minutes: a P1 with no mitigation and no confirmed hypothesis at 45 minutes has either run into a novel failure mode or run into the engineer being wrong. In both cases escalation, not more solo investigation, is the right action.

The most common shape: the runaway P1

The most common shape the budget catches is a P1 that opens at a sensible phase 1 and 2, then collapses at phase 4. The engineer does not refines the hypothesis; the engineer opens more dashboards. Phase 4 time grows past the budget; the engineer does not notice because no clock is on screen. By the time the on-call manager asks for an update, the engineer is twenty minutes into a fifteen-minute budget with three refuted hypotheses and a fourth hypothesis being typed into the change-log search box.

The shape of the runaway is recognisable:

   phase 1-2        phase 4 time
   <--->
   -----------------|---------------------------------|------------
   0 min           10 min                          25 min        ^
                                          ^               ^
                                          |               |
                              budget +50%               escalate
                              (still inside)            (stop-and-escalate)

The shape has three signatures that show up in the runbook log:

1. No hypothesis recorded before phase 4 queries. The queries are running before any falsifier was written.
2. Phase 4 elapsed time grows past the budget. The engineer does not reset the budget clock between hypothesis iterations.
3. Multiple refuted hypotheses without a confirmed one. Three refuted hypotheses means the engineer is in evidence-shopping territory.

The fix is the budget clock, on screen, reset on every iteration. A simple text-mode clock (date; phase=4; budget_remaining=4; minutes_spent=11) is enough; what the team needs is the visibility that the budget is the contract.

How to configure it

The budget is operational discipline, but the platform can encode the stop-and-escalate at time as an alert:

# /etc/prometheus/rules/incident-budget.rules.yml
groups:
- name: incident-budget
  rules:
  - alert: InvestigationOverBudgetP1
    expr: |
      max_over_time(
        alert_active_hours{alertname=~"CheckoutHighErrorRate.*", tier="P1"}
        [5m]
      ) > 0.75
    for: 0m
    labels:
      severity: ticket
      team: payments
    annotations:
      summary: 'Investigation approaching P1 stop-and-escalate wallclock (45 min)'
      description: |
        Investigation for {{ $labels.alertname }} has been active for
        {{ $value }} hours; P1 stop-and-escalate wallclock is 45 min.
        Action: confirm hypothesis is on paper, escalate if not.

The recording rule alert_active_hours is fed by the alert manager’s firing history; the alert posts a ticket at 75% of the budget, which gives the on-call engineer the next 11 minutes to commit to a hypothesis or escalate.

A runbook entry that records the budget outcome:

# Runbook: CheckoutHighErrorRate P1, 2026-08-13

## Phase 1-2 (10 min)
- Symptom: checkout 5xx above 5% in us-east-1 since 03:14
- Impact: 3,200 failed requests in 18 min; SLO burn 14x

## Phase 3 (5 min)
- Hypothesis: payment-svc connection pool size 200 saturates
  database max_connections 100; pool exhaustion surfaces as 503

## Phase 4 (8 min)
- Database active connections 100/100 since 02:50
- payment-svc pool_size gauge 200 (was 20)
- Trace of failed checkout: 4.2 s wait on payment-svc /charge

## Phase 5 (3 min)
- Confirmed. Connection-pool saturation aligns with symptom
  onset and the 02:47 deploy; the trace explains the 503.

## Phase 6 (parallel with mitigation, 14 min)
- Mitigation: rollback deploy.
- Long-term follow-up: cap pool size at 80% of max_connections;
  pre-deploy check on pool vs DB max_connections; alert at
  80% pool utilisation.

## Budget outcome
- Total wallclock: 40 min (within P1 60-min budget).
- Iterations: 1 (within P1 3-iteration limit).
- Budget exhausted: no.
- Stop-and-escalate triggered: no.
- Outcome: mitigated, root cause located, runbook entry filed.

The Budget outcome section is the recording artefact. A team that has these outcomes for every P0/P1 over the last 90 days can plot MTTM against tier and against iteration count. That plot is the input to the next budget tuning.

How to validate it

Validate the budget by reading the last 30 days of P0/P1 runbook entries and measuring three quantities:

# SEVERITY: READ-ONLY
# 1. Confirm the alert active-hours recording rule exists.
curl -s 'http://prometheus:9090/api/v1/query?query=alert_active_hours%7Balertname%3D%22CheckoutHighErrorRateP0%22%7D' \
  | jq '.data.result[0].value'

Expected output:

[1723641600, "0.4"]

# SEVERITY: READ-ONLY
# 2. Count the P1 runbook entries that recorded a budget outcome.
grep -l 'Budget outcome' \
  /srv/runbooks/checkout/p1-*.md | wc -l

Expected output:

12

# SEVERITY: READ-ONLY
# 3. Count the P1 runbook entries that exceeded the budget wallclock.
grep -l 'budget elapsed.*>.*45 min\|escalated.*P1' \
  /srv/runbooks/checkout/p1-*.md | wc -l

Expected output:

2

Twelve entries, two escalated. A 16% escalation rate at P1 is the upper end of healthy: it means the loop is running, the budget is being enforced, and the team is not afraid to escalate. A 0% escalation rate at P1 means either the team never hits the budget (suspicious) or the team never records it (worse).

# SEVERITY: READ-ONLY
# 4. Confirm the over-budget alert fired at least once in the last 30 days.
curl -s 'http://prometheus:9090/api/v1/query?query=ALERTS%7Balertname%3D%22InvestigationOverBudgetP1%22%7D' \
  | jq '.data.result | length'

Expected output:

2

The four checks together confirm that the budget is wired, the runbook entries record the outcome, and the team is escalating when the budget is exceeded.

How it can fail

Six failure shapes occur when the budget is missing or when it is enforced without the runbook log.

1. Budget-as-target. The engineer spends the budget on every phase, regardless of how fast they finished. Phase 4 stops at 15 minutes even though the engineer’s first hypothesis confirmed at minute 4. Symptom: MTTM is flat at the budget ceiling even when iterations confirm early.
2. Budget-as-punishment. The engineer hides runaway investigations from the on-call manager because the budget exceeded is treated as a failure. Symptom: no over-budget alerts in 90 days; the alert history shows fires but no escalations.
3. No clock on screen. The budget is in the runbook but not in front of the engineer. Phase 4 minute-count is estimated, not measured. Symptom: runbook entries record “phase 4: 15 min” for every entry, regardless of actual elapsed.
4. Budget per phase, not per iteration. The engineer spends 15 minutes on one hypothesis, refines, and resets; phase 4 elapsed is now 30 minutes but the iteration count is two. Symptom: per-iteration budget is enforced; per-incident total budget is not.
5. Tier misapplied at page time. A P1 page opens with a P2 budget because the engineer “thought it looked like a P2”. Symptom: stop-and-escalate wallclock is measured against the engineer tier, not the alert tier.
6. Stop-and-escalate is a soft signal. The budget is exceeded but no one escalates; the engineer keeps running the loop alone. Symptom: 90 minutes of solo investigation against a 45-minute stop-and-escalate wallclock; the runbook entry records no escalation contact.

How to troubleshoot a runaway investigation

When the on-call engineer reports “I’ve been on this for two hours and I’m no closer”, the diagnostic order is:

1. Capture the timeline. Open the change log; mark each minute the engineer can account for. Phase 1-2 minute count; phase 3 hypotheses written; phase 4 queries and result timestamps.
2. Count the iterations. How many hypotheses have been tested? If three refuted, the engineer is in evidence-shopping territory.
3. Check whether the symptom is still observable. Phase 1 may have drifted; the user-impact metric may have recovered; the engineer is investigating a closed incident.
4. Check whether the impact has changed. SLO burn rate at 4x (P1) versus 14x (P0) is a different investigation with a different budget.
5. Re-tier if needed. A incident that began P2 may have escalated to P1 by burn-rate alone.

The action at each diagnostic step is bounded:

• Step 1 takes 5 minutes; do not skip it even when the pressure is “just go faster”.
• Step 2-3 are a 5-minute check; if either answer is unusual, escalate.
• Step 4-5 may trigger re-tiering, which resets the budget clock to the new tier’s wallclock.

Security implications

The time budget is mostly orthogonal to security, but two interactions matter.

• Access permissions during long P3 investigations. A P3 has a multi-day budget; the credentials issued at page time expire while the investigation is open. The credentials must be renewable, or the investigation will stall at the IAM layer instead of finishing on the methodology budget.
• Audit trail completeness. A budget that is exceeded without an escalation is a soft signal that audit logs may not capture. The audit log is the only place the post-incident review finds the actual wallclock; the runbook entry and the audit log must agree.

Performance implications

The time budget has two performance implications:

1. Phase 4 query cost is bounded. A 15-minute phase 4 budget against the production Prometheus / Loki / Tempo is a bounded query cost. A 90-minute phase 4 against the same store is uncosted query load during an incident. Bounding the budget is bounding the platform load during a page.
2. The budget is the throughput limiter. A team that runs the loop with no budget has theoretical throughput of one incident per engineer per shift; a team that runs the loop with the budget has the measured throughput of the budget. The budget is the rate at which the on-call rotation rotates.

Production guidance

• Put the budget clock on screen. A simple date; phase=B4; budget_remaining=3; minutes_spent=12 line is enough. Without the clock, the budget is theoretical.
• Record the budget outcome in every runbook entry. Total wallclock, iterations, budget exhausted Y/N, escalation contact.
• Review budget outcomes monthly. The plot of MTTM against tier and against iteration count is the input to the next budget tuning.
• Treat stop-and-escalate as a hard wallclock. It is not a suggestion; the engineer does not negotiate it.
• Re-tune after each tier change. When the SLO changes or the team size changes, the budget changes with it. The budget is a property of the platform, not a historical artefact.

Verification

You should now be able to answer:

• What is the per-tier phase budget for a P1 versus a P3?
• What are the four stop conditions in the methodology, and what action does each trigger?
• What is the runaway P1 shape, and how does the runbook log expose it?
• Why is the stop-and-escalate wallclock the most overlooked column of the per-tier budget table?
• Which of the six failure shapes (budget-as-target, budget-as-punishment, no clock on screen, budget per phase, tier misapplied, stop-and-escalate as soft signal) corresponds to a team that never fires the over-budget alert in 90 days?

Quiz