Workflow Iteration

The first storage-full incident took ninety minutes. The second took forty-five. The third took twenty. The fourth took ten. The MTTM is not converging to a number the team discovered; it is converging because the team refined the template after each incident. The fourth incident was an on-call engineer reading a template that captured everything the previous three engineers had learned.

This lesson is the iteration loop. The discipline is to refine the template after every incident, not to refine the template only when the team has time.

What it is

Workflow iteration is the discipline of refining the template after each incident so the next incident of the same class is faster. A refinement is one of:

• A new hypothesis candidate added to phase 3.
• A new evidence surface added to phase 4.
• A new falsifier added to a hypothesis.
• A new root cause category added to phase 6.
• A new mitigation category added to phase 6.
• A new follow-up category added.

A refinement produces a new version of the template; the old version is retained. The version history is the team’s incident learning, visible in the git log of the runbooks repository.

Why a sysadmin cares

A team that iterates on its templates resolves the Nth incident of an incident class in roughly the same time as the second. A team that does not iterate resolves the Nth incident in roughly the same time as the first. The difference compounds: over a year, an iterating team may resolve fifty incidents of one class at an average MTTM of twelve minutes; a non-iterating team resolves the same fifty incidents at an average MTTM of sixty. The compound effect is hours of on-call time saved per quarter.

The iteration is also what stops the team from re-learning the same lessons. A team that does not iterate learns the same lesson three times and forgets it twice; a team that iterates learns the lesson once and remembers it forever.

How it works

The iteration loop has five steps:

  1. Close the runbook log entry (phase 6 of the loop)
  2. Compare the entry against the template
  3. Identify the gaps (what the template did not predict)
  4. Refine the template (new hypothesis, new evidence
     surface, new falsifier)
  5. Commit the refinement (new template version)

The iteration loop runs after every incident. The cost is bounded: fifteen minutes per incident for the engineer who closed the entry. The benefit is bounded by the rate of incidents of the same class; an incident class that fires once a quarter justifies a one-hour total investment in iteration over a year.

Step 1. Close the runbook log entry

The phase 6 entry is the artefact that feeds the iteration. Without the entry, the iteration has no source material. The engineer who closed the entry is the engineer who knows what the template predicted and what the template missed.

Step 2. Compare the entry against the template

The comparison is structured. For each phase, ask: did the template predict what the entry recorded?

The questions are answered by reading the entry alongside the template. Where the entry matches the template, the template is correct. Where the entry diverges, the template is incomplete.

Step 3. Identify the gaps

A gap is a divergence between the entry and the template. The most common gaps:

Step 4. Refine the template

The refinement is a single edit to the template file in the runbooks repository. The edit is small: one new bullet, one new line, one new falsifier. The edit is the team’s incident learning.

A refinement example: the on-call engineer for a recent checkout-latency incident discovered that the dependency timeout was misconfigured in a way the template did not predict. The refinement adds a new hypothesis candidate to phase 3:

# Before refinement
hypothesis_candidates:
  - id: H1
    text: |
      Recent deploy changed <dependency> timeout / retry /
      connection configuration.
    falsifier: |
      F1. If dependency metric shows wait on downstream,
      H1 confirmed.
  - id: H2
    text: |
      <dependency> dependency is degraded independently.
    falsifier: |
      F2. If change log is clean, H2 more likely.

# After refinement
hypothesis_candidates:
  - id: H1
    text: |
      Recent deploy changed <dependency> timeout / retry /
      connection configuration.
    falsifier: |
      F1. If dependency metric shows wait on downstream,
      H1 confirmed.
  - id: H2
    text: |
      <dependency> dependency is degraded independently.
    falsifier: |
      F2. If change log is clean, H2 more likely.
  - id: H3  # Added in refinement
    text: |
      <dependency> timeout was raised independently of
      checkout timeout; checkout times out before dependency
      responds.
    falsifier: |
      F3. If checkout timeout < dependency effective timeout,
      H3 confirmed. Trace shows wait time matching checkout
      timeout, not dependency timeout.

The refinement is one commit, one bullet, one new hypothesis candidate. The git diff is the team’s incident learning.

Step 5. Commit the refinement

The commit message names the incident and the refinement:

runbooks/checkout: add H3 hypothesis for timeout drift

Refinement after incident 2026-08-13 (checkout-latency).
The actual root cause was a timeout configuration drift
(payment-svc timeout raised without corresponding raise in
checkout timeout). The template did not predict this.

Adds H3 with falsifier F3. Future checkout-latency
investigations will consider H3 from phase 3.

The commit is part of the team’s incident learning. The git log is searchable; future engineers can find the refinement by searching for the incident date or the template name.

How to configure it

The iteration is not Prometheus configuration; it is operational discipline. The closest configuration analogue is the post-incident review (PIR) template: the PIR is the artefact that captures the iteration and the team uses the template to run the PIR.

A PIR template that produces iteration:

# runbooks/templates/pir.yaml
title: 'Post-Incident Review: {{ incident_id }}'
sections:
  - name: 'Summary'
    fields:
      - 'Incident ID'
      - 'Incident class'
      - 'MTTM (minutes)'
      - 'MTTR (minutes)'
      - 'Customer-visible impact'
  - name: 'Timeline'
    fields:
      - 'Detection time (UTC)'
      - 'Engagement time (UTC)'
      - 'Mitigation time (UTC)'
      - 'Resolution time (UTC)'
  - name: 'Root cause'
    fields:
      - 'One-sentence description'
      - 'Which template hypothesis was confirmed'
  - name: 'What went well'
    fields:
      - 'Phases that worked as the template predicted'
  - name: 'What went wrong'
    fields:
      - 'Phases that diverged from the template'
      - 'New hypothesis candidates needed'
      - 'New evidence surfaces needed'
      - 'New falsifiers needed'
      - 'New root cause categories needed'
      - 'New mitigation categories needed'
  - name: 'Template refinements'
    fields:
      - 'Bullet list of refinements to commit'
  - name: 'Action items'
    fields:
      - 'Pre-deploy check additions'
      - 'Recording rule additions'
      - 'Alert additions'
      - 'Documentation updates'
      - 'Owner for each action'

How to validate it

Validate the iteration loop by inspecting the template version history and the recent PIRs:

# SEVERITY: READ-ONLY
# 1. Confirm the template has a version history.
cd ~/runbooks && git log --oneline -10 \
  runbooks/checkout/high-error-rate.yaml

Expected output:

a8f3c2d 2026-08-13  add H3 hypothesis for retry storm
7d1e9b4 2026-08-10  add follow-up for pre-deploy check
4f0a2c1 2026-08-08  initial template

# SEVERITY: READ-ONLY
# 2. Confirm the refinements name the incident.
cd ~/runbooks && git log --format="%h %s" \
  runbooks/checkout/high-error-rate.yaml \
  | grep -i 'incident'

Expected output:

a8f3c2d add H3 hypothesis for retry storm (incident 2026-08-13)
7d1e9b4 add follow-up for pre-deploy check (incident 2026-08-10)

# SEVERITY: READ-ONLY
# 3. Confirm the PIR templates are completed within 5 days
# of each incident.
cd ~/runbooks && find pirs -name "*.yaml" -newer pirs/.last-validated

Expected output:

pirs/2026-08-13-checkout-latency.yaml
pirs/2026-08-10-checkout-error-rate.yaml

The three checks confirm: the template has refinements (version history shows commits), the refinements name the incident (the commit messages reference the PIR), and the PIRs are completed in a timely manner (the file timestamps are within the team’s SLA).

The fourth validation is the operational one: a quarterly review finds that the templates have converged (no major refinements in the last quarter) or that the templates are still maturing (major refinements each month).

How it can fail

Five specific failure shapes for workflow iteration:

1. Iteration deferred to “after the incident”.” The on-call engineer closes the runbook log entry but defers the template refinement to “next week”. Next week, the engineer is on a different project; the refinement never happens. The template stays as it was. Symptom: the template has not been refined since the last major incident; the team is re-learning the same lessons.

2. Iteration as a narrative, not a refinement. The PIR contains a paragraph describing what went wrong, but no concrete refinement to the template. The next engineer reads the PIR and the template; the template is unchanged; the lesson is not captured. Symptom: the PIR has a “What went wrong” section but the template’s hypothesis_candidates list is unchanged.

3. Iteration that over-fits one incident. The refinement adds a hypothesis candidate that is specific to the incident (e.g., “the deploy at 09:35”). The next incident has a different deploy; the hypothesis candidate is irrelevant. Symptom: the hypothesis candidate reads as a runbook log entry, not as a reusable structure.

4. Iteration that drifts the template. Multiple refinements, each by a different engineer, each adding a small change. The template has grown to twice its original length; many hypothesis candidates are obsolete; the template is harder to read than it was before iteration. Symptom: the template is longer than five pages; new on-call engineers cannot read it in five minutes.

5. Iteration without a PIR. The on-call engineer refines the template directly, without writing a PIR. The refinement has no documented context; the next engineer who questions the refinement has no record of why it was added. Symptom: the git log shows refinements but no PIR files reference the refinements.

How to troubleshoot it

When the iteration loop is not producing refinements, the diagnostic order is:

1. Check the PIR completion rate. Are PIRs being written for each incident? If not, the iteration has no source material.
2. Check the refinement-to-narrative ratio. In the recent PIRs, is the “Template refinements” section populated with concrete refinements, or is it empty?
3. Check the template git log. Has the template been refined in the last 30 days? If not, the iteration is not happening.
4. Check the template drift. Is the template longer than it was three months ago? If yes by more than 50%, the template may need a rewrite (a new template version that consolidates the refinements).
5. Check the team ownership. Does each template have a team owner? If not, the iteration has no owner.

Security implications

The iteration does not change the security posture of the observability platform directly. The iteration may indirectly improve security by surfacing configuration invariants that were previously implicit (e.g., the timeout invariant in the checkout-latency worked example). The invariants, once documented as pre-deploy checks, are security-relevant because they prevent the configuration drift that could lead to availability incidents that have security implications (e.g., a long-running retry storm against a rate-limited dependency could be interpreted as a DoS by an external observer).

Performance implications

The iteration has no performance implications on the observability platform. The iteration is a documentation practice. The performance cost is in the time the on-call engineer spends writing the PIR and refining the template; the cost is bounded by the team’s SLA (typically fifteen to thirty minutes per incident) and is recovered on the next incident of the same class.

Production guidance

• Iteration is non-optional. Every incident produces a PIR; every PIR produces at least one refinement.
• Refinements are small. One bullet, one line, one commit. Large rewrites are new templates, not refinements.
• Refinements are committed with the PIR reference in the commit message. The git log is the iteration record.
• The template is reviewed quarterly. The review checks for drift, for missing hypothesis candidates, and for refinements that have not been incorporated.
• The team owns the template. Each template has a team owner; the team owner is responsible for the PIR completion and the quarterly review.
• A new template version is a major version bump. A refinement is a minor version bump. The distinction is important: a major bump consolidates multiple refinements into a cleaner structure; a minor bump adds one refinement.

Verification

You should now be able to answer:

• What is the difference between a refinement and a rewrite, and how is the distinction recorded in version control?
• What is the source material for an iteration, and what artefact captures it?
• Why is a refinement small (one bullet, one line) and not a paragraph or a rewrite?
• Why is the PIR a YAML file and not a Google Doc?
• How does the team know when a template has converged to a mature structure?

Quiz