Workflow Templates

After six months of on-call, the team has fifty runbook log entries. The first three are long and detailed; the engineer who wrote them was learning the loop. The next twenty are shorter and follow a recognisable structure: symptom, impact, hypothesis, evidence, test, root cause. The next twenty-five are even shorter and follow a recognisable incident class: checkout-latency, error-rate-spike, storage-full, dependency-outage. The team is no longer inventing the structure each time. The team is filling in a template.

This lesson is the template. The discipline is to write one template per incident class, to fill it in for every incident of that class, and to refine it after every incident.

What it is

A workflow template is a pre-filled runbook entry with phases 1, 2, and 3 specified for an incident class. Phase 4 (evidence) is left blank because the evidence depends on the specific incident; phases 5 (test) and 6 (root cause) follow from phase 4 and are not pre-filled.

The template is a starting point, not a straitjacket. The on-call engineer fills in the template; if the template does not match the incident, the engineer rewrites the template during phase 6 and records the new template for the next incident of the same class.

Why a sysadmin cares

A template reduces MTTM for the second incident of an incident class. The on-call engineer for the second incident opens the runbook, reads the template, and follows it. The phases that took thirty minutes of trial-and-error on the first incident now take five minutes of filling-in.

The template also reduces the variance in MTTM across the on-call rotation. The junior engineer who has never seen a checkout-latency incident follows the same template as the senior engineer who has seen twenty. The template is the shared operational practice.

How it works

The template is structured around the six phases, with phases 1-3 pre-filled and phases 4-6 left blank for the specific incident. The structure is identical to the runbook log entry; the template is a runbook log entry with the incident-specific fields blank.

The template skeleton

# Runbook: <incident-class>

## Symptom (Phase 1)
<incident-class symptom template>

## Impact (Phase 2)
<incident-class impact template>

## Hypothesis candidates (Phase 3)
H<index>. <hypothesis template>
F<index>. <falsifier template>

## Evidence (Phase 4)
- <evidence surface 1>
- <evidence surface 2>
- <evidence surface 3>
- <evidence surface 4>

## Test (Phase 5)
<test template>

## Root cause + mitigation (Phase 6)
<root-cause categories>
<mitigation categories>

## Follow-up
<follow-up categories>

Each incident class has its own template. The four canonical incident classes and their templates are below.

Template: CheckoutLatencyAboveSLO

# Runbook: CheckoutLatencyAboveSLO

## Symptom (Phase 1)
<service> p95 latency <value>s in <region> since <time>,
SLO <value>s. 5xx rate <value>% (within / outside band).
Synthetic probe and customer reports confirm.

## Impact (Phase 2)
5-minute SLO burn rate <value>; ~<value>% of 30-day error
budget in <value> minutes. Budget exhaustion projected in
<value> hours at current rate.

## Hypothesis candidates (Phase 3)
H1. Recent deploy changed <dependency> timeout / retry /
connection configuration. Latency regression aligns with
deploy window.
H2. <dependency> dependency is degraded independently.
F1. If dependency metric shows wait on downstream (not
handler CPU), H1 confirmed. If dependency error rate
elevated, H2 more likely.
F2. If change log shows deploy in last hour, H1 more likely.
If change log is clean, H2 more likely.

## Evidence (Phase 4)
- service metric by span: <expected distribution>
- dependency metric: <expected value>
- single trace: <expected span timings>
- logs: <expected log lines>

## Test (Phase 5)
Compare hypothesis prediction against evidence. Confirmed
or refuted.

## Root cause + mitigation (Phase 6)
Root cause categories:
- timeout configuration regression
- retry storm
- connection pool misconfiguration
- dependency degradation
Mitigation categories:
- rollback deploy
- revert configuration
- scale dependency
- extend SLO window

## Follow-up
- pre-deploy check on configuration invariant
- recording rule on dependency metric
- ticket-class alert on regression indicator

Template: ServiceHighErrorRate

# Runbook: ServiceHighErrorRate

## Symptom (Phase 1)
<service> 5xx rate <value>% in <region> since <time>, SLO
<value>%. Synthetic probe confirms. Customer reports
<verbatim or paraphrase>.

## Impact (Phase 2)
5-minute SLO burn rate <value>; ~<value>% of 30-day error
budget in <value> minutes.

## Hypothesis candidates (Phase 3)
H1. Recent deploy introduced regression. Error rate aligns
with deploy window.
H2. <dependency> dependency is down or rate-limited.
F1. If 5xx distribution is dominated by 4xx-from-dependency
(e.g., 429, 503), H1 more likely. If distribution is
dominated by 5xx-from-dependency, H2 more likely.
F2. If change log shows deploy in last hour, H1 more likely.

## Evidence (Phase 4)
- 5xx rate by status code: <expected distribution>
- dependency metric: rate-limit, error, saturation counters
- logs: error pattern, retry exhaustion, dependency timeout
- change log: deploys in last 24 hours
- single trace: span where the error originated

## Test (Phase 5)
Confirmed or refuted by evidence.

## Root cause + mitigation (Phase 6)
Root cause categories:
- deploy regression (retry, timeout, connection pool)
- dependency outage
- dependency rate-limit
- configuration drift
Mitigation categories:
- rollback deploy
- revert configuration
- restart dependency
- scale dependency

## Follow-up
- pre-deploy check on retry/timeout invariant
- recording rule on dependency rate-limit counter
- ticket-class alert on retry exhaustion

Template: ObservabilityStorageFull

# Runbook: ObservabilityStorageFull

## Symptom (Phase 1)
<component> scraping stopped at <time>. Grafana dashboards
flatline from <time>. <component> API returns 503 "storage
unavailable". Disk on <mountpoint> at <value>%.

## Impact (Phase 2)
Platform blind for <value> minutes at time of detection.
Cannot alert; cannot investigate other incidents.

## Hypothesis candidates (Phase 3)
H1. Retention setting was changed to 0s (or longer than
volume can sustain). Disk fills because volume sized for
previous retention.
H2. Cardinality incident introduced unexpected series growth.
F1. If retention is 0s and head series count is stable, H1
confirmed. If retention is unchanged and head series count
doubled, H2 more likely.
F2. If block count is consistent with no deletion, H1 more
likely. If block count is consistent with configured
retention, H2 more likely.

## Evidence (Phase 4)
- host disk: <expected usage>
- directory breakdown: blocks, chunks_head, wal
- retention config: <expected value>
- git history: <expected commit>
- head series count: <expected band>
- block count: <expected number>

## Test (Phase 5)
Compare hypothesis prediction against evidence.

## Root cause + mitigation (Phase 6)
Root cause categories:
- retention configuration regression
- cardinality incident
- volume undersized for configured retention
Mitigation categories:
- extend volume
- revert retention
- restart component
- prune head blocks (lossy; last resort)

## Follow-up
- pre-deploy check on retention setting
- self-monitoring rule on disk usage
- alert at 80% (ticket) and 95% (page)
- document retention / volume coupling

Template: DependencyOutage

# Runbook: DependencyOutage

## Symptom (Phase 1)
<service> 5xx rate elevated in <region> since <time>.
Synthetic probe fails. Customer reports mention
<service>-specific failure mode.

## Impact (Phase 2)
<service>-specific impact calculation.

## Hypothesis candidates (Phase 3)
H1. <dependency> dependency is down or degraded
independently.
H2. <service> deploy introduced regression that surfaces as
dependency failure.
F1. If dependency metric shows elevated error rate at
<service>-independent time, H1 confirmed. If dependency
error rate aligns with <service> deploy window, H2 more
likely.

## Evidence (Phase 4)
- <dependency> error rate panel
- <dependency> saturation panel (CPU, memory, connection pool)
- <dependency> recent change log
- <service> deploy correlation
- single trace: dependency span error pattern

## Test (Phase 5)
Compare against dependency metric.

## Root cause + mitigation (Phase 6)
Root cause categories:
- dependency outage
- dependency saturation
- dependency configuration drift
Mitigation categories:
- failover to backup dependency
- scale dependency
- revert <service> deploy (if H2)
- communicate with dependency owners

## Follow-up
- recording rule on dependency saturation
- alert on dependency error rate
- failover automation

How to configure it

The template is not Prometheus configuration; it is operational discipline. The closest configuration analogue is the alert rule annotation set: the alert that pages on the incident class must include the runbook_url annotation that points to the rendered template. The on-call engineer clicks the URL from the page message and is at the template within seconds.

An alert rule that respects the template:

groups:
- name: checkout.rules
  rules:
  - alert: CheckoutHighErrorRate
    expr: |
      sum by (region) (
        rate(http_requests_total{job="checkout",code=~"5.."}[5m])
      )
      / sum by (region) (
        rate(http_requests_total{job="checkout"}[5m])
      ) > 0.05
    for: 5m
    labels:
      severity: page
      team: payments
      service: checkout
    annotations:
      summary: 'Checkout 5xx rate above 5% in {{ $labels.region }}'
      runbook_url: 'https://runbooks.example.com/checkout/high-error-rate'

The runbook_url is the template. The on-call engineer clicks the URL, reads the template, fills it in, and writes the result back to the same URL (creating a new version).

A template file in the runbooks repository:

# runbooks/checkout/high-error-rate.yaml
title: 'Runbook: Checkout High Error Rate'
incident_class: ServiceHighErrorRate
service: checkout
template:
  symptom: |
    <service> 5xx rate <value>% in <region> since <time>,
    SLO <value>%. Synthetic probe and customer reports confirm.
  impact: |
    5-minute SLO burn rate <value>; ~<value>% of 30-day error
    budget in <value> minutes.
  hypothesis_candidates:
    - id: H1
      text: |
        Recent deploy introduced regression. Error rate aligns
        with deploy window.
      falsifier: |
        F1. If 5xx distribution is dominated by 4xx-from-
        dependency (e.g., 429, 503), H1 more likely.
    - id: H2
      text: |
        <dependency> dependency is down or rate-limited.
      falsifier: |
        F2. If change log is clean, H2 more likely.
  evidence_surfaces:
    - '5xx rate by status code'
    - 'dependency metric: rate-limit and error counters'
    - 'logs: error pattern, retry exhaustion'
    - 'single trace: span where error originated'
  root_cause_categories:
    - 'deploy regression (retry, timeout, connection pool)'
    - 'dependency outage'
    - 'dependency rate-limit'
    - 'configuration drift'
  mitigation_categories:
    - 'rollback deploy'
    - 'revert configuration'
    - 'restart dependency'
    - 'scale dependency'
  follow_up_categories:
    - 'pre-deploy check on retry/timeout invariant'
    - 'recording rule on dependency rate-limit counter'
    - 'ticket-class alert on retry exhaustion'

How to validate it

Validate the template by inspecting the runbook rendering and the alert rule:

# SEVERITY: READ-ONLY
# 1. Confirm the runbook URL resolves.
curl -sI 'https://runbooks.example.com/checkout/high-error-rate' \
  | head -n 1

Expected output:

HTTP/2 200

# SEVERITY: READ-ONLY
# 2. Confirm the alert rule references the runbook URL.
curl -s 'http://prometheus:9090/api/v1/rules' \
  | jq '.data.groups[].rules[] | select(.name=="CheckoutHighErrorRate") | .annotations.runbook_url'

Expected output:

"https://runbooks.example.com/checkout/high-error-rate"

# SEVERITY: READ-ONLY
# 3. Confirm the template file exists in the runbooks repo.
cd ~/runbooks && git log --oneline -5 \
  runbooks/checkout/high-error-rate.yaml

Expected output:

a8f3c2d 2026-08-12  add retry-exhaustion follow-up
7d1e9b4 2026-08-10  initial template

The three checks confirm: the template is rendered (HTTP 200), the alert rule references the template (the URL matches), and the template has a version history (the git log shows refinements).

The fourth validation is the operational one: a monthly audit of the last 30 days of runbook log entries finds that each incident of a known class was filled against the existing template, and each incident of a new class produced a new template.

How it can fail

Five specific failure shapes for workflow templates:

1. Templates that over-fit one incident. The first incident of an incident class produces a template that captures the specifics of that incident (e.g., “the deploy at 09:35”). The template is then useless for the next incident, which has different specifics. Symptom: the template reads as a runbook log entry, not as a reusable structure.

2. Templates that under-fit the class. The template is so generic (e.g., “investigate the issue”) that it does not help the on-call engineer. The on-call engineer re-derives the structure on each incident. Symptom: the template does not name the hypothesis candidates or the evidence surfaces.

3. Templates that are not referenced by alerts. The templates exist in the runbooks repository but the alert rules do not include the runbook_url annotation. The on-call engineer does not see the template from the page message. Symptom: the template has zero views in the runbooks analytics; the alert rules do not name the URL.

4. Templates that drift from the alerts. The alert rule is updated (new label, new threshold) but the template is not. The template references a label that no longer exists. Symptom: the on-call engineer follows the template and looks for a label that is not on the firing alert.

5. Templates that are not versioned. The template is edited through a wiki WYSIWYG; the edits are not diff-able; the team’s incident learning is invisible. Symptom: the template’s last-modified date is in the distant past; the team’s incident class has changed multiple times since.

How to troubleshoot it

When a template is not being followed, the diagnostic order is:

1. Confirm the template is rendered. Open the runbook URL. Does it return 200? Does the rendered page show the template structure?
2. Confirm the alert rule references the template. The runbook_url annotation should match the template URL. A mismatch is a configuration bug.
3. Confirm the template structure matches the incident class. The four canonical classes (latency, error rate, storage, dependency outage) are the baseline; if the incident does not match, the template needs a new class.
4. Audit recent runbook entries. Compare the template against the entries filled in the last 30 days. Where the template was not followed, identify the gap.
5. Refine the template. The template is a living document. After each incident, refine it. After ten refinements, the template is mature.

Security implications

The template references the alert rule labels, the dependency names, and the dashboard URLs. None of these are sensitive on their own; the template is a public document. The runbook log entries that are filled against the template may include sensitive information (the user-visible error count, the specific deploy SHA). The template’s structure does not constrain the entries; the entry should be sanitised before publication if the runbook is internal-only.

Performance implications

The template has no performance implications on the observability platform. The template is a document, not a query. The performance cost is in the time the on-call engineer spends filling it in; the cost is bounded by the incident duration and is recovered on the next incident of the same class.

Production guidance

• The template is a YAML file in a version-controlled repository, not a wiki page. The version history is the team’s incident learning.
• The template references the canonical incident classes (latency, error rate, storage, dependency outage). A new class is added when an incident does not match any of the four.
• The alert rule’s runbook_url annotation points to the template. The on-call engineer clicks the URL from the page message and is at the template within seconds.
• The template is refined after each incident. The refinement is a new version; the old version is retained.
• The template is reviewed monthly. The review finds templates that have drifted from the alerts and templates that no longer match the incident class.
• The template is owned. Each template has a team owner; the team owner is responsible for the monthly review.

Verification

You should now be able to answer:

• What three phases of the loop does the template pre-fill, and what three phases does it leave blank?
• Why is the template a YAML file and not a wiki page?
• What is the difference between a template that over-fits one incident and one that under-fits the class?
• How does the alert rule connect to the template?
• When is a new template (a new incident class) added?

Quiz