Change failure rate and MTTR — the second pair of DORA metrics and how to interpret them

The first pair of DORA metrics - deployment frequency and lead time - answers the question “how fast does this team ship?”. The second pair - change failure rate and mean time to restore - answers the question “how safe is this team’s shipping?”. The two pairs are not a trade-off: the teams that ship fast are the teams that ship safely, because the practices that produce speed (small changes, automated tests, fast feedback) are the same practices that produce stability.

Change failure rate

Change failure rate is the fraction of deploys to production that result in a degraded service and require remediation (typically a rollback, a hotfix, or a patch). A deploy that fails in CI is not counted; the deploy is the one that reaches production.

The metric is computed from two data sources: the deploy count (from the CI system) and the incident count (from the incident management system). For an infrastructure team, “incident” can mean a PagerDuty page, a customer-facing outage, a rollback executed by the on-call, or a post-deploy fix that addresses a regression introduced by the deploy:

OWNER=acme
REPO=platform
gh api repos/$OWNER/$REPO/actions/runs \
  --jq '[.workflow_runs[] | select(.name == "deploy-production" and .conclusion == "success")] | length'
# returns: production deploys in the window
# divide: deploys that caused an incident / total deploys

The DORA performance bands for change failure rate:

• Elite: 0-15%.
• High: 16-30%.
• Medium: 31-45%.
• Low: 46-60%.

A 5% change failure rate means 1 in 20 production deploys causes a regression. A 50% change failure rate means 1 in 2 deploys causes a regression - a team that is shipping broken code half the time, regardless of how fast it ships.

The metric must be interpreted carefully:

• A deploy that fails in CI does not count. The deploy never reached production; the pipeline’s job is to catch it.
• A deploy that triggers a rollback within 24 hours counts. The deploy reached production, caused a regression, and required remediation.
• A deploy that triggers a fix within a week counts. The deploy reached production, caused a latent bug, and required remediation.
• A deploy that triggers a planned follow-up does not count. The follow-up is a planned improvement, not a remediation of a regression.

The distinction matters because the change failure rate is supposed to measure the cost of shipping, not the cost of working. A team that conflates the two will inflate the metric and lose the diagnostic value.

Mean time to restore

Mean time to restore (MTTR) is the median time from a production incident to the restoration of service. The metric is computed from the incident management system: the timestamp of the page (or the first alert) and the timestamp of the resolution (or the all-clear).

The DORA performance bands for MTTR:

• Elite: Less than one hour.
• High: Less than one day.
• Medium: Less than one week.
• Low: More than one week.

For an infrastructure team, MTTR is the operational signal that answers “how fast does this team recover?”. A team with MTTR of 4 hours is a team that has runbooks, automated rollback, and a practiced on-call rotation. A team with MTTR of 3 days is a team that has to investigate each incident from scratch.

flowchart TB
    A["Incident"] --> B["Detection\nfirst alert or page"]
    B --> C["Triage\nidentify scope"]
    C --> D["Mitigation\nstop the bleeding"]
    D --> E["Resolution\nrestore service"]
    E --> F["Postmortem\nlearn from the incident"]

The MTTR measures the time from A to E, not from A to F. The postmortem is a separate signal (often called “mean time to learn” or MTTL) and is not part of MTTR.

The four quadrants

The two stability metrics compose with the two throughput metrics into four quadrants:

The dangerous quadrant is the quadrant to watch for. A team that ships often but breaks things often is a team that has optimised for speed without investing in safety. The team’s practices (small changes, automated tests, fast feedback) are not in place; the team is shipping large changes without tests, and each change is a roll of the dice.

Production discipline

1. Track change failure rate from incident data, not from CI failures. A deploy that fails in CI is the system working; a deploy that fails in production is the metric.
2. Track MTTR as the median, not the mean. A single multi-day incident inflates the mean and hides the baseline. The median is the operational signal.
3. Treat a high change failure rate as a structural problem, not a personal one. The cause is usually a missing practice (tests, review, staging), not a missing skill.
4. Rehearse the recovery. A team that has rehearsed its rollback procedure will have a lower MTTR than a team that figures it out under pressure.
5. Pair throughput and stability metrics. The four DORA metrics together are the picture; any one alone is incomplete.

Cross-course references

• Observability course - Part X (Incidents) covers the MTTR signal in detail; Part III (BurnRate) covers the alert discipline that affects detection time.
• This course, Part LIX (Rollback) covers the rollback discipline that drives MTTR; a team with automated rollback has a structural advantage.
• This course, Part LX (Decision) covers the rollback versus forward-fix decision that affects MTTR; a team that decides quickly has a structural advantage.

Quiz