Error rate and deploys — the deploy-to-error graph

The error rate panel shows the production error rate over time. The deploy marker shows when each release reached production. The join between the two is the deploy-to-error graph: a panel where every deploy is visible, and the error rate attributable to that deploy is queryable separately. The graph answers the question “is each deploy making the service better, worse, or unchanged?”.

Decomposing the error rate by version

The aggregate error rate - the rate of 5xx responses across all traffic - is the wrong primitive for deploy-caused error analysis. The aggregate hides which version produced the error. A service with 100 deploys in production over the last month will have a mix of versions running; the aggregate error rate is a weighted average across all of them.

The decomposition by service.version separates the versions:

NAME=api
SHA=$(git rev-parse --short HEAD)
MESSAGE="deploy $SHA release v3.4.1"
kubectl annotate deploy $NAME \
  kubernetes.io/change-cause="$MESSAGE" --overwrite
# error rate for the new version
# = rate(http_requests_total{service="$NAME",version=~"$SHA",code=~"5.."}[5m])
#   / rate(http_requests_total{service="$NAME",version=~"$SHA"}[5m])

The query returns the error rate for the specific version that was deployed. The aggregate query returns the same metric for all versions; the comparison is the difference. A deploy-caused error spike concentrates on the new version; the old version’s error rate stays at baseline.

flowchart TB
    A["Deploy marker\n14:21"] --> B["New version\nservice.version=$SHA"]
    A --> C["Old versions\nprevious releases"]
    B --> D["New version\nerror rate"]
    C --> E["Old versions\nerror rate"]
    D --> F{"New version\nspike?"}
    E --> G{"Old version\nspike?"}
    F -->|Yes| H["Deploy-caused\nerror spike"]
    F -->|No| I["Background\nnoise"]
    G -->|Yes| J["Unrelated\nshared dependency"]

The decomposition is the primitive that distinguishes “this deploy broke the service” from “the service is breaking anyway and the deploy is correlated by timestamp”.

The deploy-to-error graph

The graph is the panel that joins every deploy marker to its post-deploy error rate. The horizontal axis is time; the vertical axis is the error rate; each deploy appears as a vertical line; the post-deploy error rate is the colour or the line weight.

The graph reveals the structural pattern. A team that ships deploys whose post-deploy error rate is consistently low has a graph that is mostly green. A team that ships deploys whose post-deploy error rate spikes has a graph that is mostly red. The graph is the on-call interface for the question “is each deploy making the service better?”.

flowchart LR
    A["Deploy 1\n10:00"] --> B["Post-deploy\nerror rate 0.1%"]
    C["Deploy 2\n11:30"] --> D["Post-deploy\nerror rate 0.4%"]
    E["Deploy 3\n13:00"] --> F["Post-deploy\nerror rate 5.0%"]
    D --> G["Trend\nincreasing"]
    F --> G
    G --> H["Deploy-caused\nregression cluster"]

The graph is also the input to the change failure rate metric from the DORA lesson. Each spike in the graph contributes to the change failure rate; each stable deploy contributes to the deploy frequency. The two metrics share the same data source.

The structure of a deploy-caused error spike

Three structural properties distinguish a deploy-caused error spike from background noise:

1. Version concentration. The new version’s error rate spikes; the old version’s error rate stays at baseline. The spike is concentrated on the deploy that introduced it.
2. Timing. The spike appears within minutes of the deploy marker. The timing is consistent with the rollout speed; a slow rollout produces a slow rise, a fast rollout produces a fast rise.
3. Persistence. The spike persists across the observation window. A transient spike is a transient; a deploy-caused spike is sustained.

A spike that satisfies all three is a deploy-caused spike. A spike that satisfies one is a candidate; a spike that satisfies none is background noise.

Production discipline

1. Set service.version from the pipeline, not manually. The label is a property of the deploy, not of the operator’s memory.
2. Decompose the error rate by version in the dashboard. The aggregate hides the deploy-caused signal; the decomposition exposes it.
3. Calibrate alert thresholds per service, per deploy frequency. A single threshold across services produces alerts that over-page or under-page.
4. Surface the deploy-to-error graph on the on-call dashboard. The graph is the structural answer to “is each deploy making the service better?”.
5. Treat a persistent version-concentrated spike as a deploy-caused error spike. The version concentration is the strongest signal.

Cross-course references

• Observability course - Part LII (Exemplars) covers the version label that ties the metric to the deploy.
• Observability course - Part CX (MajorIncidents) covers the annotation surfaces that complement the version decomposition.
• This course, Part LXIII-05 (ChangeFailureRate) covers the DORA stability metric that the deploy-to-error graph feeds.
• This course, Part LX (Decision) covers the rollback decision that follows from a confirmed deploy-caused error spike.

Quiz