Performance change detection — did the deploy cause the regression

A latency regression appears at 14:23. The p99 panel goes from 120ms to 380ms. The deploy marker at 14:21 is the obvious suspect; obvious is not the same as correct. The investigator needs a structural way to decide whether the deploy caused the regression, or whether the deploy and the regression are correlated in time but not in cause.

The pre/post comparison window

A deploy-caused performance regression has a signature. Before the deploy, the latency distribution has a characteristic shape; after the deploy, the shape changes in a consistent direction. The comparison is two windows: a pre-deploy baseline window (typically 1 to 4 hours before the marker) and a post-deploy observation window (typically 30 minutes to 2 hours after the marker, depending on rollout speed).

NAME=api
SHA=$(git rev-parse --short HEAD)
MESSAGE="deploy $SHA release v3.4.1"
kubectl annotate deploy $NAME \
  kubernetes.io/change-cause="$MESSAGE" --overwrite
# baseline: histogram_quantile over the 4h before deploy
# observed: histogram_quantile over the 30m after deploy
# regression = consistent shift across p50, p95, p99

The histogram is the right primitive. A mean or median hides the tail; the tail is where regressions live. A p99 that goes from 120ms to 380ms while the p50 stays flat at 40ms is the signature of a slow path introduced by the deploy - a code path that handles a small fraction of requests but does so much more slowly.

The signature of a deploy-caused regression is a consistent direction shift across percentiles. p50, p95, and p99 all move in the same direction. A single-percentile spike is more often a transient - a slow garbage collection, a noisy neighbour, a single slow upstream - than a deploy-caused regression.

flowchart TB
    A["Deploy marker\n14:21"] --> B["Pre-deploy window\n4h baseline"]
    A --> C["Post-deploy window\n30m observation"]
    B --> D["p50, p95, p99\nbaseline"]
    C --> E["p50, p95, p99\nobserved"]
    D --> F["Compare shapes"]
    E --> F
    F --> G{"All percentiles\nshift in same direction?"}
    G -->|Yes| H["Deploy-caused\nregression candidate"]
    G -->|No| I["Likely transient\nor unrelated"]

Exemplars link the slow request to the deploy

The histogram tells the investigator that a regression exists; the histogram does not tell the investigator which request is slow, on which code path, running which version. The exemplar is the link from the aggregate metric to the single distributed trace.

A Prometheus exemplar is a reference to a trace attached to a histogram bucket. The OpenTelemetry SDK emits the exemplar when it records a measurement; the trace backend stores the full trace; the Grafana panel shows the exemplar as a clickable diamond on the histogram. Clicking the diamond opens the trace.

The exemplar carries the resource attributes the investigator uses to identify the deploy:

• service.version - the version label of the workload that handled the request.
• deployment.environment - the environment label.
• k8s.deployment.name - the Kubernetes deployment name.

A trace whose service.version matches the deploy annotation’s commit SHA is a trace that ran the new code. A trace that ran the old code is a trace from the previous version. The exemplar makes the distinction visible at the panel.

False-positive modes that must be filtered

Three false-positive modes produce a “deploy caused the regression” signal that the investigator must reject:

1. Correlated but unrelated. A deploy at 14:21 and a regression at 14:23 are correlated by timestamp but unrelated in cause. The pre/post comparison across percentiles must agree; if only one percentile moves, the correlation is coincidence.
2. Transient. A regression that lasts 5 minutes and recovers is a transient - a slow garbage collection, a noisy neighbour, a brief upstream slowdown. The post-deploy window must be long enough to confirm the regression persists; a 5-minute observation is not enough.
3. Partial rollout. A canary deploy that shifts p99 for a fraction of traffic produces a regression signal that is real but bounded. The investigator must compare against the rollout percentage and the canary’s blast radius, not against the full population. A partial-rollout regression is a canary success, not a regression.

The investigator who applies all three filters produces a deploy-caused regression decision that is defensible; the investigator who applies none produces a decision that is correlated but unreliable.

Production discipline

1. Instrument every latency-sensitive endpoint with a histogram, not a mean. The mean hides the tail; the histogram exposes it.
2. Set service.version on every workload from the pipeline. The attribute is the key that links the exemplar to the deploy.
3. Compare percentiles, not single points. A consistent direction shift across p50, p95, p99 is the deploy-caused signature.
4. Confirm the regression persists across the rollout window. A 5-minute spike is a transient; a 30-minute sustained shift is a regression.
5. Filter the three false-positive modes before declaring the regression deploy-caused. The decision is structural, not intuitive.

Cross-course references

• Observability course - Part LII (Exemplars) covers the metric-to-trace pivot and the service.version attribute that the exemplar carries.
• Observability course - Part LV (DashboardToTraces) covers the Grafana data source wiring that turns a histogram diamond into a navigation event.
• This course, Part LVIII (DeploymentPatterns) covers the canary and progressive delivery patterns that produce the partial-rollout false positive.
• This course, Part LX (Decision) covers the rollback versus forward-fix decision that follows from a confirmed deploy-caused regression.

Quiz