The 3 AM incident and the deploy marker — the real test

The marker discipline from the previous five lessons is abstract until 3 AM. At 3 AM the on-call engineer is paged for a regression in a service the engineer has never touched. The engineer opens the dashboard, sees the spike, and asks the first question of every incident: “what changed?”. The answer is on the timeline if the team has the marker discipline. The answer is in the CI system by hand if the team does not. The difference is the difference between a fast MTTR and a slow one.

The page arrives

The page arrives at 03:14 from the alerting system. The service is api; the alert is “error rate above 5% for 5 minutes”. The engineer opens the on-call dashboard, which is the joined view from the previous lesson: metric panels filtered by version, an Events panel with the deploy events, and a Loki panel with the error log lines. The spike is visible on the error rate panel.

The engineer’s first action is to read the panel. The panel shows three time series: v3.4.0 (the previous version), v3.4.1 (the current version), and v3.3.7 (an older version still receiving a small percentage of traffic from a canary). The error rate for v3.4.1 spiked at 03:09 from 0.1% to 8%; the other versions are flat. The regression is in v3.4.1.

flowchart TB
    A["Page arrives\n03:14"] --> B["Open joined view\nfiltered by version"]
    B --> C["v3.4.1 error rate\nspiked at 03:09"]
    C --> D["Identify suspect version\nv3.4.1"]
    D --> E["Identify previous version\nv3.4.0"]
    E --> F["Find deploy event\ntimestamp 02:47"]
    F --> G["Confirm rollback\ntarget version"]
    G --> H["Roll back\n01:32 elapsed"]

The marker surfaces the cause

The engineer reads the Events panel. The panel shows the deploy event for v3.4.1 at 02:47, with the annotation text “deploy 7a3f9d2 build 4527 release v3.4.1”. The event payload carries the SHA (7a3f9d2), the build number (4527), the actor (octocat), the build URL, and the environment (production). The engineer clicks the build URL; the CI run opens in a new tab.

The CI run shows the diff: the change introduced a new database query path that is not connection-pooled. The query exhausts the connection pool under load and returns 500 errors. The change-cause annotation in the pipeline log says “add new analytics query path”. The engineer has the cause, the change, the actor, and the approval chain in three clicks.

The Loki panel filtered by service_version="v3.4.1" shows the error log lines: “connection pool exhausted after 50 connections, request rejected”. The Tempo panel filtered by the same attribute shows the failing spans, all hitting the new analytics endpoint. The joined view shows the regression, the cause, the failing endpoint, and the deploy context in one screen.

The rollback decision

The engineer has the cause and the previous version. The rollback is a single kubectl rollout undo command or, in a GitOps workflow, a revert commit and a sync. The engineer announces the rollback in the incident channel, rolls back to v3.4.0, and watches the error rate return to baseline by 03:16. Total elapsed time: 1 minute 32 seconds from page to recovery.

The rollback is reliable because the marker surfaces the target version. The engineer does not have to guess which version is the previous one; the Events panel shows the deploy history and the engineer rolls back to the version immediately preceding the suspect. The rollback is auditable because the event log records the rollback action with the actor, the timestamp, and the target version; the postmortem will have the complete record of the incident and the recovery.

What the marker preserved

The marker preserved six things the on-call engineer needed in order:

1. The suspect version. The version label filtered the metric panel to v3.4.1; the engineer identified the regression in one click.
2. The deploy event. The Events panel showed the structured record with the SHA, the build number, the actor, the build URL, and the environment.
3. The change-cause. The annotation text described the change in human-readable form; the engineer read it on the panel.
4. The previous version. The version label on the previous deploy was queryable; the engineer identified the rollback target in one click.
5. The failing endpoint. The Loki and Tempo panels filtered by service_version="v3.4.1" showed the failing spans; the engineer confirmed the cause.
6. The audit trail. The rollback action was recorded in the event log; the postmortem has the complete record.

A team that has the marker discipline preserves all six. A team that does not have the marker discipline preserves one - the human-readable message in the annotation - and reconstructs the other five by hand.

What the marker missed

The postmortem captures what the marker preserved and what it missed. The marker missed three things in this incident:

• The connection-pool size at the time of the change. The diff showed the new query path but did not flag the connection-pool impact. The marker preserves the change; the marker does not preserve the capacity impact.
• The load test that should have caught the regression. The CI run included unit tests and integration tests but no load test. The marker preserves the build; the marker does not preserve the test gap.
• The rollback runbook. The team has the rollback procedure documented but the engineer had to reconstruct the exact command from memory. The marker preserves the deploy; the marker does not preserve the operational muscle memory.

The postmortem feeds these gaps back into the pipeline: add a load test to CI that exercises the connection pool, add a connection-pool-impact check to the pre-flight-validation, and add the rollback command to the on-call runbook with a link to the marker query.

Production discipline

1. Open the joined view first during an incident. The metric, the log, the trace, and the deploy event are all on one panel; the engineer does not have to cross-reference systems by hand.
2. Identify the suspect version by the metric filtered by version. The version label is the fastest path from spike to cause.
3. Roll back to the version immediately preceding the suspect. The Events panel shows the deploy history; the rollback target is the previous entry.
4. Announce the rollback in the incident channel with the marker reference. The annotation text and the build URL are the audit chain for the recovery.
5. Capture what the marker missed in the postmortem. The gaps are the inputs to the next round of marker improvements.

Cross-course references

• This course, Part CXIII (ObservabilityIntegration) covers the timeline view that the joined view extends.
• This course, Part CVI (ChangeMgmt) covers the change record and the postmortem discipline that feed the marker gaps back into the pipeline.
• Observability course - Parts covering major incidents cover the on-call interface that the joined view implements.

Quiz