The CI observability question — what gets measured, what doesn't, and the failure modes

A pipeline that has been green for six months can still be a pipeline that is unhealthy. A test suite whose average duration has crept from 90 seconds to 9 minutes is failing even though every run says success. A flaky test that passes on its third attempt is failing even though the run page says success. A queue whose depth is 14 is failing even though the deploy that is running is success. The green tick is one bit; CI observability is the layer beneath it.

What gets measured, what doesn’t

A CI system naturally records a small set of facts about every run: the commit, the trigger, the start time, the end time, the status (success, failure, cancelled), and the conclusion. These are the data points the run page surfaces by default. The gh run list command returns exactly this shape:

gh run list --json status,conclusion,databaseId
# returns: a JSON array of {status, conclusion, databaseId}
# for every run the authenticated user can see

What this list does not return is the operational signal that distinguishes a healthy pipeline from a degrading one. The list does not include the queue depth at submission, the number of retries per step, the per-step duration, the annotation count, or the artifact size. These metrics exist in the system; they just do not surface in the default view.

flowchart LR
    A["Pipeline run"] --> B["Naturally recorded:\nstatus, duration, conclusion"]
    A --> C["Operational signal:\nqueue depth, retries, step time, flake rate"]
    B --> D["Green tick\n1 bit"]
    C --> E["Observability\nmany bits"]

The green tick is the bit the system optimises for. The operational signal is what an infrastructure engineer needs to answer the question “is this pipeline getting better or worse?”. The two views are not the same view, and a team that only sees the first is a team that is surprised when the second crosses a threshold.

The four failure modes

Four failure modes appear when a team treats the green tick as evidence of pipeline health:

1. Latency creep without failure. The pipeline is green; the duration is trending up. The next failure is a timeout that nobody predicted because nobody watched the trend.
2. Flake masking with retries. A flaky test that succeeds on the third attempt is a passing test from the system’s perspective. The run is green; the suite is failing 30% of the time. The team’s confidence in the suite is misplaced.
3. Queue depth invisible to the runner. A serialised pipeline whose queue depth is 12 is a pipeline whose latency for the deploy at position 12 is 12 × service time. The runner metrics show “running” for an extended time; the dashboard shows “slow deploys today”; nobody sees the queue.
4. Annotate-and-ignore. A step that emits 50 warnings today emitted 5 six months ago. The run is green; the warnings are warnings, not errors. The next failure is when the warning becomes an error - and by then the team has forgotten what the warnings were about.

What observability is, formally

Observability is the ability to answer questions about a system from its outputs. In the SRE Book, observability is built from three signals: metrics (the aggregate numbers over time), logs (the discrete events), and traces (the causal paths). For a CI pipeline:

• Metrics are the counts and durations: runs per hour, success rate, median duration, queue depth, retry count, artifact size.
• Logs are the per-step execution trace: timestamped stdout and stderr for each command.
• Traces are the run-to-deploy path: which commit triggered which run, which run produced which artifact, which artifact was deployed by which change.

A pipeline that emits only logs is a pipeline that can be debugged but not measured. A pipeline that emits only metrics is a pipeline that can be measured but not debugged. A pipeline that emits all three is a pipeline that is observable - the team can answer operational questions without re-executing the run.

Why this comes first

This lesson is the first in the part because every subsequent lesson - queue metrics, success rate, deployment frequency, change failure rate, MTTR - is a specific answer to a specific operational question. None of those metrics matter if the team is not in the habit of asking the question. A team that measures success rate without first defining what “success” means is a team that will optimise for the wrong number. A team that measures deployment frequency without first understanding what a deployment is will count wrong deploys.

Production discipline

1. Treat the green tick as a coarse filter, not a metric. The green tick tells you whether to investigate; it does not tell you whether the system is healthy.
2. Surface operational metrics by default. Queue depth, retry count, step duration, annotation count should be on a dashboard, not buried in a log file.
3. Distinguish “run succeeded” from “system is healthy”. They are different questions; they deserve different signals.
4. Audit the metrics the system emits by default. Every metric the system exposes is a candidate; every metric the system does not expose is a gap to fill.

Cross-course references

• Observability course - Parts I (Foundations) and VIII (Metrics) cover the formal definition of metrics versus logs versus traces, and how the three compose into a observable system.
• This course, Part XXXIX-05 (PipelineStatus) covers the run page as the operational surface for a single run; this part covers the aggregate view across runs.
• This course, Part LXII-05 (Queueing) covers the queue as a serialisation mechanism; this part covers the queue as a metric.

Quiz