Queue and runtime metrics — what they reveal and the alert thresholds that matter

A pipeline run produces two timing signals: how long it ran when it ran (runtime duration), and how long it waited before it ran (queue depth). The two together are the operational truth of the pipeline. A run with a short runtime and a deep queue is a run that is healthy but slow. A run with a long runtime and a shallow queue is a run that is fast at the moment but approaching the timeout. A run with both signals high is a run that is failing without yet failing.

Runtime duration

Runtime duration is the wall-clock time from the moment the runner picks up the job to the moment the job exits. For a GitHub Actions run, the duration is exposed through the API:

OWNER=acme
REPO=platform
gh api repos/$OWNER/$REPO/actions/runs \
  --jq '.workflow_runs[] | {id: .id, status: .status, conclusion: .conclusion, run_started_at: .run_started_at, updated_at: .updated_at}'
# returns: id, status, conclusion, run_started_at, updated_at
# the duration is computable as updated_at - run_started_at

The runtime duration is the signal that predicts timeouts. Most hosted runners have a maximum job duration (60 minutes for GitHub-hosted Linux runners, 120 minutes for self-hosted with elevated limits). A job whose duration is consistently under the budget is a job that has headroom; a job whose duration trends toward the budget is a job that has no headroom and is one dependency update away from a timeout.

flowchart TB
    A["Runtime duration"] --> B["< 50% of timeout\nhealthy, headroom available"]
    A --> C["50-80% of timeout\nwarning zone, watch the trend"]
    A --> D["80%+ of timeout\nalert zone, fix before next run"]
    A --> E["100% of timeout\ntimeout fires, job fails"]

The right alert threshold for runtime duration is 80% of the timeout budget. A duration at 80% means a 25% growth in the slowest step will fire the timeout. A duration at 50% means a 100% growth is required to fire the timeout. A team that alerts at 50% is alert-fatiguing itself on runs that have room to grow; a team that alerts at 95% is finding out about the timeout from the failed run page.

Queue depth

Queue depth is the number of runs waiting to start. For a serialised pipeline (a pipeline with a concurrency group or a state lock), queue depth is the cost of safety: a queue depth of 12 means the run at position 12 waits for 12 service times before it starts. For an unserialised pipeline, queue depth is the runner-pool saturation signal: a queue depth of 12 means the runner pool is exhausted and runs are backing up.

Queue depth is not exposed by the GitHub Actions API in a single field. The team computes it as the number of runs in queued or waiting status at a point in time, or as the delta between run_started_at and the moment the trigger fired:

OWNER=acme
REPO=platform
gh api repos/$OWNER/$REPO/actions/runs \
  --jq '[.workflow_runs[] | select(.status == "queued")] | length'
# returns: the number of runs in queued status at this moment

The right alert threshold for queue depth depends on the pipeline’s expected throughput. For a serialised deploy pipeline, a sustained queue depth above 3 is a signal that the arrival rate exceeds the service rate; the queue will grow without bound. For an unserialised test pipeline, a sustained queue depth above the runner count is the same signal: the runner pool is exhausted.

The two signals together

The two signals are not independent. A pipeline with a growing queue has a growing latency; a pipeline with a growing runtime has a shrinking capacity. The pair composes:

• Queue growing, runtime stable. The arrival rate is exceeding the service rate. The pipeline is saturating from the input side. The fix is on the demand side: reduce arrival rate (batch changes, schedule deploys) or increase service rate (parallelise, optimise).
• Queue stable, runtime growing. The service rate is declining because individual runs are taking longer. The pipeline is saturating from the capacity side. The fix is on the supply side: identify the slow step and optimise it.
• Both growing. The pipeline is in a death spiral: more arrivals are arriving into a slower service. The fix is to halt non-critical triggers until the runtime is under control.

Reading the metrics in practice

The runtime and queue data come from the same API call. A useful operational query extracts both at once:

OWNER=acme
REPO=platform
gh api repos/$OWNER/$REPO/actions/runs \
  --jq '.workflow_runs | map({
    id: .id,
    status: .status,
    runtime_seconds: (((.updated_at | fromdate) - (.run_started_at | fromdate)))
  }) | sort_by(-.runtime_seconds) | .[0:5]'
# returns: the five runs with the longest runtime, including
# their current status

The output of this query is the slowest runs in the recent history. A team that runs this query weekly and tracks the top-five-runtime is a team that catches latency creep before it becomes a timeout. A team that runs this query after the timeout has fired is a team that already knows the answer.

Production discipline

1. Alert at 80% of the timeout budget. Not at 95%, not at 100%. The 80% threshold is the headroom that lets a team optimise before the failure.
2. Track the median runtime, not the mean. A single slow run inflates the mean and hides the trend. The median is the operational signal.
3. Track the queue depth as a sustained value, not a peak. A queue depth of 1 for ten minutes is a brief backlog; a queue depth of 5 for an hour is saturation.
4. Pair the two signals. Runtime alone misses queue saturation; queue alone misses runtime growth. The pair is the picture.
5. Audit the timeout budget itself. A pipeline whose timeout is 60 minutes but whose runtime is consistently 5 minutes has a budget that is too generous; the budget could be tightened to fail fast on real outliers.

Cross-course references

• Observability course - Part II (SLOs) and Part III (BurnRate) cover the alert-threshold discipline in detail: a threshold is a probability statement, not a binary trigger.
• This course, Part XXXIX-05 (PipelineStatus) covers the per-run runtime view; this part covers the across-runs runtime view.
• This course, Part LXII-05 (Queueing) covers the queue as a serialisation mechanism; this part covers the queue as a metric.

Quiz