Skip to main content
RunBook Academy

← All labs in Git, CI/CD & GitOps

Lab · intermediate · ~60 min

Lab 9: Build a multi-job CI pipeline with `needs`, matrix, and conditional execution

C · SimulationB · Nested virtualisation

Objectives

  • Author a GitHub Actions workflow with a build → test → publish DAG using the `needs:` keyword
  • Add a matrix strategy that fans out across OS and Terraform versions, with `fail-fast: false` so one failure does not abort the others
  • Use `if:` conditions to skip expensive jobs on documentation-only changes (paths filter)
  • Capture and upload artefacts from a fan-in job so downstream jobs can read the build output
  • Measure the cost savings of the conditional skip against the unconditional run
  • Document the failure-propagation rules: a failed `needs:` parent fails all children

Prerequisites

Objective

By the end of this lab you will have authored a GitHub Actions workflow that demonstrates the three primitives of multi-job orchestration: needs: for ordering, matrix for fan-out, and if: for conditional execution. The workflow builds a Terraform binary on three operating systems and three Terraform versions (a 3×3 = 9-way fan-out), runs a single integration test that joins on all nine builds, and skips the entire pipeline on documentation-only changes.

The point is not the YAML — it is the DAG. The pipeline as a graph is the mental model that lets you reason about cost, failure propagation, and the trade-offs between fan-out (more compute, faster wall time) and fan-in (more coordination, more brittle joins).

Architecture

A pipeline DAG with three layers: a build matrix (9 jobs), a fan-in test job (1 job, depends on all 9 builds), and a conditional publish job (1 job, depends on the test job, runs only on tagged releases).

flowchart TB
    L[lint]
    B1["build\nubuntu · 1.9.x"]
    B2["build\nubuntu · 1.10.x"]
    B3["build\nubuntu · 1.11.x"]
    B4["build\nmacos · 1.9.x"]
    B5["build\nmacos · 1.10.x"]
    B6["build\nmacos · 1.11.x"]
    B7["build\nwindows · 1.9.x"]
    B8["build\nwindows · 1.10.x"]
    B9["build\nwindows · 1.11.x"]
    T["test\nneeds: build.*"]
    P["publish\nneeds: test\nif: startsWith(github.ref, 'refs/tags/')"]

    L --> T
    B1 --> T
    B2 --> T
    B3 --> T
    B4 --> T
    B5 --> T
    B6 --> T
    B7 --> T
    B8 --> T
    B9 --> T
    T --> P

The 9 build jobs run in parallel, the test job joins on all of them, and the publish job is conditional on the test passing and the run being a tag push.

Requirements

  • Git 2.55.x on Linux or macOS.
  • A GitHub repository. The workflow file is the deliverable; the run is not exercised in the lab.
  • No network access required to author the workflow. The runner fleet is GitHub-hosted; the matrix dimensions are limited by what GitHub Actions offers on the team’s tier.

Scenario

A platform team supports Terraform across Linux, macOS, and Windows, and across the current and two previous minor versions. They want CI to catch regressions in any of those combinations before they ship a release. The mitigation is a matrix build that produces a binary for each combination, a fan-in test that exercises every binary, and a publish step that only runs when the test passes and the run is a tagged release.

The lab builds the workflow and the documentation that explains why each design choice was made.

Tasks

Task 1 — Build the empty repository

LAB="$HOME/multi-job-lab"
rm -rf "$LAB"
mkdir -p "$LAB"
cd "$LAB"

git init -b main
git config user.email 'ops@example.com'
git config user.name  'Ops'

mkdir -p src

# A trivial "build target" — the workflow's build job compiles a
# single Go file into a binary and uploads it as an artefact. The
# point is the workflow shape, not the build target.
cat > src/main.go <<'EOF'
package main

import "fmt"

func main() {
    fmt.Println("runbook build")
}
EOF

cat > go.mod <<'EOF'
module runbook/build

go 1.22
EOF

git add src/ go.mod
git commit -m 'initial: trivial go module as the build target'

The repository has a single Go file that the build job compiles. The “Terraform version” dimension is simulated by passing the version string into the binary as a build flag, which the workflow demonstrates.

Task 2 — Author the multi-job workflow

# check-shell-blocks: allow-invalid
cd "$HOME/multi-job-lab"

mkdir -p .github/workflows

cat > .github/workflows/multi-job.yml <<'EOF'
name: multi-job

on:
  push:
    branches: [main]
    tags: ['v*.*.*']
  pull_request:
    branches: [main]

# Concurrency: cancel any in-flight run for the same ref when a new
# push arrives. Tag pushes get a separate concurrency group so a
# release is not cancelled by an unrelated push.
concurrency:
  group: multi-job-${ github.workflow }-${ github.ref }
  cancel-in-progress: ${ github.ref != 'refs/heads/main' && !startsWith(github.ref, 'refs/tags/') }

permissions:
  contents: read

jobs:
  # ─────────────────────────────────────────────────────────────────
  # Layer 1: lint (no dependencies)
  # ─────────────────────────────────────────────────────────────────
  lint:
    name: lint
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11  # v4.1.1
      - uses: actions/setup-go@0a5cd549ddc19ed5d6e3a91e9e4380152814fbb5  # v5.0.0
        with:
          go-version: '1.22'
      - run: go vet ./...

  # ─────────────────────────────────────────────────────────────────
  # Layer 2: build matrix (3 OS × 3 terraform versions = 9 jobs)
  # ─────────────────────────────────────────────────────────────────
  build:
    name: build (${ matrix.os } · terraform ${ matrix.terraform })
    runs-on: ${ matrix.os }
    strategy:
      fail-fast: false
      matrix:
        os: [ubuntu-24.04, macos-14, windows-2022]
        terraform: ['1.9.x', '1.10.x', '1.11.x']
    steps:
      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11  # v4.1.1
      - uses: actions/setup-go@0a5cd549ddc19ed5d6e3a91e9e4380152814fbb5  # v5.0.0
        with:
          go-version: '1.22'
      - name: build
        env:
          TF_VERSION: ${ matrix.terraform }
          GOOS: ${ matrix.os == 'windows-2022' && 'windows' || matrix.os == 'macos-14' && 'darwin' || 'linux' }
        run: |
          mkdir -p out
          go build -ldflags "-X main.tfVersion=$TF_VERSION" -o out/runbook-$GOOS-$TF_VERSION ./src
      - name: upload artefact
        uses: actions/upload-artifact@5d5d22a31266ced268874388b861e4b58bb5c2f3  # v4.0.0
        with:
          name: build-${ matrix.os }-${ matrix.terraform }
          path: out/

  # ─────────────────────────────────────────────────────────────────
  # Layer 3: integration test (depends on the lint job and ALL 9
  # build jobs; runs even on docs-only changes because docs do not
  # affect the test — but skips when the change is entirely outside
  # the build/test surface)
  # ─────────────────────────────────────────────────────────────────
  test:
    name: integration test
    runs-on: ubuntu-24.04
    needs: [lint, build]
    if: |
      always() &&
      needs.lint.result == 'success' &&
      !contains(needs.*.result, 'failure') &&
      !contains(needs.*.result, 'cancelled')
    steps:
      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11  # v4.1.1
      - uses: actions/download-artifact@fa0a91b85d4f404e444e00e005971372dc801d16  # v4.0.0
        with:
          path: artifacts/
          merge-multiple: true
      - name: list artefacts
        run: ls -la artifacts/
      - name: smoke test every artefact
        run: |
          set -eu
          for f in artifacts/*; do
            echo "--- $f"
            "$f" | head -3
          done

  # ─────────────────────────────────────────────────────────────────
  # Layer 4: publish (depends on the test job; runs only on tags)
  # ─────────────────────────────────────────────────────────────────
  publish:
    name: publish
    runs-on: ubuntu-24.04
    needs: [test]
    if: startsWith(github.ref, 'refs/tags/v')
    environment:
      name: release
      url: https://github.com/${ github.repository }/releases/tag/${ github.ref_name }
    steps:
      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11  # v4.1.1
      - uses: actions/download-artifact@fa0a91b85d4f404e444e00e005971372dc801d16  # v4.0.0
        with:
          path: artifacts/
          merge-multiple: true
      - name: publish artefacts
        run: |
          echo "would publish:"
          ls -la artifacts/
EOF

git add .github/workflows/multi-job.yml
git commit -m 'ci: multi-job workflow with matrix, needs, and conditional execution'

The workflow has four jobs in three layers. The matrix produces nine build jobs, the test job joins on all of them, and the publish job is gated on startsWith(github.ref, 'refs/tags/v').

Task 3 — Document the fan-out and fan-in

# check-shell-blocks: allow-invalid
cd "$HOME/multi-job-lab"

cat > fan-out-explanation.md <<'EOF'
# Pipeline fan-out and fan-in

The `multi-job` workflow has four jobs arranged in three layers:

## Layer 1 — `lint`

A single job that runs `go vet`. No dependencies. Runs in parallel
with the build matrix.

## Layer 2 — `build` matrix

A 3×3 matrix that produces nine jobs:

| Operating system | Terraform version |
|------------------|-------------------|
| `ubuntu-24.04`   | `1.9.x`           |
| `ubuntu-24.04`   | `1.10.x`          |
| `ubuntu-24.04`   | `1.11.x`          |
| `macos-14`       | `1.9.x`           |
| `macos-14`       | `1.10.x`          |
| `macos-14`       | `1.11.x`          |
| `windows-2022`   | `1.9.x`           |
| `windows-2022`   | `1.10.x`          |
| `windows-2022`   | `1.11.x`          |

The matrix is declared with `strategy.matrix` and `fail-fast:
false`. Each job uploads its binary as a uniquely-named artefact.

## Layer 3 — `test`

A single fan-in job that depends on `lint` and all nine `build.*`
jobs:

```yaml
test:
  needs: [lint, build]

The needs: [lint, build] reference uses the build shorthand: when a matrix job has the same id as a non-matrix job, needs: build means “all jobs in the matrix”. To depend on a single matrix value, the syntax is needs: build.<matrix-value> (the “matrix object” syntax).

The if: block guards the test against running on failed matrix parents:

if: |
  always() &&
  needs.lint.result == 'success' &&
  !contains(needs.*.result, 'failure') &&
  !contains(needs.*.result, 'cancelled')

always() runs the step even when an upstream job failed, but the !contains(...) filters out the failure cases. Without this guard, a matrix failure would either skip the test (because needs.test requires success) or report the test as cancelled (which is what if: always() does, with no further filtering).

Layer 4 — publish

A single fan-out job that depends on the test job and runs only on tag pushes:

publish:
  needs: [test]
  if: startsWith(github.ref, 'refs/tags/v')

The environment: release block attaches the job to the release GitHub Environment, which can have its own protection rules (required reviewers, wait timer, branch restriction). The job is therefore doubly gated: by the if: expression and by the environment’s protection rules.

Failure propagation

The rule is: a job that depends on a failed upstream job is skipped unless its if: block overrides that behaviour. The override patterns are:

  • if: always() — run regardless of upstream state.
  • if: failure() — run only when an upstream failed.
  • if: success() — the default; run only when all upstreams succeeded.
  • if: cancelled() — run only when the run was cancelled.

The test job’s if: block combines always() with explicit checks against the needs.*.result array. This is the pattern to follow when you want a job to run only on a specific subset of upstream outcomes, not on the default success-only behaviour. EOF

git add fan-out-explanation.md git commit -m ‘docs: fan-out and fan-in explanation’


The fan-out document is what a new contributor reads to
understand why the matrix has nine jobs and not three or twenty.
It is also the place the team updates when the matrix dimensions
change.

### Task 4 — Document the paths-filter behaviour

```bash
# check-shell-blocks: allow-invalid
cd "$HOME/multi-job-lab"

cat > paths-filter-table.md <<'EOF'
# Paths-filter behaviour

The `multi-job` workflow does not have an explicit `paths:` filter
on its `on:` trigger, so it runs on every push and pull request.
This is the conservative default for a small repository. The
table below describes what each job would do under each of three
candidate paths-filter strategies; the team should choose one
based on the cost estimate in `cost-estimate.md`.

## Strategy A — no filter (current)

The workflow runs on every push and PR, regardless of which paths
changed.

| Change                                       | Lint | Build | Test | Publish |
|----------------------------------------------|:----:|:-----:|:----:|:-------:|
| `src/main.go`                                |  ✓   |   ✓   |  ✓   | tag only |
| `docs/policy-note.md`                        |  ✓   |   ✓   |  ✓   | tag only |
| `README.md`                                  |  ✓   |   ✓   |  ✓   | tag only |
| `.github/workflows/multi-job.yml`            |  ✓   |   ✓   |  ✓   | tag only |

## Strategy B — paths filter on the build/test jobs

The build matrix and test job get an explicit `if:` filter that
skips them when the change is documentation-only.

```yaml
build:
  if: |
    !contains(github.event.pull_request.changes.paths_from_origin, 'docs/**') ||
    contains(github.event.pull_request.changes.paths_from_origin, 'src/**') ||
    contains(github.event.pull_request.changes.paths_from_origin, '.github/**')
ChangeLintBuildTestPublish
src/main.go✓✓✓tag only
docs/policy-note.md✓⊘⊘tag only
README.md✓⊘⊘tag only
.github/workflows/multi-job.yml✓✓✓tag only

The third row (README-only) skips the build and test jobs because neither depends on the README. The fourth row (workflow change) still runs the build and test jobs because a workflow change can introduce regressions that the build matrix would catch.

Strategy C — dorny/paths-filter action

The dorny/paths-filter action provides a richer filter that returns named outputs based on which paths changed. The lab does not include this strategy in the workflow but documents it as the recommended evolution when the team has more than a handful of documentation paths.

- uses: dorny/paths-filter@de90cc6fb38fc0963ad72b210f1b284cd68cea36  # v3.0.1
  id: filter
  with:
    filters: |
      src:
        - 'src/**'
      docs:
        - 'docs/**'
      workflow:
        - '.github/**'

The filter step’s outputs (steps.filter.outputs.src, steps.filter.outputs.docs, steps.filter.outputs.workflow) are then used in downstream if: expressions.

Recommendation

Start with Strategy A (no filter). Move to Strategy B when the team has more than five documentation-only PRs per week. Move to Strategy C when the documentation surface grows past what an explicit if: block can comfortably enumerate. EOF

git add paths-filter-table.md git commit -m ‘docs: paths-filter strategies for the multi-job workflow’


The paths-filter table is the team's reference for "do I need to
run CI on this PR?". The table is updated when the workflow is
restructured; the cost estimate is updated when the matrix
dimensions change.

### Task 5 — Estimate the cost savings

```bash
# check-shell-blocks: allow-invalid
cd "$HOME/multi-job-lab"

cat > cost-estimate.md <<'EOF'
# Cost estimate for the multi-job workflow

The matrix dimension is 3 OS × 3 Terraform versions = 9 build jobs.
With `fail-fast: false`, every matrix job runs to completion even
on failure. The estimate below uses GitHub-hosted runner rates
(August 2026) for the standard tiers.

## Per-job cost

| Job          | Runner                | Avg duration | Rate (per minute) |
|--------------|-----------------------|--------------|-------------------|
| `lint`       | `ubuntu-24.04`        | 30 s         | $0.008            |
| `build.*`    | `ubuntu-24.04`        | 4 min        | $0.008            |
| `build.*`    | `macos-14`            | 6 min        | $0.080            |
| `build.*`    | `windows-2022`        | 6 min        | $0.016            |
| `test`       | `ubuntu-24.04`        | 2 min        | $0.008            |
| `publish`    | `ubuntu-24.04`        | 1 min        | $0.008            |

## Per-run cost (no paths filter)

Total = 1 lint + 3 ubuntu + 3 macos + 3 windows + 1 test + 0 publish
       = $0.004 + $0.096 + $1.440 + $0.288 + $0.016
       = $1.844 per push / pull-request run

For 100 PRs per week, the weekly cost is approximately **$184.40**.

## Per-run cost (with Strategy B paths filter, 50% of PRs docs-only)

For the 50 docs-only PRs:

Total = 1 lint + 0 build + 0 test
       = $0.004

For the 50 code-touching PRs:

Total = $1.844 (as above)

Weekly cost = 50 × $0.004 + 50 × $1.844 = **$92.20**

Savings: $184.40 − $92.20 = **$92.20 per week**, or **50%**.

## Caveats

1. The estimate uses runner-minute costs for the public linux
   runner tier. Self-hosted runners are billed differently, and
   the cost saving calculation changes.
2. The 4-minute and 6-minute build durations are placeholders.
   Real builds of a Terraform binary take longer (15-30 minutes);
   the proportion saved is the same, but the absolute number is
   higher.
3. The `fail-fast: false` setting is the cost-vs-speed trade-off;
   switching to `fail-fast: true` would save compute at the cost
   of not knowing the full state of the matrix on a failure.
4. The `concurrency.cancel-in-progress` setting cancels
   in-progress runs on the same ref when a new push arrives,
   which can save additional compute on PRs that see rapid
   pushes. The estimate does not include this saving.

## Recommendation

If the team has more than 50 documentation-only PRs per week,
adopt Strategy B (the explicit `if:` filter). Below that
threshold, the engineering cost of maintaining the filter is
likely higher than the compute saved.
EOF

git add cost-estimate.md
git commit -m 'docs: cost estimate for the multi-job workflow'

The cost estimate is what an engineering manager reads to decide whether the multi-job workflow is worth the spend. The numbers are illustrative — replace them with the team’s actual runner rates and average build durations before applying the workflow to production.

Task 6 — Capture the deliverables

cd "$HOME/multi-job-lab"

# The workflow file is already in the repository.
cp .github/workflows/multi-job.yml "$HOME/multi-job.yml"

ls -l .github/workflows/multi-job.yml \
       fan-out-explanation.md \
       paths-filter-table.md \
       cost-estimate.md

The deliverable workflow file is in $HOME/multi-job.yml, and the three documentation files are in the repository for team review.

Task 7 — Validate the YAML structure

cd "$HOME/multi-job-lab"

# YAML parse check.
python3 -c "
import yaml
with open('.github/workflows/multi-job.yml') as f:
    doc = yaml.safe_load(f)
jobs = doc['jobs']
print('jobs:', list(jobs.keys()))
print('matrix:', jobs['build']['strategy']['matrix'])
print('needs of test:', jobs['test']['needs'])
print('if of publish:', jobs['publish']['if'])
"

The output must list four jobs (lint, build, test, publish), with build having a matrix of three OS values and three Terraform versions, test having needs: [lint, build], and publish having the if: startsWith(...) expression.

Validation

  • .github/workflows/multi-job.yml parses as valid YAML and has exactly four jobs: lint, build, test, publish.
  • jobs.build.strategy.matrix has 3 OS values and 3 Terraform version values, producing 9 matrix combinations.
  • jobs.build.strategy.fail-fast is false.
  • jobs.test.needs is [lint, build] and the if: block uses always() plus the !contains(needs.*.result, ...) filter.
  • jobs.publish.if is startsWith(github.ref, 'refs/tags/v').
  • Every uses: reference is a pinned commit SHA.
  • fan-out-explanation.md, paths-filter-table.md, and cost-estimate.md exist and are non-empty.
  • The deliverables in $HOME and the repository match the expected file set.

Expected Outcome

A workflow file that produces a 9-way matrix, a fan-in test, and a conditional publish, with three documents that explain the design choices.

$HOME/multi-job-lab/
├── .github/workflows/multi-job.yml   # the workflow file, pinned to SHAs
├── fan-out-explanation.md            # DAG explanation
├── paths-filter-table.md             # paths-filter strategies
├── cost-estimate.md                  # weekly cost estimate
├── go.mod
├── src/main.go
└── README.md

The workflow is the configuration; the three documents are the durable explanation of why the configuration exists. A new contributor reads the documents on day one; the workflow file is exercised by the platform on every push.

Troubleshooting

A matrix job fails and the test job is skipped. This is the default behaviour when needs: build is specified and a matrix job fails. The if: block on the test job is what makes it run despite the failure; verify the if: expression matches the example. The expression !contains(needs.*.result, 'failure') is the key guard.

A matrix job is cancelled because the run was cancelled. The !contains(needs.*.result, 'cancelled') guard in the test job’s if: is what prevents a cancelled matrix from cancelling the test. Without it, a cancelled run produces no test output, which is unhelpful for debugging.

needs: build does not depend on all matrix jobs. That is correct: needs: build (without an index) does depend on all matrix jobs. If you want to depend on a single matrix value, the syntax is needs: build.&lt;matrix-key&gt; (the “needs object” syntax). The lab uses the shorthand because the test depends on all matrix jobs.

The concurrency group cancels a tagged release. The cancel-in-progress expression excludes tag pushes: ${ github.ref != 'refs/heads/main' && !startsWith(github.ref, 'refs/tags/') }. If a tag push is being cancelled, the expression has been edited; restore it to the original form.

The matrix dimension exceeds the plan limit. GitHub Actions has a per-matrix limit on the number of jobs. The default is 256 jobs; the team can request an increase. The lab’s 3×3 matrix is well within the default and is not affected.

Cleanup

LAB="$HOME/multi-job-lab"

# Keep the deliverables.
mv "$LAB"/fan-out-explanation.md "$LAB"/paths-filter-table.md \
   "$LAB"/cost-estimate.md "$HOME"/ 2>/dev/null
mv "$LAB/.github/workflows/multi-job.yml" "$HOME/multi-job.yml" 2>/dev/null

rm -rf "$LAB"

find "$HOME" -maxdepth 1 -name 'multi-job-lab' -print
# expected: (no output)

If you applied the workflow to a real GitHub repository during the lab, disable or delete it through the GitHub UI or the API:

gh api --method DELETE \
  /repos/$OWNER/$REPO/actions/workflows/multi-job.yml

What You Learned

  • A pipeline is a DAG. Jobs are nodes, needs: declarations are edges, and the graph is what the runner platform executes. Mental models that treat CI as “a list of commands” lose this.
  • needs: build (without an index) depends on all matrix jobs. To depend on a single matrix value, use the “needs object” syntax: needs: build.&lt;matrix-key&gt;.
  • fail-fast: false is the cost-vs-speed trade-off. It is the right setting for compatibility testing, the wrong setting for cost. Document the choice explicitly so a future contributor knows it was deliberate.
  • if: always() plus explicit needs.*.result checks is the pattern for fan-in jobs that must run despite upstream failures. The default if: success() skips the test when an upstream fails, which is usually the wrong behaviour for an integration test.
  • concurrency.cancel-in-progress is conditional on the ref. Tag pushes and main pushes should not be cancelled by an unrelated event; PR pushes should be.
  • The cost estimate is the durable artefact. The workflow file is configuration; the cost estimate is the answer to “is this worth the spend?”. A team that does not maintain the estimate will over-spend on the matrix without realising it.

Deliverables

  • · .github/workflows/multi-job.yml — the workflow file with matrix, needs, and conditional execution
  • · fan-out-explanation.md — a description of the matrix fan-out, the fan-in, and the failure-propagation rules
  • · paths-filter-table.md — a table showing which jobs run for which paths-filter results
  • · cost-estimate.md — a worked estimate of the runner-minutes saved by the conditional skip

Verification status

Last reviewed
2026-08-25
Executed end to end
not yet run on hardware

The commands and configuration here have been reviewed against the verified software versions, but nobody has run this lab start to finish on a system meeting its prerequisites. Treat the Expected Outcome as the intended result rather than an observed one, and keep the Cleanup section to hand.