Parallel execution and fan-out — matrix builds, the use case, and the cost

A matrix build is one job definition expanded into N jobs at schedule time. The N jobs differ only in the values of the matrix variables; the steps, the runner, the secrets are identical. The CI scheduler walks the DAG, sees N ready jobs where there used to be one, and starts them in parallel against the runner pool. The wall-clock time of the matrix is the slowest cell, not the sum of the cells.

Declaring a matrix

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        os: [ubuntu-latest, macos-latest, windows-latest]
        python-version: ["3.11", "3.12", "3.13"]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: ${ matrix.python-version }
      - run: pytest

The matrix has two axes: os (3 values) and python-version (3 values). The product is 9 cells; the scheduler creates 9 jobs and runs them in parallel:

flowchart LR
    subgraph M["test (matrix 3x3 = 9 jobs)"]
        C1["ubuntu + py3.11"]
        C2["ubuntu + py3.12"]
        C3["ubuntu + py3.13"]
        C4["macos + py3.11"]
        C5["macos + py3.12"]
        C6["macos + py3.13"]
        C7["windows + py3.11"]
        C8["windows + py3.12"]
        C9["windows + py3.13"]
    end

The wall-clock time is the slowest cell. If windows + py3.13 takes 8 minutes and the rest take 2 minutes, the matrix finishes in 8 minutes.

One, two, and three axes

flowchart LR
    subgraph A1["One axis: 3 cells"]
        X1["os: ubuntu"]
        X2["os: macos"]
        X3["os: windows"]
    end
    subgraph A2["Two axes: 9 cells"]
        Y1["os x python"]
        Y2["os x python"]
        Y3["os x python"]
    end
    subgraph A3["Three axes: 18 cells"]
        Z1["os x python x deps"]
        Z2["os x python x deps"]
        Z3["os x python x deps"]
    end

The cell count is the product of the axis lengths. A matrix with axes of lengths 3, 3, and 2 has 18 cells. Each cell is a separate runner invocation; the runner-minute cost is 18 times the per-cell duration.

# Inspect the matrix declaration in a workflow file
grep -A 10 'matrix:' .github/workflows/test.yml
# shows the axes and their values

Computing the cost

The matrix cost is not subtle. If each cell takes 5 minutes on a hosted runner billed at $0.008/minute:

The wall-clock is constant (the slowest cell); the runner-minute cost grows with the cell count. A matrix that the team runs on every PR, every push to main, and every nightly build is a matrix whose cost compounds.

Fail-fast and cancellation

By default, a matrix keeps running all cells even after one fails. This is correct when a cell failure is independent (you want to know which cells fail); it is wasteful when a cell failure implies a common-cause failure (a syntax error that breaks every cell).

strategy:
  fail-fast: true
  matrix:
    os: [ubuntu-latest, macos-latest]
    python-version: ["3.11", "3.12"]

fail-fast: true cancels the remaining cells as soon as one fails. The cancellation is in-band: the surviving cells are marked as cancelled, not failed. The pipeline status reflects the first failure; the wall-clock time drops to the first failure rather than the slowest cell.

fail-fast: false (the default for fail-fast is false) lets every cell finish. The pipeline status reflects the slowest cell. Use it when every cell is independent.

When the matrix is the right tool

The matrix is right when the failure mode is runtime-dependent:

flowchart TB
    Q{"Does the failure mode\ndepend on a variable?"}
    Q -->|"OS, version, runtime, architecture"| YES["Use a matrix"]
    Q -->|"Configuration, region, deploy target"| NO["Use a loop or multiple jobs"]

• Cross-OS testing. A library that targets Linux, macOS, and Windows must run on all three. One matrix, three cells.
• Cross-version testing. A library that supports multiple language versions must run on each. One matrix, N cells.
• Cross-architecture testing. ARM and x86 builds of a Go binary. One matrix, two cells.

The matrix is wrong when the failure mode is configuration-dependent:

• Cross-region deployment. A deploy job that targets us-east-1, eu-west-1, and ap-south-1 is not a matrix; it is a list of jobs with different inputs. The steps are the same; the inputs differ.
• Cross-environment promotion. A deploy job that promotes from staging to canary to production is a linear chain, not a matrix.

Matrix versus loop versus multiple jobs

Three ways to do parallel work:

flowchart TB
    Q{"Are the steps\nidentical?"}
    Q -->|"yes, only inputs differ"| A["Matrix"]
    Q -->|"yes, but steps vary"| B["Multiple jobs"]
    Q -->|"no, runtime varies"| C["Loop in a single job"]

• Matrix. Same steps, same runner label, different inputs declared in the matrix axes. The scheduler handles parallelism; the user declares the axes.
• Multiple jobs. Different steps or different runners per job. Declared as separate jobs in the workflow file; parallelised by the absence of needs:.
• Loop in a single job. A single job that iterates over a list of inputs sequentially or in background. No matrix; parallelism is intra-job (rare and complex).

Production discipline

1. Compute the runner-minute cost of every matrix before adding an axis. If the cost is more than the team would willingly spend on a PR feedback run, the matrix is wrong.
2. Set fail-fast deliberately. true for common-cause matrices; false for independent-cell matrices.
3. Use include: and exclude: for asymmetric matrices. A matrix where windows skips python-version: 3.13 (because the runtime does not support it) uses exclude: to drop the unsupported cells.
4. Cap the matrix size. A matrix with 50+ cells is a matrix that overwhelms the runner pool; consider a smaller matrix or a separate scheduled workflow for the long-tail cells.
5. Audit the cells. A cell that has not failed in six months is a candidate for removal. The matrix exists to catch a bug; if the bug class has not appeared, the cell may be over-coverage.

Cross-course references

• Ansible for Production Sysadmins - Part XXXVII (RepoArch) applies the matrix pattern to AWX job templates: one template, N inventories.
• Terraform for Production Sysadmins - Parts IX-XII (State) apply the matrix pattern to multi-region Terraform plans: one plan job, N regions.
• Linux for Production Sysadmins - Part XXXII (KernelBuild) uses the matrix pattern to build kernel modules against multiple kernel headers.

Quiz