Job dependencies and DAG — needs, requires, dependencies, fan-in, fan-out

A pipeline is a directed acyclic graph. The nodes are jobs; the edges are dependencies. The DAG is what the CI scheduler walks when it decides which job to start next. Get the DAG right and the scheduler does the right thing in parallel. Get it wrong and the scheduler does the wrong thing in parallel - which is faster, not better.

The DAG of a pipeline

flowchart LR
    L["lint"] --> P["plan"]
    L --> S["scan"]
    P --> A["apply"]
    S --> A

Three shapes recur:

flowchart LR
    subgraph FO["Fan-out (one to many)"]
        A1["build"] --> B1["unit"]
        A1 --> B2["integration"]
        A1 --> B3["contract"]
    end
    subgraph FI["Fan-in (many to one)"]
        C1["unit"] --> D1["package"]
        C2["integration"] --> D1
        C3["contract"] --> D1
    end
    subgraph DI["Diamond (fan-out + fan-in)"]
        E1["build"] --> F1["unit"]
        E1 --> F2["integration"]
        F1 --> G1["deploy"]
        F2 --> G1
    end

• Fan-out is one job whose completion enables many. The one job’s wall-clock time is paid once; the many jobs run in parallel.
• Fan-in is many jobs whose completion enables one. The one job does not start until all the many have finished.
• Diamond is fan-out followed by fan-in. The widest point of the diamond is the parallelism; the narrow point is the synchronisation.

Expressing the DAG in GitHub Actions

The needs: clause declares an edge from each listed job to the current job. The current job waits for every listed job to complete (success or failure) before starting.

jobs:
  build:
    runs-on: ubuntu-latest
  unit:
    runs-on: ubuntu-latest
    needs: [build]
  integration:
    runs-on: ubuntu-latest
    needs: [build]
  contract:
    runs-on: ubuntu-latest
    needs: [build]
  package:
    runs-on: ubuntu-latest
    needs: [unit, integration, contract]
  deploy:
    runs-on: ubuntu-latest
    needs: [package]
    environment: production

Read the DAG from the edges. package waits for unit, integration, and contract; unit, integration, and contract wait for build. build has no needs:, so it starts immediately.

# Inspect a single job's dependencies from the workflow file
grep -A 2 '^  [a-z-]*:$' .github/workflows/ci.yml | grep -B 1 needs
# shows: jobs and their needs: clauses

Expressing the DAG in GitLab CI

GitLab CI has two ways to declare the DAG:

# Stages model: implicit DAG via stage order
stages:
  - build
  - test
  - package
  - deploy

build:
  stage: build
  script: make build
unit:
  stage: test
  script: pytest
integration:
  stage: test
  script: pytest --integration
package:
  stage: package
  script: make package
deploy:
  stage: deploy
  script: deploy.sh

The DAG is: build → (unit, integration) → package → deploy. All jobs in test run in parallel; they all wait for build. All jobs in package run after every job in test finishes.

# needs: model: explicit DAG, escaping stage ordering
package:
  stage: package
  needs:
    - job: unit
      artifacts: true
    - job: integration
      artifacts: true
  script: make package

The needs: model lets package start as soon as unit and integration succeed, without waiting for any other job in the test stage. The DAG edge is explicit; the stage ordering is overridden.

Cycles

A cycle in the DAG is a dependency that loops back: A needs B, B needs A. The CI system rejects this at lint time because a cycle makes the pipeline unschedulable - there is no job whose dependencies are all satisfied.

flowchart LR
    A["A"] --> B["B"]
    B --> A

Both CI vendors detect cycles before scheduling. GitHub Actions reports a workflow syntax error; GitLab CI reports a pipeline configuration error. Neither schedules a run.

The error message usually points at the offending edge. The fix is to break the cycle by introducing an intermediate job that does not depend on the others, or by re-thinking the dependency.

Implicit versus explicit dependencies

There are two ways a job can end up depending on another:

flowchart TB
    subgraph EX["Explicit (needs:)"]
        A1["unit"] -->|needs: package| B1["deploy"]
    end
    subgraph IM["Implicit (artifact download)"]
        A2["package"] -->|uploads artifact| X["storage"]
        X -->|downloads artifact| B2["deploy"]
    end

• Explicit is the needs: clause. The job that lists another job in its needs: waits for that job.
• Implicit is the artifact download. A step that uses actions/download-artifact or artifacts: true requires the uploading job to have completed. If the downloading job does not also declare the uploading job in needs:, the download may race against the upload.

The implicit path is the most common source of “why is my deploy running without the package?” bugs. The fix is to declare the dependency explicitly and rely on the artifact for data transfer, not for ordering.

Designing the DAG

The design rules:

flowchart LR
    subgraph R1["1. Maximise parallelism"]
        A1["build"] --> B1["unit"]
        A1 --> C1["integration"]
        A1 --> D1["scan"]
    end
    subgraph R2["2. Synchronise at boundaries"]
        B1 --> E1["package"]
        C1 --> E1
        D1 --> E1
    end
    subgraph R3["3. Serialise mutations"]
        E1 --> F1["deploy-staging"]
        F1 --> G1["deploy-production"]
    end

1. Maximise parallelism early. Every job that does not depend on another should not declare one. The default for a new job in GitHub Actions is “no needs”, which means it runs immediately.
2. Synchronise at logical boundaries. A job that needs the output of multiple jobs (a deploy that needs all tests to pass) is a fan-in; declare every predecessor in needs:.
3. Serialise mutations. A job that mutates shared state (Terraform apply, kubectl rollout) should depend on its predecessor in the same logical chain. Concurrency groups handle cross-run serialisation; needs: handles in-run serialisation.

Production discipline

1. The DAG is a code review artefact. A change to needs: is a change to ordering; require a reviewer who can read the graph.
2. Do not use needs: to express “this job is slow, run it later”. Use a separate workflow, a scheduled run, or a manual trigger. needs: is ordering, not scheduling.
3. Avoid implicit dependencies. If job B downloads an artifact from job A, declare B.needs: [A] explicitly. The artifact is data; the dependency is ordering.
4. Visualise the DAG. Use mermaid, graphviz, or the vendor’s pipeline visualisation to draw the graph and review it before merging.
5. Reject cycles at review time. A pipeline with a cycle is a pipeline that cannot run; do not let it merge.

Cross-course references

• Ansible for Production Sysadmins - Part XXXVII (RepoArch) applies the same DAG concept to AWX job templates: a workflow is a DAG of job templates with explicit edges.
• Terraform for Production Sysadmins - Part XIII (Modules) treats terraform plan and terraform apply as a serial DAG edge even when other jobs run in parallel around them.
• Linux for Production Sysadmins - Part XXX (BootProc) uses DAG concepts for systemd After= / Requires= ordering; the syntax is different but the semantics are identical.

Quiz