Fan-in and fan-out — parallelism at job boundaries

Three DAG shapes recur in every pipeline: fan-out, fan-in, and diamond. They are the vocabulary for reading and writing DAGs.

Fan-out

Fan-out is one job whose completion enables many successors.

flowchart LR
    B["build"] --> U["unit"]
    B --> I["integration"]
    B --> SC["scan"]

build runs first; the three start as soon as it completes and a runner is free. Cost is max(unit, integration, scan).

Fan-in

Fan-in is many predecessors whose completion enables one successor. The successor starts only when every predecessor has completed.

flowchart LR
    U["unit"] --> PKG["package"]
    I["integration"] --> PKG
    C["contract"] --> PKG

package waits for all three. Wall-clock cost is max(unit, integration, contract).

Diamond

Diamond is fan-out followed by fan-in.

flowchart LR
    B["build"] --> U["unit"]
    B --> I["integration"]
    U --> DP["deploy"]
    I --> DP

Cost is build + max(unit, integration) + deploy.

Parallelism at boundaries

Parallelism is bounded by the DAG, the runners, and concurrency:.

concurrency:
  group: ${ github.workflow }-${ github.ref }
  cancel-in-progress: true

concurrency: for cross-run; DAG for within-run.

Designing DAGs

flowchart LR
    B1["build"] --> T1["test-a"]
    B1 --> T2["test-b"]
    B1 --> T3["test-c"]
    T1 --> D1["deploy"]
    T2 --> D1
    T3 --> D1

1. Maximise parallelism early.
2. Synchronise at boundaries.
3. Serialise mutations with needs:.

Production discipline

1. Identify fan-in and slowest predecessor.
2. Optimise slowest predecessor first.
3. Avoid fan-in to jobs not needing all predecessors.
4. concurrency: for cross-run; DAG for within-run.

Cross-course references

• Ansible for Production Sysadmins - Part XXXVII applies fan-out/fan-in to AWX.
• Linux for Production Sysadmins - Part XXX uses diamond for systemd.

Quiz