Jobs and steps — the units of work and how they relate

A CI pipeline is a graph, not a list. The list runs inside one job; the graph runs across many. A step is a command - run: terraform validate, run: ansible-lint playbooks/. A job is a unit of work: a runner, a checkout, a set of steps, and an outcome. The relationship between jobs is the design surface of the pipeline, and almost every production mistake in CI comes from conflating “many steps in one job” with “many jobs that should be independent”.

Steps are commands; jobs are units of work

flowchart TB
    subgraph JOB["Job: terraform-plan"]
        S1["Step 1: checkout"]
        S2["Step 2: setup-terraform"]
        S3["Step 3: terraform init"]
        S4["Step 4: terraform plan"]
    end
    S1 --> S2 --> S3 --> S4

A step is a single shell command or a single action call. It runs on the runner allocated to its job, in sequence, in the working directory of the checkout. Steps share state - files, environment variables, the working directory - because they share the runner.

A job is a boundary. A job has:

• Its own runner. Each job runs on a fresh runner allocation (often a fresh machine). The runner is not shared with another job.
• Its own secrets scope. The secrets available to a job are the secrets the runner was configured to expose. A different job can have a different secret scope.
• Its own outcome. A job ends in success, failure, or cancellation. The outcome is reported against the commit.
• Its own rules for when it runs. A job can declare if: conditions, needs: dependencies, and runner labels.

The mistake is to put everything in one job. A single job that runs terraform plan, ansible-lint, trivy scan, and terraform apply is a single job that:

• Runs sequentially even when the steps could be parallel.
• Holds every secret for every step (cloud credentials present for the lint step that does not need them).
• Reports one outcome for four independent concerns.
• Cannot be retried in isolation when one step fails.

# Wrong shape: one job, four concerns
jobs:
  everything:
    runs-on: ubuntu-latest
    steps:
      - run: terraform validate
      - run: ansible-lint
      - run: trivy scan
      - run: terraform apply    # uses cloud creds the lint step did not need

The three job relationships

Jobs relate to each other in exactly three ways:

flowchart TB
    subgraph PAR["Parallel (no needs:)"]
        A1["lint"] --> R1["report"]
        A2["plan"] --> R1
        A3["scan"] --> R1
    end
    subgraph SEQ["Sequential (needs:)"]
        B1["build"] --> B2["test"]
        B2 --> B3["publish"]
    end
    subgraph FAN["Fan-out / fan-in"]
        C1["build"] --> C2["integration-test"]
        C1 --> C3["security-scan"]
        C2 --> C4["deploy"]
        C3 --> C4
    end

1. Parallel. Two jobs that do not declare needs: run concurrently on separate runners. They share no state and must produce artefacts that other jobs consume via the artefact store, not via the filesystem.
2. Sequential. A job that declares needs: [build] waits for build to complete. If build fails, the dependent job is skipped (unless if: always() is set).
3. Fan-out / fan-in. A build job feeds both integration-test and security-scan, both of which feed deploy. This is the graph shape for a pipeline that needs independent test and scan results before deploy.

The relationships are declared in the pipeline file. There is no implicit ordering beyond needs:. If two jobs do not declare needs:, they run in parallel.

Designing the job graph for an infrastructure pipeline

flowchart TB
    P["pull_request"] --> L["lint\n(ansible-lint, tflint)"]
    P --> V["validate\n(terraform validate,\nplaybook --syntax-check)"]
    P --> S["scan\n(checkov, trivy)"]
    L --> D["plan\n(terraform plan)"]
    V --> D
    S --> D
    D --> R["report status to PR"]

For an infrastructure repository, the canonical job graph is:

• Pull request triggers a parallel fan-out. Lint, validate, and scan run concurrently. They do not need each other’s output. They produce a status that the PR displays.
• Plan runs after all three. It needs the artefacts (lint output, scan output) to be present. needs: [lint, validate, scan] makes the dependency explicit.
• Apply is a separate job, on a different trigger. It fires on push to main (or manual), reads production secrets, and runs on an isolated runner pool.

# Correct shape: jobs as units, secrets scoped
jobs:
  lint:
    runs-on: ubuntu-latest
    steps: [checkout, run: ansible-lint]
  validate:
    runs-on: ubuntu-latest
    steps: [checkout, run: terraform validate]
  scan:
    runs-on: ubuntu-latest
    steps: [checkout, run: checkov -d .]
  plan:
    needs: [lint, validate, scan]
    runs-on: ubuntu-latest
    steps: [checkout, run: terraform plan -out=tfplan]
  apply:
    needs: [plan]
    if: github.event_name == 'push'
    runs-on: deploy-runner
    environment: production
    steps: [checkout, run: terraform apply -input=false tfplan]

The apply job holds the cloud credentials; lint, validate, and scan do not. The apply job runs only on push to main, not on pull_request. The runner label deploy-runner selects the isolated pool.

Production discipline

1. One concern per job. Lint, validate, scan, plan, and apply are separate jobs. Each has a runner, a secret scope, and an outcome.
2. Secrets scoped to the job that needs them. The job that holds AWS_* is the apply job, not the lint job.
3. Apply is a different trigger than plan. Plan runs on pull_request; apply runs on push to main or on manual. The if: guard makes this explicit.
4. Runner labels select the pool. The job that holds production secrets runs on deploy-runner; the job that processes untrusted PRs runs on build-runner. They are different pools.

Cross-course references

• Linux for Production Sysadmins - Part XXXIV (ConfigMgmt) applies the same job-graph principle to package builds: source-build, test, sign, and publish are separate jobs on separate runners.
• Ansible for Production Sysadmins - Part XXXVII (RepoArch) treats molecule matrix jobs as parallel jobs under one plan job.
• Terraform for Production Sysadmins - Parts IX-XII (State) cover the plan-on-PR, apply-on-merge split; the job-per-concern model is the explicit shape.

Quiz