CI versus the developer laptop — why "it works on my machine" is the bug

“It works on my machine” is not a defence; it is a confession. The sentence admits that the change runs in exactly one environment - the laptop - and that environment is not part of the system that ships the change. The CI runner is. The runner is reproducible: given the same commit and the same pipeline file, the runner produces the same result. The laptop is not. The runner is ephemeral: state does not carry between jobs. The laptop is not. The runner is audited: every step is logged, every artefact is bound to a SHA, every secret is scoped. The laptop is not. CI is required precisely because the laptop is not the system of record; it is one developer’s private environment, and every property the system of record needs - reproducibility, isolation, auditability - is absent.

What drifts between laptop and CI

flowchart LR
    L["Developer laptop"] -->|"commits + pushes"| R["CI runner"]
    subgraph DRIFT["Drift sources"]
        D1["OS version + patches"]
        D2["Language runtime version"]
        D3["Dependency versions\n(pip, npm, apt)"]
        D4["Local config (~/.aws, ~/.kubeconfig)"]
        D5["Tooling the developer installed\n(yesterday's fix, today's shortcut)"]
        D6["Stale caches, leftover state"]
    end
    L -.->|varies along| DRIFT
    R -.->|pinned in pipeline file| PIN["Pinned runner image\n(ubuntu-22.04)\n+ locked dependencies"]

Six categories of drift:

• OS and patches. The laptop runs an OS the developer installed a year ago; the runner runs the image the pipeline pins. A library that compiles on macOS Sonoma may not compile on the runner’s Ubuntu 22.04.
• Language runtime version. Python 3.12.3 on the laptop with PEP 668 site-packages overrides, vs Python 3.11 on the runner. A module that imports on the laptop raises ImportError on the runner.
• Dependency versions. pip freeze from last Tuesday, vs the requirements.txt the pipeline installs from. The laptop is pip install’d ahead; the runner is locked.
• Local credentials and config. ~/.aws/credentials with a personal access key, vs the OIDC-scoped deploy identity the runner uses.
• Local tooling. The developer’s terraform is 1.7.4 from the homebrew install; the runner’s is pinned in the pipeline.
• Local caches and leftover state. ~/.terraform.d, a half-applied state file, a stale Docker image.

Every one of these is a place where “works on my machine” can be true and “works on CI” can be false - or vice versa, where the laptop hides a bug the runner exposes.

The properties CI adds over the laptop

flowchart TB
    L["Laptop\n(mutable, one-off, unaudited)"]
    CI["CI runner\n(ephemeral, reproducible, audited)"]

    L -->|lacks| R1["Reproducibility\n(different result on next run)"]
    L -->|lacks| I1["Isolation\n(state carries between runs)"]
    L -->|lacks| A1["Auditability\n(no log, no SHA binding)"]
    L -->|lacks| S1["Secret scoping\n(everything sees everything)"]

    CI -->|provides| R1
    CI -->|provides| I1
    CI -->|provides| A1
    CI -->|provides| S1

• Reproducibility. The runner is allocated from a pinned image (e.g. ubuntu-22.04) with pinned tool versions (actions/setup-python@v5 with python-version: '3.11'). Given the same commit and the same pipeline, the runner produces the same result. The laptop’s result is one execution out of many possible executions.
• Isolation. The runner is ephemeral. State from the previous job does not survive. The laptop is persistent; a .terraform.d cache, a half-applied state, a stale Docker image all carry into the next run.
• Auditability. Every step is logged. Every artifact is bound to a SHA. Every secret access is recorded. The laptop’s terminal scrollback is not auditable in any sense the operations team will accept.
• Secret scoping. The runner has only the secrets the job requested. The laptop has every credential the developer has ever pasted into ~/.aws/credentials, regardless of what the current task needs.

These are not luxuries. They are the difference between a change that can be reproduced, isolated, audited, and scoped - and a change that cannot.

Why “it works on my machine” is the bug

The sentence is an admission of unreproducibility. By saying “my machine”, the speaker is naming a one-off environment as the only place the change works. The fix is not “make the laptop match CI” (which fails the moment the developer installs a new tool); the fix is “make CI the executor and catch laptop drift at PR time”.

git push origin feature/add-logs-bucket
gh run watch --exit-status

git push is the handoff. Everything before the push is the developer’s environment; everything after is the runner’s. The runner is where the change either reproduces or fails.

The laptop-to-CI handoff

flowchart LR
    L["Developer laptop"] -->|"git commit"| S["Staged change\n(index)"]
    S -->|"git push"| R["Remote ref"]
    R -->|"trigger"| C["CI runner"]
    C -->|"plan / scan / test"| V{"Verifies\nchange"}
    V -->|green| M["Merge to main"]
    V -->|red| B["Block merge\n(branch protection)"]

The handoff has four steps:

1. Commit locally. git add and git commit move the change from the working tree to the index to the local repository. The laptop is still the only place the change exists.
2. Push to the remote. git push origin <branch> moves the change to the shared repository. The forge now has the SHA.
3. CI fires on the push. The runner checks out the SHA, runs the pipeline, reports status against the SHA.
4. Branch protection gates the merge. The status is the green check (or red X). The merge is allowed or blocked.

Steps 1 and 2 are the developer’s responsibility. Steps 3 and 4 are the system’s. The handoff is clean: the developer owns the change up to the push, the system owns the change after it.

Catching laptop drift at PR time

The pipeline must be designed to catch every category of drift:

• Pinned runner image. runs-on: ubuntu-22.04 rather than runs-on: latest. The OS is reproducible.
• Pinned tooling. actions/setup-terraform@v2 with a version, not whatever the developer’s terraform happens to be.
• Locked dependencies. pip install -r requirements.lock, npm ci, go.sum. The lockfile is the contract.
• No local-state dependencies. The pipeline does not read ~/.aws/credentials. The secrets are forge-managed or OIDC-scoped.
• Explicit failure on diff. The pipeline runs terraform plan, ansible-lint, trivy scan - all of which are designed to fail loudly when the bytes the runner sees differ from the bytes the developer wrote.

Production discipline

1. The laptop is an editor, not an executor. Any change that ships goes through git push and a CI run. Direct applies from the laptop are forbidden for production.
2. Catch drift at PR time. The pipeline runs every category of check that would fail if the laptop’s environment were different from the runner’s.
3. Audit requires the runner. The deploy that reaches production is the one CI ran. The change the auditor reasons about is the one CI built.
4. “Works on my machine” is a debugging starting point, not a conclusion. The conclusion is “the pipeline is green”; the debugging starts from “the pipeline is red despite my laptop saying it’s fine”.

Cross-course references

• Linux for Production Sysadmins - Part XXXIV (ConfigMgmt) treats the build host as the executor for the package; the developer laptop is where the source is edited, but the package is built by the build host.
• Ansible for Production Sysadmins - Part XXXVII (RepoArch) applies the same model: the playbook is edited locally, the molecule test runs in CI, the production apply runs from the deploy job.
• Terraform for Production Sysadmins - Parts IX-XII (State) make the laptop-to-CI handoff explicit: terraform plan runs in CI; terraform apply runs from CI; the developer’s laptop is not in the apply path at any point.

Quiz