Cache poisoning and staleness — the failure modes of trusting a cache

A cache is best-effort by design, but “best-effort” is a performance semantic, not a trust semantic. A cache that the runner restores is bytes the runner treats as authoritative for the rest of the job. If the bytes are stale (because the key has not tracked the lockfile), the build links against outdated dependencies. If the bytes are malicious (because an attacker has written to the cache store under a key the workflow will restore), the build links against attacker-controlled dependencies. The two failure modes are correctness and security respectively, and the defences are different.

Staleness: when the lockfile-key invariant breaks

The cache key is supposed to be a hash of the inputs that produced the cache entry. The invariant is:

When the inputs change, the key changes; when the inputs are the same, the key is the same.

The invariant breaks in three ways:

• Fixed-string keys. The key is a literal string, not a hash. The lockfile changes; the key does not; the runner restores outdated bytes. This is the most common staleness failure.
• Hashed inputs that are not the lockfile. The key includes hashFiles('README.md') or hashFiles('*.py') instead of hashFiles('requirements.txt'). The lockfile changes; the source files change too (usually); the key sometimes matches by coincidence and sometimes does not. The runner restores outdated bytes when the lockfile changed but the hashed inputs did not.
• Coarse-grained hash inputs. The key includes hashFiles('**/package.json') and the lockfile is one of many package.json files in the repository. A change to an unrelated package.json invalidates the cache; a change to the lockfile does not.

flowchart LR
    A["requirements.txt"] --> B["hashFiles()"]
    B --> C["key"]
    D["README.md"] -. "wrong input" .-> E["key (same)"]
    E --> F["Restore outdated cache"]
    F --> G["Install against wrong package versions"]

The diagram shows the most common staleness failure: the lockfile changed but the key was derived from a different file (the README, a source file, an unrelated manifest). The key is unchanged; the runner restores outdated bytes; the install step runs against the wrong versions.

Poisoning: the threat model

A cache is a writeable store scoped to the repository, branch, and workflow. Anyone who can run the workflow on the same branch can write to the cache under any key. The threat model is:

• A pull request from a fork can run actions/cache@v4 with a key the upstream workflow will later restore. The fork runs with its own scope; it cannot read upstream caches, but it cannot write to them either - GitHub isolates fork PR cache scope. The threat from a fork is therefore limited to its own scope.
• A compromised workflow step can write a malicious payload to the cache under a key the next run will restore. The threat is from a step that has access to the same scope as the cache step, which is any step in the same job.
• An attacker who has compromised a maintainer account can push a workflow change that writes a malicious payload to the cache under any key. The threat is from any code path that runs in the repository’s scope.

flowchart TB
    subgraph Attack["Attack paths"]
        A1["Compromised step in same job"]
        A2["Compromised workflow file"]
        A3["Compromised third-party action"]
    end
    subgraph Cache["Cache store"]
        C1["legitimate pip wheel"]
        C2["malicious pip wheel"]
    end
    Attack --> C2
    Cache --> D["Restore in next run"]
    D --> E["Install malicious wheel"]
    E --> F["Production binary links against attacker code"]

The diagram shows the three attack paths and the consequence: a malicious wheel restored in the next run and linked into the production binary. The attack is silent: the cache step reports success, the install step reports success, the build links against the attacker’s code.

Defences

Four defences reduce the risk of cache poisoning and staleness. None of them is sufficient on its own; the four together are the production-grade posture.

• Hash-derived keys. The key is derived from hashFiles() of the lockfile. A key that does not change when the lockfile changes cannot be the right key.
• Branch scoping. The cache is scoped to the branch by default. A pull request from a fork cannot write to the upstream branch’s cache. A feature branch cannot poison the main branch’s cache. The scope is the platform’s primary defence against cross-branch poisoning.
• Post-restore verification. After the cache restore, the workflow verifies the cache contents against a known-good signature. For pip, this can be pip install --require-hashes with a requirements.lock that pins the hashes. For npm, this can be npm ci with a package-lock.json that pins the versions. The verification step runs after the cache restore and rejects restored content that does not match.
• No secrets in the cache path. The cache path is readable by anyone with workflow read permission. A ~/.aws directory in the cache path leaks credentials. The cache path is narrow and explicit.

- name: Cache pip
  uses: actions/cache@v4
  with:
    path: ~/.cache/pip
    key: pip-${ runner.os }-${ hashFiles('requirements.txt', 'requirements.lock') }
    restore-keys: |
      pip-${ runner.os }-
- name: Install dependencies
  run: pip install --require-hashes -r requirements.txt

The key includes both requirements.txt and requirements.lock, so a lockfile change invalidates the cache. The pip install --require-hashes step rejects any wheel whose hash does not match requirements.lock, even if the cache restored an attacker-controlled wheel. The two defences together reduce the attack surface to a window between the cache restore and the hash verification - a window that the attacker can exploit only by producing a wheel that hashes to a value already in requirements.lock, which is computationally infeasible.

Detecting poisoning in production

Three operational signals suggest that a cache has been poisoned:

• The hash verification step fails after a cache restore. The restored wheel hashes to a value not in requirements.lock. This is the definitive signal; the cache has been tampered with.
• The install step succeeds but the resulting binary behaves unexpectedly. The wheel passed hash verification (it was in requirements.lock) but its contents have been altered in a way that is not detected by the hash. This is rare but possible; it requires a sophisticated attacker who can modify the upstream package registry.
• The cache size grows anomalously. A poisoned entry is often larger than the legitimate entry it replaced. Cache size monitoring catches the third signal; the first two require hash verification.

Production discipline

1. Always hash the lockfile, not the source files. A key derived from requirements.txt and requirements.lock invalidates the cache when the lockfile changes; a key derived from *.py invalidates on unrelated source changes.
2. Use hash-pinned installs after cache restore. pip install --require-hashes, npm ci with package-lock.json, go mod download with go.sum - all of these verify the installed packages against a known-good hash list.
3. Treat the cache path as untrusted input. Never put secrets, never put executable code that is not verified, never put configuration that the workflow trusts without verification.
4. Monitor cache size and cache hit rate. Anomalous growth suggests a poisoned entry; anomalous miss rate suggests an eviction or a key drift.

Cross-course references

• Linux for Production Sysadmins - Part XXXIV (ConfigMgmt) applies the same hash-pinning defence to apt: apt install with a configured Signed-By and a Release.gpg verification.
• Ansible for Production Sysadmins - Part XXXVII (RepoArch) applies the pattern to molecule dependencies, with requirements.yml pinning the collection versions.
• Terraform for Production Sysadmins - Parts IX-XII (State) apply the strictest version: Terraform’s provider plugins are signed by HashiCorp, and the lockfile pins the provider versions.

Quiz