Stale cache returned wrong dependency version

Reported symptoms

●A production binary linked against library `<old-version>` despite the merged PR claiming `<new-version>`
●`package-lock.json` (or `Cargo.lock`, `go.sum`, `poetry.lock`) on the merged commit pins `<new-version>`
●The CI log shows "Cache hit for key: `<broad-key>`" — the key did not change when the lockfile did
●`git diff HEAD~1 HEAD -- package-lock.json | head` shows the version bump; `git diff HEAD~1 HEAD -- <cache-key-input>` shows the cache key unchanged
●The CI workflow's `actions/cache` step uses `key: ${ runner.os }-deps` and a `restore-keys: ${ runner.os }-deps-` fallback
●Production incident reproduces a behaviour the new library version was supposed to fix
●A manual `npm ci` (without the cache) produces a different `node_modules/` tree than the cached one

Evidence

· `cat .github/workflows/<workflow>.yml | grep -A5 "actions/cache"` shows the broad key and the fallback restore-keys
· The cache step logs show: `Cache hit for key: linux-deps-abc123` followed by `Restored cache for key: linux-deps-abc123`
· `sha256sum package-lock.json` and `sha256sum node_modules/<lib>/package.json` differ in the version field
· `npm ls <lib>` in the CI workspace shows `<old-version>`; the same command in a fresh `npm ci` environment shows `<new-version>`
· The first cache miss for the bumped lockfile would have shown: `Cache hit for key: linux-deps-abc123` (because restore-keys matched the broader key) instead of `Cache miss for key: linux-deps-<lockfile-hash>`
· A second run with the lockfile reverted shows the same wrong-version tree — the cache poisoned the next build too
· `gh cache list --key linux-deps-` returns a key with no lockfile hash component
· The cache is the only step that has changed recently; the build script has not

Diagnosis and resolutionclick to reveal

Root cause

The cache key is missing the lockfile hash, so any change to the lockfile produces a key that already exists from a previous build. With a fallback `restore-keys` that matches the prefix, the cache restoration picks up the older, lockfile-mismatched content. `actions/cache` will silently prefer the restored content over a re-install because the workflow trusts the cache; nothing in the workflow verifies that the restored `node_modules/` matches the current `package-lock.json`. The result is a build that is reproducible (every run gets the same wrong tree) but incorrect (the wrong tree does not match the lockfile). This is the canonical cache poisoning pattern: the cache outlives the input it was supposed to mirror, and the workflow trusts the cache rather than re-deriving the truth from the lockfile.

Remediation

Fix the cache key to include the lockfile hash: `key: ${ runner.os }-deps-${ hashFiles(''**/package-lock.json'') }`. For monorepos, scope by path: `${ runner.os }-${ matrix.app }-deps-${ hashFiles(''apps/<app>/package-lock.json'') }`. Then evict the poisoned cache: `gh cache delete <broad-key>` (or delete it via the UI for a specific workflow). Re-run the build; the first run will be a cache miss and will `npm ci` (or equivalent) from the lockfile, producing the correct tree. Subsequent runs will hit the new, correctly-keyed cache. Add a verification step: after `npm ci`, run `npm ls <lib>` and compare the installed version against `jq -r .dependencies.<lib>.version package-lock.json` — if they differ, the cache is the problem and the workflow should fail loudly, not produce a wrong binary.

Verification

`sha256sum node_modules/<lib>/package.json` and `jq -r .dependencies.<lib>.version package-lock.json` agree on the version. The CI log for a run with the corrected key shows `Cache hit for key: <lockfile-hash-suffixed>` and `Restored cache for key: <same>`. The poisoned cache key is deleted from the cache store (`gh cache list` no longer returns it). A run that deletes the cache and re-builds from scratch produces the same tree as a run that hits the cache.

Prevention

The cache key must be a function of every input that affects the cached content. For Node, include `hashFiles(''**/package-lock.json'')` in the key and avoid `restore-keys` that match a broader prefix than the primary key — `restore-keys` should only broaden on the assumption that an empty cache is acceptable, which is true for read-mostly caches but not for dependency trees. Bump a "schema version" into the cache key (`v2-linux-deps-...`) whenever the cache layout or restoration strategy changes, so a stale cache cannot survive a re-platforming of the workflow. Add a post-install verification: assert that the installed version matches the lockfile version, and fail the job if it does not — making the cache a derived view of the lockfile, not a source of truth that the lockfile has to be reconciled with.

A cache key that does not include every input that affects the cached content is a cache that can outlive the input it was supposed to mirror. The fix is the lockfile hash in the key, a narrow restore-keys list, and a verification step that treats the cache as a derived view, not a source of truth.