critical riskcluster affecting~240 min

Runbook: Compromised repository response

1 · Prerequisites

Confirm every item is in place before any state change.

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Confirm the compromise: vendor advisory, package misbehaviour, or signing key mismatch
· Identify the window: from when the compromised key/repository was trusted to now
· Identify the affected hosts: those that ran apt update or dnf upgrade in the window
· Confirm the audit trail: who added the key, when, and from what source

Execute each step in order. Verify the expected output of a step before moving to the next.

1Contain. Disable the compromised repository on every host. Set enabled=0 in /etc/yum.repos.d/ or remove from /etc/apt/sources.list.d/. apt update / dnf check-update to flush.
2Quarantine. Identify hosts that installed packages from the compromised repository in the window. The SBOM and the package manager history are the source of truth.
3Rotate the trust. Remove the compromised key from /etc/apt/trusted.gpg.d/ or /etc/pki/rpm-gpg/. If the vendor has rotated the key, install the new key. Verify the new key fingerprint against the vendor published fingerprint via an out-of-band channel.
4Reinstall packages. For affected packages, re-verify them against a different mirror or the new vendor key. If verification fails, the package was tampered with - reinstall from a clean source.
5Rotate secrets. Any secret that was on the affected hosts (private keys, API tokens, database credentials) must be considered compromised. Rotate all secrets on affected hosts.
6Reboot if necessary. For deeply compromised hosts (kernel or initramfs tampered), reboot from known-good media, restore from snapshot, or rebuild. The judgement depends on what was accessed.
7Document. Record the timeline: when the compromise was detected, when the trust was disabled, when secrets were rotated, when hosts were restored. The post-incident review uses this.

Confirm the procedure actually fixed the problem.

If verification fails, undo the procedure in reverse order.

↶If a host cannot be brought back to a known-good state (no backup, no reinstall path), rebuild from known-good media. A compromised host that cannot be verified is not safe to return to production.
↶Document the failed host in the incident log; do not return it to production until it is rebuilt.

When the runbook isn't enough, contact:

· If the compromise is in a vendor signing key, escalate to the vendor immediately; coordinate with them on the remediation timeline.
· If the compromise is in a third-party repository, escalate to the third-party vendor; consider removing the repository fleet-wide until the vendor confirms remediation.
· If the compromise includes active exploitation (not just tampered packages), engage the incident response team and consider network isolation of affected hosts.

A vendor advisory reports a compromised signing key or repository.
A host shows unexpected packages, processes, or network connections that are not in the SBOM.
A package signature verification fails.
An out-of-band communication (vendor email, security advisory) reports a breach.

The first step is to confirm the compromise and the scope. Do not act on rumours; act on confirmation.

Vendor advisory — read the vendor’s published advisory; confirm the key fingerprint, the affected packages, and the remediation timeline.
Host inventory — query every host for packages from the affected repository in the window. Use the SBOM and the package manager history.
Signature verification — on a sample host, run rpm --checksig or dpkg --verify against a known-good package. If the signature fails, the host is affected.
Audit trail — who added the key, when, from what source, and via what procedure. A documented procedure for adding keys is the difference between “we know exactly what to revert” and “we are guessing”.