Emergency patching - reacting to critical CVEs under pressure · Linux

A critical CVE with active exploitation is an emergency. The response must be fast, coordinated, and documented.

The first 30 minutes

T+0  : CVE published with active exploit
T+5  : Security team confirms the host is affected
T+10 : Security team determines scope
T+15 : On-call team notified
T+30 : Patch staged, change communicated, deploy begins

Scope determination

For each host, check:

# Substitute your own values before running:
PACKAGE=nginx
SERVICE=nginx
PORT=443

# Is the affected package installed?
rpm -q "$PACKAGE"          # RHEL family
dpkg -l "$PACKAGE"         # Debian

# Is the service running?
systemctl status "$SERVICE"

# Is the service internet-facing?
ss -tlnp | grep "$PORT"

Document the affected hosts.

Communication

For emergency changes, notify:

On-call team.
Service owner.
Incident management (if the patch may cause outage).
Management (for critical services).

INCIDENT: PATCH-EMERGENCY 2026-08-09
CVE: CVE-2024-XXXXX (Critical, active exploit)
Affected: web01, web02, web03
Action: Patch within 60 minutes
Risk: &lt;5 min downtime per host during patch
Rollback: revert via package manager
On-call: <contact>

Patch under pressure

Patch the affected package, not the whole host, and stop at the first host that fails its health check:

#!/usr/bin/env bash
set -euo pipefail

PKG=nginx
FIXED=1.27.2-1          # the fixed version from the advisory

# Test in staging first
ssh staging "sudo apt-get update && \
    sudo DEBIAN_FRONTEND=noninteractive apt-get install -y \
    -o Dpkg::Options::=--force-confold --only-upgrade ${PKG}=${FIXED}"
ssh staging "sudo systemctl restart ${PKG}"
curl -fsS -o /dev/null https://staging.example.com/health

# Then production, one host at a time, with a gate between hosts
for h in web01 web02 web03; do
    ssh "$h" "sudo apt-get update && \
        sudo DEBIAN_FRONTEND=noninteractive apt-get install -y \
        -o Dpkg::Options::=--force-confold --only-upgrade ${PKG}=${FIXED}"
    ssh "$h" "sudo systemctl restart ${PKG}"
    sleep 30
    if ! curl -fsS -o /dev/null "https://${h}.example.com/health"; then
        echo "ABORT: ${h} unhealthy after patch - roll it back." >&2
        echo "Tier is PARTIALLY patched. Do not continue." >&2
        exit 1
    fi
    echo "${h} patched and healthy"
done

Four details in that loop carry the whole procedure:

curl -fsS plus the if ! test. curl alone returns 0 for an HTTP 500 - -f makes it fail on 4xx and 5xx, and the test is what turns the result into a gate.
--only-upgrade ${PKG}=${FIXED} patches one package to one known version. Under emergency pressure you want the smallest change that closes the CVE.
DEBIAN_FRONTEND=noninteractive and -o Dpkg::Options::=--force-confold. Over ssh there is no tty, so a locally modified conffile would stop the run on a dpkg prompt nobody can answer.
apt-get install --only-upgrade rather than apt upgrade. Plain apt upgrade silently skips any upgrade that would remove a package, which is how kernel and some security updates get held back without an error.

A loop with no gate patches the whole tier into an outage

The natural shape of this loop is the dangerous one:

# WRONG - no health gate, blanket upgrade
for h in web01 web02 web03; do
    ssh $h "sudo apt update && sudo apt upgrade -y"
    ssh $h "sudo systemctl restart nginx"
    sleep 30
    curl -I https://$h.example.com
done

The curl result is never tested, so the loop continues whatever it printed. If the patch breaks nginx on web01, web02 receives the identical change 30 seconds later and web03 30 seconds after that. A one-third outage becomes a total one, during an incident, under maximum time pressure.

apt upgrade -y compounds it. Under an emergency you have tested one package change in staging; the blanket upgrade ships every pending package update on the host alongside it, so any regression is now impossible to attribute. Blanket upgrades belong in the scheduled maintenance window.

Verify

After the patch:

Smoke tests pass.
New CVE not present in the next scan.
No new errors in journald or auditd.
Service level objectives met.

Post-mortem

Within 24-48 hours:

What was the CVE?
How was the host affected?
How was it detected (vendor advisory, scanner, exploit attempt)?
What was the response time?
What worked, what did not?
What changes for next time (faster detection, better testing, automated patches)?

The post-mortem is the input to process improvement.

Post-emergency: audit and cleanup

After the immediate fire:

Run a full vulnerability scan.
Verify no other hosts are affected.
Update the patch priority decision rules.
Document exceptions for any hosts that could not be patched.
Communicate to management.