Runbook: Recover from firewall lockout

This runbook recovers from a firewall lockout. A wrong firewall rule can drop SSH from your management network and make a remote host unreachable except via the console. The recovery is to access the console, inspect the rules, and correct the offending rule.

When to use this runbook

Use this runbook when:

• A host becomes unreachable via SSH after a firewall change.
• The host responds to ping but SSH times out.
• Multiple management hosts are also locked out.
• The cloud console or hypervisor console is available.

Inputs

Gather before starting:

• The host name and IP.
• Cloud or hypervisor console access credentials.
• Break-glass credentials for the host.
• The expected SSH source networks.
• Any recent firewall changes (git log, configuration management history).

Procedure

Step 1: Access the console

For cloud hosts:

AWS: EC2 > Instances > Connect > Serial Console or Session Manager
Azure: Virtual Machines > Support + troubleshooting > Serial Console
GCP: Compute Engine > VM > Connect > Serial Console

For on-prem hypervisors:

VMware: vSphere > Host > Open Console
Proxmox: VM > Console
Hyper-V: VM > Connect

Log in with break-glass credentials.

Step 2: Capture the live ruleset before you change it

Do this before the first nft delete, not after. The rollback section of this runbook points at the file this step creates; nothing else creates it, and on a console with no scrollback there is no other copy of what was running.

BACKUP=/var/backups/nftables-$(date +%F-%H%M%S).conf
sudo nft list ruleset | sudo tee "$BACKUP" >/dev/null
sudo test -s "$BACKUP" && echo "captured: $BACKUP"

sudo test -s is not decoration. nft list ruleset on a host with no ruleset loaded prints nothing and still exits 0, which would leave you with an empty “backup” that silently flushes everything if you ever load it.

Note that the redirection has to go through tee: in sudo nft list ruleset > /var/backups/... the shell opens the output file before sudo runs, so the write happens as the unprivileged user and fails with Permission denied on /var/backups.

Step 3: Inspect the firewall

sudo nft list ruleset
sudo iptables -L -n -v
sudo firewall-cmd --list-all
sudo ufw status verbose

Identify the firewall tool in use. Look for rules that might be blocking SSH:

• An INPUT chain with policy drop and no ACCEPT for SSH source.
• A specific DROP rule that matches your management network.
• A FORWARD rule that drops traffic between management and the host.
• A recent rule added that may have broken connectivity.

Step 4: Identify the offending rule

Compare to a known-good ruleset (in version control or the configuration management system):

# If managed by Ansible
cat /etc/nftables.conf
git log /etc/nftables.conf

# If using firewalld
firewall-cmd --list-all

The most common lockouts:

• policy drop on INPUT with no allow for the management network.
• An explicit DROP for the management source IP.
• A FORWARD chain that blocks management->host traffic.

Step 5: Apply the fix at runtime only

Restore access first. Do not persist anything yet, and do not reload the firewall. Reloading re-applies the saved configuration, which is the configuration that locked you out.

For nftables:

# Insert an allow for the management network
sudo nft insert rule inet filter input ip saddr 10.0.0.0/24 accept

For iptables:

sudo iptables -I INPUT -s 10.0.0.0/24 -j ACCEPT

For firewalld:

# Runtime change - takes effect immediately and restores access
sudo firewall-cmd --zone=trusted --add-source=10.0.0.0/24

For ufw:

sudo ufw allow from 10.0.0.0/24

Step 6: Test from a known-good source

From your workstation or another host, before you persist anything:

HOST=203.0.113.10          # substitute the host address
ssh -o ConnectTimeout=5 user@"$HOST"

Set HOST rather than pasting an angle-bracket placeholder: ssh user@<host-ip> is not a prompt to substitute a value, it is a shell redirection, and it fails with a syntax error instead of telling you what to fill in.

Confirm SSH works. If it does not, stay on the console and go back to Step 4. Persisting an unverified ruleset turns a runtime mistake into a permanent one.

Step 7: Persist the verified fix

The runtime fix is gone on reboot. Only now, with access proven, promote it:

# firewalld - promote the whole working runtime config
sudo firewall-cmd --runtime-to-permanent

# ufw
# Already persistent if you used ufw allow

For nftables, do not pipe nft list ruleset straight over the boot file:

# WRONG - destroys the file's script header
sudo nft list ruleset | sudo tee /etc/nftables.conf

nft list ruleset prints only table ... { } blocks. It does not print the #!/usr/sbin/nft -f interpreter line, and it does not print flush ruleset. On Debian and Ubuntu the shipped /etc/nftables.conf begins with exactly those two lines. Overwrite them and the file stops being a valid nft script, and - worse - the next systemctl restart nftables or boot adds the rules on top of whatever the kernel already holds instead of replacing them. You get duplicated chains and duplicated rules, in an order you never tested, during an incident you have just closed.

Write the header back, parse-check, then install:

{ printf '#!/usr/sbin/nft -f\n\nflush ruleset\n\n'; sudo nft list ruleset; } \
| sudo tee /etc/nftables.conf.new >/dev/null

# Parse-only. Loads nothing, changes nothing.
sudo nft -c -f /etc/nftables.conf.new

# Install, keeping a backup of the file that locked you out
sudo install -m 0755 -b /etc/nftables.conf.new /etc/nftables.conf

install -b leaves the previous file as /etc/nftables.conf~, which is your evidence of the rule that caused the lockout. Copy it somewhere durable before it is rotated away.

firewall-cmd --runtime-to-permanent writes the current runtime configuration into /etc/firewalld/, so runtime and permanent now agree. That is what makes the next reload safe. The alternative is to re-state the change with --permanent and then reload:

sudo firewall-cmd --permanent --zone=trusted --add-source=10.0.0.0/24
sudo firewall-cmd --reload

That form is safe only because the permanent config already contains the allow rule before the reload runs. If you get the order wrong, or mistype the permanent rule, the reload locks you out again. Prefer --runtime-to-permanent during an incident.

Confirm the two configurations match:

sudo firewall-cmd --zone=trusted --list-sources
sudo firewall-cmd --permanent --zone=trusted --list-sources

Step 8: Document and prevent

Document in the incident log:

• What rule caused the lockout.
• What fixed it.
• What should change to prevent recurrence (e.g. configuration management test, automated rollback, always have out-of-band access).

For prevention:

• Arm a timed auto-revert before every firewall change on a remote host. This is the control that prevents the lockout; everything else on this list only shortens the recovery.
• Use configuration management (Ansible, Puppet) to apply firewall rules. Manual changes are error-prone.
• Test firewall changes in a non-production environment first.
• Use atomic reloads (nft -f) instead of incremental changes.
• Validate the ruleset after every change with an external scan.
• Keep the working SSH session open and test from a second, independent session. Your existing session survives on an established conntrack entry even when the new rules block every new connection, so it proves nothing.
• Always have out-of-band access (cloud console, IPMI, iLO).

The auto-revert pattern

Two commands, run before the change, not after:

# 1. Snapshot the known-good ruleset. sudo tee, not a redirect - see Step 2.
sudo nft list ruleset | sudo tee /root/nft-rollback.conf >/dev/null
sudo test -s /root/nft-rollback.conf || { echo 'empty snapshot - do not arm the revert'; exit 1; }

# 2. Arm the revert
sudo systemd-run --on-active=5min --unit=nft-rollback \
   /usr/sbin/nft -f /root/nft-rollback.conf

# 3. Make the change, then confirm access from a NEW session

# 4. Only then disarm
sudo systemctl stop nft-rollback.timer

If step 3 fails, do nothing at all. The timer restores the snapshot in five minutes and you never needed the console. The at equivalent works the same way where at is installed:

echo '/usr/sbin/nft -f /root/nft-rollback.conf' | at now + 5 minutes

firewalld has this built in. A runtime rule with a timeout expires on its own, so a bad rule cannot outlive your session:

sudo firewall-cmd --zone=public --add-service=ssh --timeout=5m

Give the snapshot file the same #!/usr/sbin/nft -f and flush ruleset header described in Step 7 if you intend to keep it as a boot file. For a five-minute revert the missing flush is harmless, because the snapshot is a superset of what is loaded.

Common patterns

Escalation

Escalate when:

• The console is unavailable (provider outage).
• Break-glass credentials do not work.
• The host is part of a critical cluster and the lockout affects quorum.
• The lockout is one of multiple symptoms of a larger incident.

Bring: the host, the firewall tool, the expected ruleset, the recent changes.