Firewall and routing recovery

Recovering a network device’s configuration onto its own hardware, where the previous lesson ended, restores a file that still means what it meant. Recover that file onto a replacement chassis or into a recovery site and every statement in it becomes conditional on something that changed. Each rule names an interface, each translation names an address, each tunnel names a peer and an identity, and none of those names is checked against the world when the configuration loads. The device applies what it was given and reports that it did. The first honest signal arrives later, as a packet that does not.

The firewall that boots, reports healthy and forwards nothing

The dominant failure in firewall recovery is not a corrupt backup. It is a good backup, applied faithfully, onto hardware that enumerates its interfaces differently from the hardware the configuration was written for.

Every platform has the same join and spells it differently. On OPNsense the join is an assignment between a named interface and a physical port: the interface documentation states that most interfaces have to be assigned to a physical port, that LAN and WAN are assigned to ports 0 and 1 by default, and that the assignment is changed under Interfaces ‣ Assignments, where the interface name sits on the left and the physical port is chosen from a dropdown. On VyOS the join is the interface name in the configuration tree, eth0 and its siblings, and the saved configuration records hardware beside it: the CLI documentation’s own example of a running configuration carries an hw-id holding a MAC address under interfaces ethernet eth0, and the ethernet documentation adds a mac option that sets a custom MAC address on the interface. On a general-purpose Linux host, systemd’s naming scheme derives names such as enp2s0f1 from firmware index numbers, PCI paths and MAC addresses: the name is a function of where the card is, not of what it does. Each scheme is stable while the hardware is, and each is a name the configuration uses to mean the segment where the users are. Replacement hardware breaks the join in one of two ways, and the second is far worse.

In the first, the named device is simply absent. A chassis with different network cards presents device names the configuration never mentions, or four ports where the old one had six. The interface never comes up, rules referencing it cannot match, and the segment behind it is dark. This is the benign case, because it is loud.

In the second, the name exists and is attached to a different segment. The replacement has the same device names, the cabling was done from port numbers rather than roles, and the configuration binds WAN policy to the cable that runs to the user VLAN. Every check the device can perform is satisfied — link up, addresses configured, rule set loaded, dashboard green — while the internet-facing policy is enforced on the inside and the inside policy on the internet-facing port.

Nothing on the device detects either case, because every health signal a firewall emits is a statement about the firewall. Link state says a cable is in a socket, “rules loaded” says the rule set compiled, a healthy HA member says two devices can talk to each other. None of them describes the path a packet takes from a laptop to the payroll server, which is the only thing anybody wanted to know.

The interface map belongs in the backup, not in a diagram

Because the join between configuration and hardware is what breaks, the join is what has to be captured. A configuration export records the left-hand side of the mapping — the role, the logical name, the rules that reference it — and nothing about the right-hand side, which is the physical world. Record that half separately, on the same schedule, beside the export it belongs to: per port, the logical role, the device name the platform reports, the MAC address, the PCI or bus path, the kernel driver bound to the card, the upstream switch and port learned from LLDP, the VLAN tags, the addresses and gateway, and the MTU. On a Linux-based device the capture attaches to the job that already exports the configuration; lspci -nnk is the flag that adds the bound driver to the vendor and device identifiers, and lldpcli show neighbors details is the lldpd invocation documented for the neighbour dump.

MAPDIR=/backup/netmap/edge01
mkdir -p "$MAPDIR"
ip -o link show > "$MAPDIR/links.txt"
ip -o addr show > "$MAPDIR/addrs.txt"
ip route show table all > "$MAPDIR/routes.txt"
lspci -nnk > "$MAPDIR/pci.txt"
lldpcli show neighbors details > "$MAPDIR/lldp.txt" 2>/dev/null ||
  printf 'no lldp daemon on this device\n' > "$MAPDIR/lldp.txt"

The LLDP capture is the line people leave out and the one that ends arguments during a rebuild, because it records what each port is connected to from the port’s own point of view rather than a diagram’s.

With the map in hand, recovery onto different hardware becomes a translation rather than a hope. Enumerate the interfaces the new device has, write the new map beside the old one, and rewrite the identifiers in the configuration before it is applied. On VyOS that means reviewing every MAC address the saved configuration carries — the hw-id recorded beside an ethernet interface and any mac statement setting a custom address — because each of them names a card the replacement chassis does not have. On OPNsense it means redoing the assignment under Interfaces ‣ Assignments so each interface name sits against a physical port that exists, which is the documented path and not a workaround. Both are deliberate edits with a reviewable diff.

The state a stateful firewall cannot restore

A stateful firewall makes two decisions and the configuration describes only one of them. The rule set decides whether a new conversation may begin; the state table records the conversations that already did. A restore brings back the first and cannot bring back the second, a record of connections in flight at a moment which has passed.

Everything unpleasant about the first minutes after a firewall recovery follows from that. Established sessions are gone, so routing protocol peerings, database connections, replication streams and administrative logins all re-establish, and an application holding a connection open without a keepalive discovers the loss only when a timeout expires. Clients that never noticed the outage send their next packet mid-stream: not a connection opening, matching no state, and irrelevant to a rule about new connections. It is dropped, the client hangs, and recovery looks worse from the client than from the device until the retries work through.

NAT bindings are state as well. The mapping between an inside address and port and the translated port the far side saw is allocated when the conversation starts and goes with the rest of the table, so anything that pinned a translated port meets a different one after recovery.

State synchronisation between HA members exists precisely so a peer can take over without dropping sessions, and it is a design decision taken long before the event. A device recovered alone at a DR site has no peer to learn from, so sessions break; write that into the plan as an expected outcome with a stated duration.

Rules and NAT that name addresses the recovery changed

A recovery site is a different address space, and a rule set is a document full of addresses. Outbound NAT written with an explicit translation address only true on the production WAN either fails to apply or translates traffic to something the new upstream will not route. Port forwards point at servers rebuilt into a different subnet, so the rule is valid, loads cleanly, and forwards where nothing is listening.

Aliases and address groups defined by hostname fail most quietly of all. They resolve when the rule set is built or refreshed, and DNS is frequently part of what is still being recovered. One that resolves to nothing yields a rule matching nothing and reports no error; one that resolves to a stale record permits the wrong host, which is worse and equally silent.

What predicts whether a rule survives re-addressing is whether it names a segment through its interface or through a literal prefix. Interface-relative constructs follow the address the interface actually holds and go on meaning “the users’ segment” at the new site; literals meant what they meant at the old one. The audit begins by counting the literals and naming who owns each value at the recovery location.

CONF=/backup/edge01/config.xml
grep -Eo '([0-9]{1,3}\.){3}[0-9]{1,3}(/[0-9]{1,2})?' "$CONF" | sort -u > /tmp/literals.txt
wc -l < /tmp/literals.txt

Routing carries the same problem with a larger blast radius. The default gateway is a different next hop, monitoring targets may be unreachable for reasons unrelated to the gateway, and static routes point at sites themselves in recovery. Dynamic routing is the sharpest edge: a recovered router that forms an adjacency and re-advertises the prefixes it used to own can pull live traffic toward a site that is only half recovered, converting a partial outage into a total one. Bring the routing plane up last, with export policy that advertises nothing until the services behind each prefix have been validated from a client, then relax it deliberately. The same caution applies to virtual router identifiers: a restored device speaking VRRP or CARP with an identifier still live elsewhere puts two speakers on one address. And asymmetric paths, where traffic leaves through one device and returns through another, give the firewall one direction of a conversation and no state match on the other, failing connections in a pattern that tracks load balancing rather than policy and reads as an application fault for hours.

Tunnels are agreements, and identity is not restorable by one side

A restored IPsec configuration describes a relationship. The far end holds the matching description, naming your old public address, your old identity and frequently an access list containing that address; restoring your half changes nothing there. Where the peer belongs to another organisation, the change is a request in their change process measured in days — which the plan should pre-arrange as a second peer definition for the recovery address, agreed in advance and left disabled until needed.

Identity is the part people expect to be configuration and find is not. Peers authenticate as an identity — an address, an FQDN, or a certificate subject — and each end matches the one it expects. A recovery site with a different public address fails that match before authentication says anything, and the log line reads like a key problem when it is a naming problem. An FQDN identity backed by a record you control survives re-addressing, at the cost of putting DNS on the critical path ahead of the tunnel.

Key material is the other half, and it may or may not be inside the configuration export. Stripping it is defensible, since an unstripped export is one file holding every secret the device has, but the restored device then authenticates nothing until the escrowed material arrives — and the escrow has to be reachable without the tunnel waiting for it. Leaving it in means the configuration backup is key material and inherits every custody rule from the key recovery material earlier in this course. Static-key tunnels make the point exactly: the peer relationship is the key pair, so restoring the device’s private key preserves every peer’s trust, while a new one makes each peer’s configuration wrong until its administrator changes it. Certificates from an internal CA lost in the same event put the PKI ahead of the firewall in the recovery order.

Validation from a client position, and staged application

Testing a firewall from the firewall tests the wrong thing. Traffic the device originates uses its own routing table and source address selection, and locally originated traffic is not the transit traffic the rule set was written about. A successful ping from the console to the DMZ server establishes that the firewall can reach the server, which nobody asked. The question is whether a client in the user segment can, and only a client in that segment answers it.

Build the matrix before the incident: for each pair of segments that matters, one flow that must work and one that must not. Assert the denials, because a device that lost its rule set passes every allow test on the sheet. Test the application transaction rather than the handshake, and include a transfer large enough to be fragmented, because a smaller MTU at the recovery site yields a network that completes every connection and stalls every transfer.

TARGET=10.20.0.15
ALLOWED_PORT=443
DENIED_PORT=22
if timeout 3 bash -c "exec 3<>/dev/tcp/$TARGET/$ALLOWED_PORT" 2>/dev/null; then
  printf 'ok: allowed flow to %s:%s completed\n' "$TARGET" "$ALLOWED_PORT"
else
  printf 'FAIL: allowed flow to %s:%s was blocked\n' "$TARGET" "$ALLOWED_PORT"
fi
if timeout 3 bash -c "exec 3<>/dev/tcp/$TARGET/$DENIED_PORT" 2>/dev/null; then
  printf 'FAIL: denied flow to %s:%s was permitted\n' "$TARGET" "$DENIED_PORT"
else
  printf 'ok: denied flow to %s:%s did not complete\n' "$TARGET" "$DENIED_PORT"
fi

Staging is what keeps a mistake from removing the hands doing the work. Secure an access path that does not depend on the rules being edited — a console server, an out-of-band port, or a documented physical route — and prove it works before touching anything. Then apply in dependency order: interfaces and addressing, routing, NAT, rules, tunnels last, validating from a client at each step. A failure in addressing produces symptoms that read exactly like a rule problem, and an hour spent debugging correct rules comes out of the recovery window.

Where the platform offers a commit that reverts itself unless confirmed, use it. VyOS documents commit-confirm, taking a number of minutes, as a temporary commit that requires confirm inside the window and otherwise reverts to a previous configuration, with a documented default window of ten minutes. The documented default action is a reboot into the saved configuration, while set system config-management commit-confirm action reload selects the alternative, which the documentation describes as loading the most recent completed configuration without a reboot. Either converts a lockout into a wait, and the choice between them decides how long that wait is. Platforms without that mechanism need the equivalent built by hand — a scheduled restore of the previous configuration, cancelled once the new one is proven from a client — with that previous rule set kept retrievable, so rolling back is a restore rather than a re-derivation under pressure.

Production discipline

1. Capture the interface map on the schedule that captures the configuration. Role, device name, MAC, bus path, driver, LLDP neighbour, VLAN tags, addresses and MTU, stored beside the export, which describes only the half of the mapping that lives in software.
2. Translate interface identifiers before applying, never after. Enumerate what the replacement presents, rewrite the MAC addresses the configuration records and the interface-to-port assignments as a reviewable diff, and treat “apply it and see” as a second incident started inside the first.
3. Plan for an empty state table. Sessions break, mid-stream packets are dropped and NAT bindings are re-allocated; state synchronisation between HA members is a design decision taken before the event, not something a lone recovered device acquires afterwards.
4. Prove reachability from a client in each segment, including the denials. The device’s own signals report on the device, and a green dashboard is equally consistent with correct cabling and with two segments swapped.
5. Apply in stages behind an access path the rules cannot remove. Console or out-of-band first, then addressing, routing, NAT, rules and tunnels, with a timed rollback such as commit-confirm where the platform offers one and a hand-built equivalent where it does not.

Cross-course references

• OPNsense for Production Network & Security Administrators — Part VII (Interface Management) configures the logical-to-physical assignment this lesson insists on redoing deliberately on replacement hardware; Part XI (Firewall States) and Part XXVIII (pfsync and Configuration Synchronisation) cover the state table a restore cannot bring back, and Part XXXIX (Lockout Prevention and Console Recovery) is the independent access path staged rule application depends on.
• VyOS for Production Network Engineers — Part VII (Interface Fundamentals) covers the ethernet settings a restored configuration carries, including the MAC addresses that tie a stanza to one particular card, Part VI (Commit and Rollback Safety) is the commit-confirm mechanism recommended above, and Part LV (Backup, Restore, Disaster Recovery) is the device-side procedure this lesson extends by asking what the restored file assumes about the world.
• Linux for Production Sysadmins — Part XX (Linux Network Configuration) explains the predictable naming that makes an interface name a function of the card’s position rather than its role, the mechanism behind the renaming failure above, and Part XXV (Firewalls) covers host rule sets and connection tracking, where the same distinction decides what a rebuilt host does with traffic in flight.

Quiz