The firewall restored onto replacement hardware and passed no traffic

Reported symptoms

●The replacement firewall rbdr-fw-edge-02 boots in eleven minutes, shows carrier on all five interfaces, holds every address the design document specifies, resolves DNS from its own diagnostics page and reaches the internet from itself
●No workstation on the office VLAN can reach its default gateway 10.20.10.1, and the failure is silent rather than refused: packets are neither answered nor rejected
●Machines that renewed a DHCP lease after the cutover came back with addresses in 10.20.40.0/24, the voice range, although they are cabled to the office switch as they always were
●The DMZ web host is unreachable from outside and cannot reach out, and the management VLAN can no longer poll anything behind the firewall
●The firewall logs contain no blocks for the affected traffic and no rule-load errors; the state table shows the handful of sessions the device originated itself and nothing else
●The four copper cables were re-made socket for socket from the rack diagram, and every socket shows link at the negotiated speed
●The incident bridge opens on the theory that the configuration restored incompletely, because a firewall that passes nothing usually has lost its rules or its NAT

Evidence

· The interface assignment read out of the configuration export: WAN on ix0, LAN on igb0, DMZ on igb1, MGMT on igb2, VOICE on igb3, and the same five pairings read back off the restored device, unchanged
· The device-to-socket map established on the replacement chassis by pulling one cable at a time and watching which device lost carrier: panel socket 1 is igb3, socket 2 is igb2, socket 3 is igb1, socket 4 is igb0 - the reverse of the chassis that died, because the four copper ports are now two dual-port controllers whose slots probe in the opposite order
· A fresh configuration export taken from the restored device, normalised and diffed against the source export: the rule sets, aliases, NAT entries and interface assignments are identical, and the only differences are the fields the appliance regenerates for itself, namely the system UUID, the SSH host keys and the self-signed GUI certificate
· A packet test from rbdr-probe-lan on the office VLAN: ping to 10.20.10.1 draws no reply and the neighbour entry for that address stays incomplete, so the exchange fails at address resolution, before any firewall rule is consulted
· A capture taken on the firewall at the same moment shows those address-resolution requests for 10.20.10.1 arriving on igb3, an interface configured 10.20.40.1/24, which is why the device correctly declines to answer them
· The post-restore checklist signed off before handover, whose seven items are all executed by the device on itself: interfaces up, addresses correct, gateway online, services running, DNS resolving, ping to an external host, GUI reachable

Diagnosis and resolutionclick to reveal

Root cause

Two defects. The first put the addresses on the wrong wires; the second is the reason a checklist was signed and the fault reached the morning. **Interface names are derived from the hardware, and the hardware changed.** A configuration file does not store ports, it stores names, and a name such as `igb1` is not an identifier the appliance chose. It is assembled by the driver that claimed the device and the order in which the bus was probed. Restore that file onto a different board and every name still resolves, because the replacement also has four `igb` ports, so nothing errors and nothing is dropped. What the names now point at is a different set of sockets. On the chassis that died the four copper ports were one quad controller enumerated left to right; on the replacement they are two dual-port controllers whose slots probe in the opposite order, so panel socket 1 is `igb3` and panel socket 4 is `igb0`. The cabling was re-made faithfully to the rack diagram and is therefore faithfully wrong: the office switch now lands on the interface holding `10.20.40.1/24` and the voice VLAN rules, and the rest of the segments are rotated to match. **The control that should have caught it was a self-test.** Every item on the post-restore checklist is a question the device asks about itself, and a device is on every one of its own networks by definition. `10.20.10.1` exists whether it is configured on the socket facing the office switch or the socket facing the voice switch, so "the LAN interface holds the right address" is true in both worlds. The checklist could not fail. It contains no step that requires a second machine, and a firewall exists to be a path between machines, so the property nobody measured is the only property that matters.

Remediation

Change nothing in the rule set. The rules are correct and were restored correctly; editing them to make traffic flow would encode the wrong mapping permanently and leave a rule base nobody can reconcile with the design document. Fix the assignment instead. Establish the device-to-socket map empirically on the replacement rather than assuming it: pull one cable at a time and record which device loses carrier, or read the neighbour that each switch reports on each port. Then re-point each logical interface at the device that carries its network - on OPNsense through Interfaces then Assignments in the web interface, or option 1 of the console menu when the web interface is on the wrong wire and unreachable. Reassignment rewrites only the logical-to-device pairing; the rules, aliases and NAT entries hang off the logical name and follow it. Re-cabling to match the enumeration would also restore service and is the wrong choice: it leaves the physical plant disagreeing with the rack diagram and the port labels, and the next person to touch the rack recreates the incident. Then clear the leases the wrong scope handed out. A workstation holding a `10.20.40.0/24` address will keep it until the lease expires, so the segment is not recovered when the assignment is fixed - it is recovered when the clients have renewed.

Verification

Verification for this class of fault is defined by where the prober stands. Nothing measured from the firewall counts, because the firewall is on all of these networks whatever the cabling says. Place a host on each protected segment - `rbdr-probe-lan`, `rbdr-probe-dmz`, `rbdr-probe-mgmt` - and from each one resolve the gateway at layer 2 first: the neighbour entry for the gateway address must complete, which proves the address is present on that wire and not merely present somewhere on the device. Then exercise a path the rules are supposed to permit and a path the rules are supposed to refuse, and confirm the refusal appears in the firewall log against the interface you expect. A permitted flow proves the wire; a logged block proves the rule set is attached to the wire you think it is. Take a DHCP lease on each segment and check the range it comes from. That single observation would have caught this incident on its own, and it costs one command from one client. Finally, record the device-to-socket map of the replacement chassis alongside the configuration export, so the next restore begins from a measurement rather than from the assumption that two boards enumerate alike.

Prevention

**Treat the interface map as part of the recovery artefact, not as a property of the configuration file.** An export captures logical names. What those names denote is a property of the chassis, and a chassis is exactly the thing that changes during a hardware recovery. Store a map of logical interface, device name, MAC address, panel socket and switch port next to the export, and expect the device-name column to be wrong on any replacement. **Ban self-tests from post-restore validation of network devices.** Every check that the device runs against itself passes in both the working and the broken topology. Require at least one probe host per protected segment and make its result the sign-off, the same way this course requires a performed restore rather than a green job. **Prefer identifiers that survive a chassis swap where the platform offers them.** Linux systems can name interfaces from firmware and topology rather than probe order, and some appliances let you bind a logical interface to a MAC address. Neither removes the problem on replacement hardware, since the new board has new MAC addresses, but both remove the silent reordering that happens on the same board after a card is added. **Rehearse the restore onto the spare, not onto the same model on paper.** A DR test that restores the export onto identical hardware answers a question nobody will be asked at three in the morning. The chassis on the shelf is the one the recovery will use, and the enumeration difference is discovered in the rehearsal or in the incident. **Alert on segment reachability, not on device health.** A monitoring check that polls the firewall confirms the firewall is running. A check that traverses the firewall from each segment confirms it is doing its job, and it is the only one of the two that would have paged.

Reported symptoms

The edge firewall rbdr-fw-edge-01 lost both power supplies overnight. The spare, rbdr-fw-edge-02, is a later model from the same vendor: same SFP+ WAN port, same four copper ports, different board. The team restored the nightly configuration export onto it and re-made the cables socket for socket from the rack diagram.

Eleven minutes later the device is green: WAN gateway online, carrier on all five interfaces, every logical interface holding the address the design document specifies, the internet reachable from the firewall.

Nothing behind it is. Office workstations have no default gateway; machines that renewed a lease came back in 10.20.40.0/24, the voice range, though cabled to the office switch as always; the DMZ host is unreachable both ways. The firewall log holds no blocks and no rule-load errors.

Evidence provided

The interface assignment, before and after. From the export: WAN on ix0, LAN on igb0, DMZ on igb1, MGMT on igb2, VOICE on igb3 — and the same five pairings read back off the restored device. The map that moved is the other one:

Service impact possiblewhich device name is which panel socket

$ ifconfig -a | grep -E '^(igb|ix)[0-9]|status:'

Pulling one cable at a time gives socket 1 as igb3, socket 2 as igb2, socket 3 as igb1, socket 4 as igb0 — the reverse of the dead chassis, whose copper ports were a single quad controller.

The rule set, still naming the old interfaces. A fresh export from the restored device, normalised and diffed against the source export:

EXPORT=/srv/rbdr-restore/rbdr-fw-edge-01-config.xml
FRESH=/srv/rbdr-restore/rbdr-fw-edge-02-config.xml

xmllint --format "$EXPORT" > "$EXPORT.norm"
xmllint --format "$FRESH" > "$FRESH.norm"
diff -u "$EXPORT.norm" "$FRESH.norm"

Rules, aliases, NAT entries and assignments are identical. The only differences are fields the appliance regenerates for itself: the system UUID, the SSH host keys and the GUI certificate.

A packet test from a client. From rbdr-probe-lan, on the office VLAN:

GW=10.20.10.1

ip -4 neigh flush all
ping -c 3 -W 2 "$GW"
ip -4 neigh show "$GW"

No reply, and the neighbour entry stays incomplete — the exchange fails at address resolution, before any rule is consulted. A capture on the firewall at the same moment shows those requests for 10.20.10.1 arriving on igb3, an interface configured 10.20.40.1/24:

Read-only / Safewhere the client traffic actually lands

$ tcpdump -n -i igb3 -c 20 'arp or (udp port 67 or udp port 68)'

Work the evidence before reading on

The diff shows the rule set intact. What does that settle, and what does it leave open?
The client fails at address resolution rather than being refused. Which layers does that exonerate, and which does it accuse?
The logical-to-device map is unchanged, the device-to-socket map reversed. Which of the two does an export contain?
What do the seven post-restore checks have in common, and which could have answered differently in the broken topology?

Root cause

Interface names are hardware-derived, and the hardware changed

An export does not store ports. It stores names, and igb1 is not an identifier the appliance chose: it is assembled from the driver that claimed the device and the order the bus was probed in. Restore that file onto a different board and every name still resolves, because the replacement also has four igb ports. Nothing errors. What the names denote has moved.

The office switch now lands on the interface holding 10.20.40.1/24 and the voice rules, and the other segments rotate to match. The firewall is behaving correctly: it declines address resolution for 10.20.10.1 on igb3 because it does not hold that address there, and it offers voice-range leases to the office VLAN because that is the scope those broadcasts reach.

The alternative this rules out. The first theory was an incomplete restore — a truncated file, a partial import, a rule set that failed to load. The normalised diff kills it: rules, aliases, NAT and assignments equal the source, and the only deltas are identity fields the appliance regenerates. A partial import would put the missing rules in that diff, and would show as refusals rather than silence.

The control that should have caught it was a self-test

The post-restore checklist has seven items: interfaces up, addresses correct, gateway online, services running, DNS resolving, ping to an external host, GUI reachable. Every one is a question the device asks about itself, and a device sits on every one of its own networks by definition. 10.20.10.1 exists whether it is on the socket facing the office switch or the one facing the voice switch, so “the LAN interface holds the right address” is true in both worlds. The checklist could not fail: it requires no second machine, and a firewall exists to be a path between machines.

Resolution

Change nothing in the rule set. The rules are correct and were restored correctly; editing them to make traffic flow encodes the wrong mapping for good.

Fix the assignment. Establish the device-to-socket map empirically, then re-point each logical interface at the device carrying its network, through Interfaces then Assignments, or option 1 of the console menu when the web interface is itself on the wrong wire. Reassignment rewrites only the logical-to-device pairing; rules, aliases and NAT follow the logical name. Re-cabling to match the enumeration also restores service and is the wrong repair: it leaves the plant disagreeing with the rack diagram, so the next person in the rack recreates the incident.

Then clear the leases the wrong scope handed out: a workstation keeps its 10.20.40.0/24 address until it expires, so the segment recovers when the clients have renewed, not when the assignment is fixed.

Verification

Where the prober stands is the whole of the method; nothing measured from the firewall counts.

Put a host on each protected segment and resolve the gateway at layer 2 first: the neighbour entry must complete, which proves the address is on that wire rather than merely somewhere on the device. Then exercise one path the rules permit and one they refuse, and confirm the refusal appears in the log against the interface you expect. A permitted flow proves the wire; a logged block proves the rule set is attached to it. Finally take a DHCP lease on each segment and check its range — one command from one client, and it would have caught this incident by itself.

Prevention

Treat the interface map as part of the recovery artefact. The export captures logical names; what they denote belongs to the chassis, and the chassis is what changes in a hardware recovery. Keep logical interface, device name, MAC address, panel socket and switch port together, and expect the device-name column to be wrong on any replacement.

Ban self-tests from post-restore validation of network devices. Every check the device runs against itself passes in both topologies. Require a probe host per protected segment and make its result the sign-off — the discipline this course applies to backups, where a performed restore is the evidence and a green job is not.

Rehearse onto the spare: the chassis on the shelf is the one the recovery will use. And alert on segment reachability rather than device health — polling the firewall confirms it is running; traversing it confirms it is doing its job.