Availability and redundancy — why a second firewall, what HA is actually solving · OPNsense

A second firewall on the network does not, by itself, produce high availability. Two firewalls that fight for the same address, that fail over in minutes instead of seconds, or that lose state on every transition produce a more expensive outage — the operator now has two devices to debug instead of one. High availability is a deliberate design with measurable targets, a defined failure model, and a topology that matches the budget. This lesson covers the vocabulary — the nines, MTBF, MTTR, recovery point versus recovery time — and the single points of failure that justify a second firewall.

The vocabulary of availability

Availability is the fraction of time a service is usable. The classic “nines”:

Target	Downtime per year	Notes
99%	3.65 days	One long weekend a year
99.9%	8.77 hours	A working day per year
99.99%	52.6 minutes	A long lunch per year
99.999%	5.26 minutes	Carrier-grade — fibre and redundant power

The numbers hide their true cost. The jump from 99.9% to 99.99% (the jump from “a working day” to “a long lunch”) typically requires redundant firewalls, redundant switches, redundant paths, redundant power, and a tested failover process. The jump from 99.99% to 99.999% typically requires geographic diversity — two sites, two ISPs, two fibre paths — and is the point at which the engineering budget stops resembling the rest of the IT budget.

The two numbers behind availability:

MTBF (Mean Time Between Failures). The average time a device runs before it fails. A commodity firewall with consumer-grade components might have an MTBF of 30,000 hours (about 3.4 years); a carrier-grade device with redundant power and hot-swappable fans might exceed 100,000 hours.
MTTR (Mean Time To Repair). The average time to restore service after a failure. For a single firewall with no HA, MTTR is the time to notice the failure, the time to drive to the data centre (or VPN in), and the time to swap the device — easily 30 to 240 minutes. With HA, MTTR is the failover time — typically 1 to 5 seconds for the network path, plus the time to repair or replace the failed node.

The relationship: Availability = MTBF / (MTBF + MTTR). To increase availability, you either increase MTBF (buy better hardware) or decrease MTTR (add redundancy). HA is the MTTR strategy.

Recovery point and recovery time

Two related numbers that HA affects:

RPO (Recovery Point Objective). The maximum data loss the operator accepts. For stateful firewalls, this is the firewall state table — connections that existed at the moment of failure. With no state synchronisation, the RPO is the entire state table: every active connection drops when the backup takes over.
RTO (Recovery Time Objective). The maximum downtime the operator accepts. This is the time from failure to full operation on the backup. For HA firewalls, the RTO is the failover time — the time for the backup to take over the virtual IPs.

The two are independent. A deployment with state synchronisation has a low RPO (only the last few seconds of state are lost) but the RTO is the same as without synchronisation (the failover time is the same). A deployment with no state synchronisation has a high RPO (every active connection drops) but the RTO can still be 1 to 5 seconds.

The single points of failure

A single-firewall deployment has several single points of failure (SPOFs). Each one is a candidate for the HA investment:

Component	Failure impact	HA mitigation
Firewall hardware	Total outage	Second firewall
Firewall power supply	Total outage	Dual-PSU hardware or two firewalls
WAN link	Internet outage	Multi-WAN (covered in Part XIX)
Switch the firewall connects to	Network outage on that segment	Redundant switches, LACP
ISP hand-off	Internet outage	Second ISP, geographic diversity

The HA investment is the second firewall. The other SPOFs need their own investments — second switch, second ISP, dual PSU. A deployment that adds a second firewall but leaves the upstream switch as a single point of failure has spent the HA budget on a SPOF that is not the firewall.

What HA is and what it is not

HA solves:

Firewall device failure. The backup takes over when the primary fails.
Firewall software crash. The backup takes over when the primary’s daemons stop responding.
Planned maintenance. The operator can fail the primary to the backup and patch it without downtime.

HA does not solve:

WAN link failure (this is multi-WAN).
Switch or cabling failure (this is L2 redundancy).
ISP failure (this is a second ISP or geographic diversity).
Configuration errors that affect both nodes (this is change management and staged rollouts).
Distributed denial of service (this is DDoS mitigation upstream).
Power failure to the rack (this is dual PSU, dual feeds, UPS, generator).

The trap is to think of HA as a generic availability solution. HA is a specific solution to a specific failure model: a single firewall device that can fail, and a backup that can take over. The operator who treats HA as “everything is now redundant” misses the other SPOFs.

Why the nines are an asymptotic curve, not a ladder

The nines suggest a clean ladder — 99%, 99.9%, 99.99%, 99.999%, each step 10 times more available than the last. In practice, each step is a different engineering problem, with different failure models and different costs.

The first nine (99% to 99.9%) is usually solved by operational discipline — monitoring, change windows, fast response. The second nine (99.9% to 99.99%) is usually solved by HA — second device, automatic failover, tested failover. The third nine (99.99% to 99.999%) is usually solved by geographic diversity — two sites, two ISPs, two fibre paths, and a DNS or BGP mechanism to route around either site.

The trap is to plan the engineering budget as if the ladder were uniform. The jump from 99.9% to 99.99% (HA) costs roughly twice as much as the jump to 99.9% (discipline). The jump from 99.99% to 99.999% (geographic diversity) costs roughly an order of magnitude more than the jump to 99.99%. The operator who skips straight to 99.999% without the discipline of 99.9% has bought expensive hardware that fails on the same operator errors that the discipline would have caught.

Summary

Availability = MTBF / (MTBF + MTTR). HA reduces MTTR, not MTBF.
The nines are not uniform — each step is a different engineering problem with a different cost.
RPO is the data loss budget; RTO is the downtime budget. HA affects both, differently.
A single-firewall deployment has multiple SPOFs. The second firewall addresses one of them; the operator still needs to address the switch, the WAN, the power, and the ISP.
HA solves device failure. It does not solve WAN, switch, ISP, configuration, DDoS, or power failures.

Read-only / Safeavailability evidence— uptime shows the firewall has been running 142 days — the MTBF is at least 142 days, and the MTTR is undefined because no failure has occurred. systat -v shows the open-file count per user — a small number consistent with normal operation. The evidence is not "the firewall is up" but "the firewall has been up for 142 days, and the daemon footprint is what we expect". A firewall whose uptime resets every weekend has been failing every weekend — even if the operator does not see the failures.

$ uptime; systat -v 1 2

12:34:56 up 142 days,  4:17, 2 users, load averages: 0.12, 0.08, 0.05
                  open files      
uid     count 
root    18
nobody  4
_avahi  1
_dhcp   2
_dns    1

Illustrative output