Monitoring the firewall itself — what to instrument before anything else

When an incident lands at 03:00 — “the firewall is dropping traffic”, “the VPN is slow”, “users cannot reach the Internet” — the operator’s first decision is whether the firewall itself is the problem. That decision is made on evidence, not on intuition. The evidence is metrics: CPU, memory, state table occupancy, disk, interface counters, temperature. The operator who has these metrics graphed and the operator who has to SSH in and run top are not operating the same network.

This lesson covers the metrics that describe the firewall as a system, the built-in OPNsense graphs that visualise them, and the discipline of separating firewall-health from network-health when an incident is being diagnosed.

The two kinds of metrics

The metrics an OPNsense operator collects fall into two categories:

• Firewall-health metrics describe the firewall as a system: CPU utilisation, memory pressure, state table occupancy, swap usage, disk space, temperature, daemon health. These answer the question “is the firewall working as a computer?”.
• Network-health metrics describe the traffic flowing through the firewall: packets per second, bits per second, drops per interface, latency, TCP retransmits, NAT translations. These answer the question “is the network delivering the service?”.

The two are independent. A firewall with perfect CPU, memory, and state-table occupancy can still be dropping traffic because of a misconfigured rule. A firewall at 100% CPU can still be passing every legitimate flow because the bottleneck is on a path that does not go through the firewall. The operator who conflates the two — “CPU is high, so users must be having problems” — will waste time investigating the wrong subsystem.

The metrics that matter

The minimum set of firewall-health metrics the operator should collect:

Of these, CPU interrupt share and state table occupancy are the two most often missed. CPU utilisation alone can look healthy while the packet path is bottlenecked on interrupt handling. State table occupancy can be 80% with no alarms, then cross 95% and start dropping flows with no warning.

Reading the built-in OPNsense graphs

OPNsense ships with built-in graphs (Lobby → Traffic, System → Reporting → Health) that visualise CPU, memory, state table, and interface counters. These are not a replacement for an external monitoring stack — the data lives only on the firewall and is lost on reboot — but they are the operator’s first stop when an incident is in progress.

$ vmstat 1 3

procs  memory  page  faults  cpu
r b w  avm  flt  re  pi  po  fr  sr  cy  in  sy  cs  us  sy  id
0 0 0  1.2G   0   0   0   0   0   0   0  12  481  612  2   1  97
1 0 0  1.2G   0   0   0   0   0   0   0  18  512  705  3   2  95
0 0 0  1.2G   0   0   0   0   0   0   0  15  498  648  2   1  97

The OPNsense graphs visualise the same data with one-minute resolution and 24 hours of retention. They are useful for “what does the firewall usually look like” — the operator who has not looked at the graphs in calm conditions has no baseline to compare against during an incident.

Firewall-health versus network-health in practice

The classic 03:00 incident: “users cannot reach the Internet”. The operator’s first three checks are firewall-health:

1. Can I reach the firewall? (Is it alive at all?)
2. What is the CPU and state table? (Is the firewall itself the bottleneck?)
3. What does the system log say? (Is a daemon crashing or a service stopped?)

Only if all three are clean does the operator move to network-health: are the WAN and LAN interfaces up? Is the upstream reachable? Is there a recent rule change? The operator who skips step 1 — who goes straight to “check the rules” while the firewall is at 99% CPU — finds nothing in the rules and wastes an hour.

What to instrument first

If the operator has no monitoring in place, the priority order is:

1. State table occupancy — the canary metric. Graph it first.
2. CPU and interrupt share — the firewall will tell you when it is CPU-bound.
3. Memory — a slow killer; OOM kills crash daemons.
4. Disk — full disk breaks logging.
5. Interface counters — drops, errors, broadcasts.
6. Temperature — relevant on appliances; less so on VMs.

Items 1 to 4 are firewall-health; 5 is network-health; 6 is environment. The operator who has 1 to 4 covered has the basis for triage; the rest can be added incrementally.

$ pfctl -s info

Status: Enabled for 142 days 04:17:23
Debug: err
State Table                          Total             Rate
current entries                        3127
searches                           18452394          1500.3/s
inserts                              189324             15.4/s
removals                             186197             15.2/s
Counters:
match                              189324             15.4/s
bad-offset                             0                0.0/s
fragment                              0                0.0/s
short                                0                0.0/s
normalize                           182                 0.0/s
memory                              35.3M
bad-timestamp                          0                0.0/s
congestion                             0                0.0/s
ip-options                            0                0.0/s
proto-cksum                          0                0.0/s
state-mismatch                      12                 0.0/s
state-insert                           0                0.0/s
state-limit                        0                0.0/s
src-limit                            0                0.0/s
synproxy                              0                0.0/s

Summary

• Firewall-health metrics (CPU, memory, state table, disk) describe the firewall as a system; network-health metrics (packets, drops, latency) describe the traffic.
• Triage order: firewall-health first, network-health second.
• The minimum metric set: state table occupancy, CPU and interrupt share, memory, disk, daemon health, temperature.
• The OPNsense built-in graphs are a starting point but not a replacement for an external monitoring stack.
• Trends matter more than single snapshots — a state table growing from 30% to 80% is a warning even when the alert has not fired.