Observability of the firewall architecture — the discipline of seeing the whole stack, not just the device

A firewall with monitoring is not an observable system. The firewall is one component of a network that includes switches, routers, upstream ISPs, applications, and clients. An incident that looks like “the firewall is dropping traffic” might originate in the switch (a misconfigured VLAN), the upstream ISP (a routing change), the application (a server-side bug), or the client (a misconfigured DNS resolver). The operator who can see only the firewall will guess; the operator who can see the whole stack will know.

This lesson covers the observability model — metrics, logs, and traces — the stack that surrounds an OPNsense deployment, and the discipline of correlating events across systems.

The three signals: metrics, logs, traces

Modern observability distinguishes three kinds of signal:

• Metrics — numerical values sampled at regular intervals. CPU, memory, packets per second, state table occupancy. Cheap to store, easy to graph, ideal for alerting on thresholds.
• Logs — event records with timestamps and structured fields. Firewall log entries, system log entries, application log lines. Expensive to store at scale, ideal for forensic investigation of specific events.
• Traces — request flows across multiple services, with timing for each segment. Useful for application-layer debugging; less relevant for network-layer investigation.

For a firewall deployment, metrics and logs are the dominant signals. Traces are useful when the operator needs to debug a specific user request that traverses many services; for a “is the firewall dropping traffic” question, metrics and logs are sufficient.

The observability stack around an OPNsense deployment

A production deployment typically has:

The SIEM (Security Information and Event Management) is the central aggregator: it receives logs from every component, normalises them, and provides search and correlation. Common choices are Graylog, ELK (Elasticsearch + Logstash + Kibana), Splunk, Loki, and Datadog.

The metrics stack — Prometheus + Grafana — is separate from the SIEM. Metrics are time-series; logs are events. The two are correlated by timestamp and by labels (device, site, request_id).

End-to-end correlation

The hardest part of observability is correlating events across systems. A user complaint “I cannot reach the application” becomes:

1. User → firewall log: connection to 203.0.113.50:443 denied at 12:34:56.789.
2. Firewall → switch: ARP for 203.0.113.50 succeeds; packets forwarded.
3. Switch → upstream: packets routed to ISP.
4. ISP → application: SYN received by server.
5. Server → application: HTTP 200 returned.

The operator who can see all five steps knows the firewall denied a legitimate flow (a rule mistake) versus an asymmetric path versus a server-side rejection. The operator who sees only step 1 has to guess at steps 2 to 5.

The correlation mechanisms:

• Timestamps — every event has a timestamp; aligning them within a second is the first step.
• IPs and ports — source and destination IPs and ports appear in firewall logs, switch logs, application logs, and packet captures. The tuple is the join key.
• Flow IDs — NetFlow, sFlow, or IPFIX records carry flow identifiers that allow correlation across routers.
• Request IDs — application logs often include request IDs that can be propagated through proxies.

The cost of end-to-end observability

End-to-end observability is not free. The costs:

• Storage — logs at scale are terabytes per week. Retention is bounded by cost.
• CPU and network — every component sending logs and metrics uses bandwidth and CPU. A saturated firewall has less headroom for sending telemetry.
• Operational complexity — more components means more failure modes. The Prometheus server can fail; the SIEM can fail; the syslog forwarder can drop events.

The discipline is to instrument what is actionable. Metrics that never trigger alerts and logs that never get searched are wasted telemetry. The operator should review the dashboards and the SIEM queries periodically and prune what is unused.

$ curl -s 'http://prometheus:9090/api/v1/query?query=opnsense_pf_states_total' | python3 -c 'import json,sys; d=json.load(sys.stdin); print(d["data"]["result"])'

[{'metric': {'__name__': 'opnsense_pf_states_total', 'device': 'opnsense-fw-01'}, 'value': [1723411200.123, '3127']}]

The observability of a CARP pair

A HA deployment of two OPNsense firewalls requires double the observability — and correlation across both nodes. The two firewalls have separate metric streams; the operator must be able to:

• Compare CPU, memory, state table, and interface counters across both nodes.
• See which node is MASTER and which is BACKUP per VHID.
• See pfsync state synchronisation rates.
• See failover events (CARP state transitions).

Grafana dashboards with a $node variable that filters by node make this trivial. The discipline is to confirm during every Grafana refresh that both nodes are reporting — a node that stops reporting metrics may have failed silently.

The discipline of “what to look at first”

When an incident lands, the operator’s first move is to the dashboard that correlates firewall-health with network-health and with upstream path metrics. The order of checks:

1. Is the firewall healthy? CPU, memory, state table, CARP.
2. Are the interfaces up? Drops, errors, bps per interface.
3. Is the upstream reachable? Ping the ISP gateway from the firewall.
4. Are the switches healthy? SNMP poll of uplink switches.
5. Is the application healthy? HTTP probe from the firewall.

Each check has a clear yes/no answer. The operator who can answer all five within two minutes knows exactly where the problem is (or, just as importantly, knows exactly where the problem is not).

Summary

• Modern observability distinguishes metrics (numerical, cheap, alerting), logs (events, expensive, forensic), and traces (request flows, application-layer).
• An OPNsense deployment is one component in a stack that includes switches, upstream routers, servers, applications, and clients.
• End-to-end correlation uses timestamps, IPs, ports, flow IDs, and request IDs as join keys.
• Clock skew breaks correlation — NTP must be enforced across every component.
• The discipline of “what to look at first” — firewall-health first, then network-health, then upstream, then application.