API monitoring and webhooks — driving observability from firewall events · OPNsense

The same API that drives rule automation is also a read interface. A monitor that needs interface counters, PF state-table size, gateway state, or a list of running services can read those from the API with a least-privilege key — no scraping the GUI, no SSH-into-the-firewall-and-parse-output. The trade-off is rate: the API serves one-shot reads well but is too slow for high-frequency metrics. For those, Prometheus or syslog is the right tool. For event-driven notifications — webhook-style triggers when a firewall event happens — the API works in both directions: read for state, write for action.

This lesson covers the read endpoints a monitor should consume, the pattern for issuing a least-privilege key to a monitor, and the external workflow integrations that hook firewall events into a ticket system or chat. The final question — when is the API right, when is Prometheus right, when is syslog right — frames the trade-offs honestly.

Read endpoints for a monitor

A monitor that wants the current state of the firewall reads from these endpoints on a 30-60 second cadence:

Endpoint	What it returns
`GET /api/core/system/status`	CPU, memory, disk, load, uptime
`GET /api/interfaces/overview/interfacesInfo`	All interfaces, addresses, status
`GET /api/diagnostics/interface/getInterfaceStatistics`	Per-interface counters (bytes, packets, errors)
`GET /api/diagnostics/firewall/pf_states`	PF state table (counts, sizes)
`GET /api/diagnostics/firewall/pf_rules`	Loaded PF ruleset
`GET /api/diagnostics/firewall/log`	Recent firewall log lines
`GET /api/diagnostics/interface/getSocketStatistics`	Open sockets, listeners
`GET /api/diagnostics/interface/getProtocolStatistics`	Per-protocol counters
`GET /api/routes/gateway/status`	Gateway state (up, down, packet loss, RTT)
`GET /api/wireguard/service/show`	WireGuard peer states
`GET /api/core/service/search`	Which services are running

The set is read-only. Each call returns JSON; a monitor builds its metric view from those JSONs.

Read-only / Safeenumeration loop— A looped check of every read endpoint a monitor needs, authenticated once with the key/secret pair as Basic credentials. Each call returns a JSON shape; the jq filter probes the top-level keys so the monitor sees what it will receive. Two shapes recur: endpoints built on the framework search helper return the paginated envelope of current, rowCount, rows and total, while the rest return a bespoke object. A monitor should probe the shape in a discovery phase rather than hard-coding it — a key that disappears after an upgrade is a quiet production outage.

$ for ep in core/system/status interfaces/overview/interfacesInfo diagnostics/interface/getInterfaceStatistics routes/gateway/status core/service/search; do echo "--- /api/$ep ---"; curl -sk -u "$KEY:$SECRET" https://localhost/api/$ep | jq 'if type=="array" then length else keys end'; done

--- /api/core/system/status ---
[ "status", "subsystems" ]
--- /api/interfaces/overview/interfacesInfo ---
[ "current", "rowCount", "rows", "total" ]
--- /api/diagnostics/interface/getInterfaceStatistics ---
[ "statistics" ]
--- /api/routes/gateway/status ---
[ "items", "status" ]
--- /api/core/service/search ---
[ "current", "rowCount", "rows", "total" ]

Illustrative output

The least-privilege monitor key

A monitor does not need to modify the firewall, so its API key should have the minimum possible privileges. The privilege that grants “read every state and diagnostic endpoint” is page-status-services and page-diagnostics-logs — a service account with these two privileges and no others. The monitor can:

Read interface counters.
Read PF state, rules, log.
Read system status.
Read gateway state.

The monitor cannot:

Add or modify rules, aliases, NAT.
Modify users, API keys, certificates.
Reboot or trigger upgrades.

The blast radius of a leaked monitor key is “the monitor can be impersonated”, which is “the monitor says things that are not true”. The attack does not grant the adversary any write capability.

Webhook-style integrations: external workflow triggers

The API can also drive external workflows. A firewall event — gateway down, certificate expiring, the box running out of disk — needs to reach a human (or a ticketing system or a chat channel) quickly. Two patterns:

The monitor is upstream and pushes. A monitor that scrapes the API on a 30-second cadence sees gateway state, sees the transition from “up” to “down”, and calls a webhook URL on a chat platform or ticketing system with the event payload. The firewall does not directly integrate with chat — the monitor does. The firewall is an API source; the chat is a webhook sink.
OPNsense sends an alert directly. Under Services → Monit → Settings or similar plugins, OPNsense can be configured to call an external URL when a check fails (a service dies, a CPU threshold crosses). The plugin sends an HTTP POST. The webhook receiver is responsible for parsing the payload, deduplicating, and routing to a human.

Pattern 1 is more common because it gives the integration code full control over deduplication, alerting rules, and routing. Pattern 2 is appropriate for OPNsense-native checks (an internal service died) where building a separate monitor is overkill.

Read-only / Safefirewall event to chat— The monitor detects WAN gateway down and pushes to a chat platform. This is the chat platform's credential scheme, not the firewall's — most chat APIs take a bearer token, whereas OPNsense takes a key/secret pair as Basic auth, so the monitor holds two credentials of different shapes and must not confuse them. The chat token has chat-write scope only; the OPNsense pair is read-only on the firewall API. Each is stored separately and rotated on its own schedule.

$ curl -sk -X POST -H "Authorization: Bearer $CHAT_TOKEN" -H "Content-Type: application/json" -d '{"channel":"#firewall-alerts","text":"*Gateway WAN_DHCP* is *down* on opnsense-fw-01 — packet loss 100% over 3 checks"}' https://chat.example.com/api/v1/chats.postMessage

{
"ok": true,
"channel": "C01234567",
"ts": 1692012985.000200
}

Illustrative output

Deduplication and rate

A monitor that pushes every API change to chat creates noise. Three disciplines:

Deduplicate on transition. Push only when the state changes (gateway up → down, or down → up), not on every check. Persistent noise is worse than missed alerts.
Rate-limit the webhook. A flaky gateway that flaps every minute does not need 60 messages an hour. Use the monitor’s own rate limit per key.
Acknowledge on recovery. A monitor that pushes “gateway is down” but not “gateway is back” leaves the operator guessing whether the alert is still active.

The chat platform’s read receipts (thread thread continuation, ts timestamp) make follow-up readable. The firewall API does not directly support threads, so the monitor must store the thread identifier itself.

When the API is the right tool

The API is the right choice for:

Stateful pull. A monitor that wants the current state of the firewall on a moderate cadence (10-60 seconds) and can tolerate per-scrape costs reads from the API.
One-off diagnostics. An on-call engineer reading the firewall state at 02:00 uses the API (or the GUI, which uses the API internally).
External workflow triggers. A webhook receiver that talks to a ticketing or chat system reads state from the API and pushes events to those systems.

The API is not the right choice for:

High-frequency metrics. Prometheus’s text-format exporter runs on port 9100 with millisecond resolution; the API serves one request at a time per worker and has more overhead.
Event streams. A monitor that wants every firewall log line as it happens should consume the syslog stream, not poll for log slices. The API’s log endpoint returns recent log lines on demand; it does not push new lines.

The API's role vs Prometheus's role

The OPNsense REST API and the Prometheus exporter serve different observability roles. The API is a “current state” interface — read when you want, get the snapshot. Prometheus is a “time-series collection” interface — scrape on a tight cadence, build a time series, alert on rate-of-change.

The API has per-call overhead (TLS, JSON parsing, request processing) and finite worker capacity; pushing it through a 1-second scrape interval would saturate the firewall. The Prometheus exporter is a lightweight text-format endpoint designed for that cadence. The right division of labour: API for ad-hoc reads and external integrations; Prometheus for metric collection and alerting; syslog for event streams.

The exporter itself reads from the same sources the API uses (PF counters, sysctl values, system calls). The two are not in conflict; they complement each other.

A reasonable monitoring topology

A typical production setup:

Prometheus scrapes os-opnsense-exporter every 30 seconds for metrics — interface counters, PF state size, system state. Records and alerts on thresholds.
A push-style monitor reads /api/diagnostics/firewall/log and other API endpoints on demand (when an alert fires) to gather context.
Webhook receivers (chat, ticketing) are fired by the monitor on state transitions, deduplicated and rate-limited by the monitor.
Syslog streams to a central observability stack for full log retention.

The single rule: choose the right tool for the data shape. Metrics → Prometheus. Events → syslog. State and ad-hoc reads → API. Webhooks → API + integration script.

Summary

Read endpoints cover system status, interfaces, PF state, gateway state, and logs. A monitor can build a complete view from the API.
The monitor’s API key should be a separate service account with read-only privileges (page-status-services, page-diagnostics-logs).
Webhook-style integrations: the monitor (or OPNsense’s own notifications) calls chat/ticketing webhooks on state transitions, deduplicated and rate-limited.
API for state, Prometheus for metrics, syslog for events. Each tool has the right shape for its data.
Do not poll the firewall log via the API in production; configure remote syslog and consume the log there.