System and service logs — what lives outside the filter log · OPNsense

The filter log answers “what traffic did the firewall block”. The other logs answer “what did the firewall itself do”. Authentication failures, DHCP lease errors, DNS resolution failures, VPN tunnel lifecycle events, IDS alerts, and proxy access — these are the logs that diagnose why a service is not working. Reading them is the difference between “the firewall is dropping traffic” and “the firewall dropped traffic because the resolver stopped validating and the client retried against a different address”.

Reading a line on disk

Every local destination is written by syslog-ng in RFC 5424 form, so every line in every directory has the same shape before the message begins:

<PRI>1 TIMESTAMP HOSTNAME PROGRAM PID MSGID [STRUCTURED-DATA] message

MSGID is normally -, and the structured data is normally just [meta sequenceId="N"]. Everything after that closing bracket is what the program actually said. The header is fixed-width in fields but not in characters, and the structured-data block is two whitespace-separated fields rather than one, so it takes awk '{ $1=$2=$3=$4=$5=$6=$7=$8=""; print }' to strip all eight header fields when the raw form gets in the way — but for grepping, the header is useful, because it carries the program name that put the line in this directory in the first place.

System log — `/var/log/system/`

The system log is the fall-through. It receives anything at notice severity or above that no application filter claimed:

Kernel messages: device drivers, link state, hardware errors.
Service lifecycle for daemons without a dedicated destination.
CARP and interface state transitions.
Whatever a newly installed component emits before it has a filter of its own.

Read-only / Safesystem log sample— A link on igb0 dropped and came back six seconds later, which triggered the hotplug handler to reconfigure WAN. Forty-two seconds after that hotplug event, VHID 1 on igb1 moved from MASTER to BACKUP for the reason "more frequent advertisement received" — the peer was advertising with the lower effective skew and won the election — and thirty-six seconds later it took the role back with "master timed out", meaning the peer went silent for three advertisement intervals. Read the reason string, not just the arrow: neither line is the demotion counter, which is the separate node-wide net.inet.carp.demotion knob. Both transitions are written by the same kernel logging path at the same severity and are gated by net.inet.carp.log, so they cannot be told apart by syslog priority — match on the carp: prefix and the vhid@interface tag instead. Resist collapsing all four lines into one story: the election starts forty-two seconds after the link had already recovered, far too late for the interface-down demotion to be the cause, so what the file holds here is a link flap and a separate CARP election that happen to be adjacent.

$ tail -5 /var/log/system/latest.log

<134>1 2026-08-14T16:30:15+00:00 opnsense.example.com kernel - - [meta sequenceId="8112"] igb0: link state changed to DOWN
<134>1 2026-08-14T16:30:21+00:00 opnsense.example.com kernel - - [meta sequenceId="8113"] igb0: link state changed to UP
<134>1 2026-08-14T16:30:22+00:00 opnsense.example.com opnsense 41255 - [meta sequenceId="8114"] /usr/local/etc/rc.linkup: Hotplug event detected for WAN(igb0)
<134>1 2026-08-14T16:31:04+00:00 opnsense.example.com kernel - - [meta sequenceId="8115"] carp: 1@igb1: MASTER -> BACKUP (more frequent advertisement received)
<134>1 2026-08-14T16:31:40+00:00 opnsense.example.com kernel - - [meta sequenceId="8116"] carp: 1@igb1: BACKUP -> MASTER (master timed out)

Illustrative output

Audit log — `/var/log/audit/`

The audit destination claims both the audit program and the whole auth facility, which puts GUI logins and SSH logins in the same place:

Successful and failed GUI authentication.
SSH session establishment and failure.
Privilege escalation and session lifecycle.
Configuration changes attributed to a user.

Read-only / Safeaudit log, failures only— One source address, 198.51.100.10, is trying SSH as root, then SSH as a non-existent user, then the GUI as bob. That progression across two services from one address is a credential-stuffing attempt rather than a user having a bad day. The operator should confirm root SSH login is disabled, confirm the management interface is not reachable from where this address sits, and check whether bob is a real account.

$ grep -iE 'failed|invalid|error' /var/log/audit/latest.log | tail -4

<38>1 2026-08-14T16:31:30+00:00 opnsense.example.com sshd 23457 - [meta sequenceId="411"] Failed password for root from 198.51.100.10 port 33333 ssh2
<38>1 2026-08-14T16:31:33+00:00 opnsense.example.com sshd 23458 - [meta sequenceId="412"] Invalid user admin from 198.51.100.10 port 33341
<38>1 2026-08-14T16:32:30+00:00 opnsense.example.com audit 41260 - [meta sequenceId="413"] user 'bob' could not authenticate from 198.51.100.10
<38>1 2026-08-14T16:32:35+00:00 opnsense.example.com audit 41260 - [meta sequenceId="414"] user 'bob' could not authenticate from 198.51.100.10

Illustrative output

DNS — `/var/log/resolver/` and `/var/log/dnsmasq/`

Unbound writes to /var/log/resolver/; Dnsmasq, when it is the DNS service, writes to /var/log/dnsmasq/. Query and reply logging are both off by default, because both are expensive and both record who asked for what.

With query and reply logging enabled, Unbound emits one line per query and one per reply. The reply line carries the client address, the name, the type and class, the return code, the resolution time in seconds, a flag for whether the answer came from cache, and the response size.

Read-only / Saferesolver log sample— The first pair is a query and its reply: example.com resolved with NOERROR in 31 milliseconds, not from cache, 54 bytes. The third line answered NXDOMAIN in zero time from a local source, which is the signature of a blocklist entry — the resolver answered it locally rather than asking upstream. The fourth line is a DNSSEC validation failure, and it means the resolver refused to return an answer it could not verify. The operator should treat the blocklist NXDOMAIN as expected and the validation failure as a fault to chase.

$ grep -F ' info: ' /var/log/resolver/latest.log | tail -4

<149>1 2026-08-14T16:30:01+00:00 opnsense.example.com unbound 32109 - [meta sequenceId="9001"] [32109:0] info: 192.0.2.50 example.com. A IN
<149>1 2026-08-14T16:30:01+00:00 opnsense.example.com unbound 32109 - [meta sequenceId="9002"] [32109:0] info: 192.0.2.50 example.com. A IN NOERROR 0.031472 0 54
<149>1 2026-08-14T16:30:05+00:00 opnsense.example.com unbound 32109 - [meta sequenceId="9003"] [32109:0] info: 192.0.2.50 ads.example.org. A IN NXDOMAIN 0.000000 1 43
<149>1 2026-08-14T16:30:10+00:00 opnsense.example.com unbound 32109 - [meta sequenceId="9004"] [32109:0] info: validation failure <bank.example.com. A IN>

Illustrative output

A validation failure line is worth chasing rather than suppressing. The usual causes are clock skew on the firewall, since DNSSEC signatures have validity windows; an expired DS record at the parent zone; or something on the path rewriting DNS responses. drill or dig +dnssec against the zone from the firewall separates the three.

DHCP — `/var/log/kea/` and `/var/log/dnsmasq/`

Which directory holds the leases depends on which DHCP service is running. Kea writes to /var/log/kea/, Dnsmasq to /var/log/dnsmasq/. The deprecated ISC DHCP plugin, where it is still installed, writes to /var/log/dhcpd/.

Kea logs one message per decision point, each with a message identifier that can be looked up in the Kea Messages Manual.

Read-only / SafeKea lease allocation— The first client is offered 192.0.2.100 and then allocated it for an hour, which is the complete healthy path: advert followed by alloc, same transaction id. The third line is a second client being offered an address with no alloc following it in this window. That asymmetry is the signature to look for — a client that is advertised a lease but never allocated one is not completing the exchange, which points at packet loss, a second DHCP server answering, or a relay problem.

$ grep -E 'DHCP4_LEASE_(ALLOC|ADVERT)' /var/log/kea/latest.log | tail -3

<134>1 2026-08-14T16:30:01+00:00 opnsense.example.com kea-dhcp4 51002 - [meta sequenceId="221"] INFO  [kea-dhcp4.leases/51002.99584] DHCP4_LEASE_ADVERT [hwtype=1 aa:bb:cc:11:22:33], cid=[01:aa:bb:cc:11:22:33], tid=0x4e1c2a7b: lease 192.0.2.100 will be advertised
<134>1 2026-08-14T16:30:01+00:00 opnsense.example.com kea-dhcp4 51002 - [meta sequenceId="222"] INFO  [kea-dhcp4.leases/51002.99584] DHCP4_LEASE_ALLOC [hwtype=1 aa:bb:cc:11:22:33], cid=[01:aa:bb:cc:11:22:33], tid=0x4e1c2a7b: lease 192.0.2.100 has been allocated for 3600 seconds
<134>1 2026-08-14T16:30:19+00:00 opnsense.example.com kea-dhcp4 51002 - [meta sequenceId="223"] INFO  [kea-dhcp4.leases/51002.99584] DHCP4_LEASE_ADVERT [hwtype=1 aa:bb:cc:11:22:44], cid=[01:aa:bb:cc:11:22:44], tid=0x77aa31d0: lease 192.0.2.101 will be advertised

Illustrative output

VPN — `/var/log/ipsec/`, `/var/log/openvpn/`, `/var/log/wireguard/`

Each VPN implementation has its own directory, matched on its own daemon name: charon for IPsec, openvpn, and wireguard plus kernel messages naming a wg interface.

Read-only / SafeIPsec log sample— An IKE_SA and its CHILD_SA come up at 16:30. At 17:45 dead peer detection stops getting answers and starts retrying. At 17:46 a new IKE_SA is established with a higher instance number, so the tunnel recovered on its own. That is the shape of a brief peer outage. Had it not recovered, the next lines would name the failure instead — NO_PROPOSAL_CHOSEN for a mismatched proposal, AUTHENTICATION_FAILED for a credential problem, or TS_UNACCEPTABLE for mismatched traffic selectors.

$ grep -E 'established|rekey|not responding' /var/log/ipsec/latest.log | tail -4

<134>1 2026-08-14T16:30:01+00:00 opnsense.example.com charon 45012 - [meta sequenceId="612"] 05[IKE] <con1|1> IKE_SA con1[1] established between 192.0.2.1[opnsense.example.com]...198.51.100.1[peer.example.com]
<134>1 2026-08-14T16:30:01+00:00 opnsense.example.com charon 45012 - [meta sequenceId="613"] 05[IKE] <con1|1> CHILD_SA con1{1} established with SPIs c1234abc_i 9f8e7d6c_o
<134>1 2026-08-14T17:45:30+00:00 opnsense.example.com charon 45012 - [meta sequenceId="708"] 12[IKE] <con1|1> peer not responding, trying again (2/5)
<134>1 2026-08-14T17:46:04+00:00 opnsense.example.com charon 45012 - [meta sequenceId="712"] 13[IKE] <con1|4> IKE_SA con1[4] established between 192.0.2.1[opnsense.example.com]...198.51.100.1[peer.example.com]

Illustrative output

IDS — `/var/log/suricata/`

Suricata is the one component in the default install that keeps two logs in two different mechanisms:

/var/log/suricata/suricata_YYYYMMDD.log — the daemon’s own messages, routed through syslog-ng like everything else: startup, rule load, configuration errors.
/var/log/suricata/eve.json — the EVE output, written directly by Suricata and rotated by newsyslog rather than by the daily-file scheme. One JSON object per line, one line per event.

The EVE file is the one worth querying. Each object carries timestamp, event_type, src_ip, dest_ip, and, for alerts, an alert object with the signature and severity.

Read-only / SafeSuricata alerts, projected with jq— Because EVE is one JSON object per line, jq projects the fields that matter without any parsing by position. Both alerts involve the same internal host and the same external address. The action field differs between them: one was blocked and one was allowed, which happens when only some signatures in the ruleset are set to drop. The operator should read the repeated internal source as the thing to investigate, and the mixed actions as a policy question about which categories should drop rather than alert.

$ jq -c 'select(.event_type=="alert") | {ts:.timestamp, src:.src_ip, dst:.dest_ip, sig:.alert.signature, sev:.alert.severity, act:.alert.action}' /var/log/suricata/eve.json | tail -2

{"ts":"2026-08-14T16:30:01.123456+0000","src":"192.0.2.50","dst":"198.51.100.10","sig":"ET MALWARE Observed DNS Query to Suspicious Domain","sev":1,"act":"blocked"}
{"ts":"2026-08-14T16:30:04.881210+0000","src":"192.0.2.50","dst":"198.51.100.10","sig":"ET MALWARE Observed DNS Query to Suspicious Domain","sev":2,"act":"allowed"}

Illustrative output

Proxy — `/var/log/squid/`

The Squid plugin can write its access log two ways, selected by its log target setting. Written directly, it lands in /var/log/squid/access.log and is rotated by newsyslog. Sent through syslog, it is matched on the squid- program prefix and lands in the daily-file scheme under /var/log/squid/access/ instead. Check which one is configured before writing a query against a path.

The record format is the Squid access log format: timestamp, duration, client address, result code and status, bytes, method, URL, user, hierarchy, and content type.

Building an incident timeline

The diagnostic skill is correlation across directories. When a service misbehaves:

Start with the user-visible symptom. “Name resolution stopped working at about half past four.”
Go to the directory that owns that symptom. /var/log/resolver/ for name resolution.
Widen to the same window in the neighbouring directories. /var/log/system/ for link and service events, /var/log/gateways/ for upstream reachability, /var/log/filter/ for whether the queries left at all.
Order everything by timestamp. The cause is usually the earliest anomalous line, not the loudest one.

Because every directory uses the same daily naming, step three is one glob. grep -h '16:3' /var/log/{resolver,system,gateways}/latest.log | sort -k2 puts three services on one timeline.

Why correlation requires consistent time

Every log entry is timestamped by the device that produced it. If the clocks differ — the firewall five minutes ahead of the SIEM — the correlation breaks. An event at 16:25 on the firewall appears at 16:20 in the SIEM, and the operator searching around 16:20 never finds it.

The on-disk timestamps carry an offset from UTC, so the information is technically recoverable, but nothing in the toolchain recovers it for you. sort -k2 on lines from two hosts in two zones sorts them wrongly, and a grep for an hour matches the wrong hour.

The discipline: every device in the estate runs NTP, every device is set to UTC, and clock drift is monitored rather than assumed. The cost of drift is invisible almost all the time. It becomes visible at three in the morning, when the operator cannot line the firewall up against anything else and cannot find the cause.

Summary

Every service logs into its own directory under /var/log/, named after the program that emits the messages.
Lines are RFC 5424: a syslog header carrying program and PID, then the message the program actually wrote.
System log: the fall-through for kernel, link, CARP, and unclaimed messages.
Audit log: GUI and SSH authentication, in one place, because it claims the auth facility as well as the audit program.
DNS, DHCP, and VPN logs live under the daemon that produces them, not under a generic service name.
Suricata keeps EVE JSON outside the daily-file scheme, rotated by newsyslog; query it with jq.
Correlation across directories requires consistent time. NTP, UTC, monitored drift.

Reading a line on disk

System log — /var/log/system/

Audit log — /var/log/audit/

DNS — /var/log/resolver/ and /var/log/dnsmasq/

DHCP — /var/log/kea/ and /var/log/dnsmasq/

VPN — /var/log/ipsec/, /var/log/openvpn/, /var/log/wireguard/

IDS — /var/log/suricata/

Proxy — /var/log/squid/

Building an incident timeline

Summary

System log — `/var/log/system/`

Audit log — `/var/log/audit/`

DNS — `/var/log/resolver/` and `/var/log/dnsmasq/`

DHCP — `/var/log/kea/` and `/var/log/dnsmasq/`

VPN — `/var/log/ipsec/`, `/var/log/openvpn/`, `/var/log/wireguard/`

IDS — `/var/log/suricata/`

Proxy — `/var/log/squid/`