Skip to main content
RunBook Academy

← All break/fix scenarios in OPNsense

intermediateNetworking~30 min

Break/Fix: A new VLAN gets an address and resolves DNS but cannot reach the internet

Reported symptoms

  • ●Hosts on the new VLAN 40 (10.40.0.0/24) get a DHCP lease, gateway and DNS servers
  • ●Those hosts can ping the firewall, reach the web GUI, and resolve public names
  • ●Any attempt to reach a public IP address times out with no ICMP error returned
  • ●The original LAN (192.168.10.0/24) reaches the internet normally from the same firewall
  • ●The firewall itself can ping 1.1.1.1 and 8.8.8.8 without loss
  • ●The firewall live log shows the VLAN 40 traffic being passed, not blocked

Evidence

  • · Firewall > Log Files > Live View filtered on 10.40.0.0/24 shows `pass` on `vlan40` for every outbound flow
  • · `pfctl -sn` lists outbound NAT rules for 192.168.10.0/24 and 192.168.20.0/24 only
  • · Firewall > NAT > Outbound is set to "Hybrid outbound NAT rule generation", not "Automatic"
  • · `tcpdump -ni igb0 host 1.1.1.1` on the WAN shows `10.40.0.51 > 1.1.1.1: ICMP echo request` leaving untranslated
  • · No echo reply for that flow ever arrives on the WAN interface
  • · `pfctl -ss | grep 10.40.0.51` shows states in `SINGLE:NO_TRAFFIC` with no translation column
  • · `netstat -rn | head` on the firewall shows a healthy default route via the WAN gateway
  • · The DNS that "works" is answered by Unbound on the firewall itself, so it never left the LAN side
Diagnosis and resolutionclick to reveal

Root cause

Outbound NAT mode was changed from "Automatic" to "Hybrid" months earlier so that one host could be given a static one-to-one style translation. Hybrid mode still auto-generates rules, but only for the networks OPNsense knew about at the time the mode was set; the auto rules are rendered from the interface list as it exists, and a subnet added later is only covered if the generated set is refreshed. In this configuration the generated rules cover the two original internal subnets and nothing else, so packets from 10.40.0.0/24 match no outbound NAT rule at all. pf therefore forwards them out the WAN with their original RFC 1918 source address. The first upstream router discards them silently, which is why there is no ICMP error and why the firewall log shows a clean `pass` - the rule engine did exactly what it was asked to do. DNS appeared to work only because Unbound on the firewall answered from cache and from its own upstream queries, which are translated because they are sourced from the WAN address.

Remediation

Add an explicit outbound NAT rule for the new subnet rather than reverting the whole mode. Firewall > NAT > Outbound > Add, with interface WAN, source the 10.40.0.0/24 network (or better, a "RFC1918_Internal" alias holding every internal subnet), destination any, translation "Interface address", and a description that names the change ticket. Place it below any specific static-port or one-to-one rules and above the automatic block. Apply, then reset the affected states so existing broken flows are re-evaluated: `pfctl -k 10.40.0.0/24` clears only that source rather than flushing the whole table. If the original reason for Hybrid mode no longer exists, schedule a follow-up to return to Automatic mode, which regenerates from the interface list on every apply and removes this failure class entirely.

Verification

From a VLAN 40 host, `ping -c 4 1.1.1.1` must succeed and `curl -s https://ifconfig.io` must return the WAN public address, not a private one. On the firewall, `tcpdump -ni igb0 host 1.1.1.1` must now show the WAN address as the source rather than 10.40.0.51. `pfctl -sn | grep 10.40` must list the new rule, and `pfctl -vsn` must show its evaluation counter incrementing under test traffic - a rule that exists but never matches is the same failure with extra steps. Confirm a state now carries a translation: `pfctl -ss | grep 10.40.0.51` should show the WAN address in the translated tuple. Repeat the test from a second VLAN 40 host so the result is not an artefact of one client's cache.

Prevention

Treat "add an interface" and "add outbound NAT" as one change, not two: put the step in the VLAN provisioning checklist so the NAT rule ships with the subnet. Source outbound NAT rules from an alias that lists every internal subnet, so a new subnet is added in one place instead of being forgotten in a second. Prefer Automatic outbound NAT unless a specific requirement forces Hybrid, and record that requirement in the rule description so the next operator knows whether the mode can be reverted. Add a post-change test that curls an external echo service from every internal VLAN and asserts the returned address is the expected public IP - that single check catches missing translation, wrong translation and policy-routing mistakes in one pass.

Scenario

A network engineer added VLAN 40 for a new contractor workspace on Friday afternoon. The interface, DHCP scope, and firewall rules were all created from the existing VLAN 20 template. On Monday morning the contractors report that “the internet is down”, but the helpdesk finds they can reach the intranet, the print server, and the firewall’s own login page. Nothing was changed on the WAN, and every other network on the same firewall is healthy.

You have console and GUI access. The change that actually caused this was made by someone else, several months ago, and is not in the recent change log.

Architecture (brief)

  • One OPNsense firewall, single WAN (igb0), trunk to the access switch on igb1.
  • Internal networks: LAN 192.168.10.0/24 (igb1), VLAN 20 192.168.20.0/24 (vlan20), VLAN 40 10.40.0.0/24 (vlan40, new).
  • Unbound runs on the firewall and answers for all internal networks.
  • Outbound NAT was set to Hybrid mode during an unrelated project so that one accounting server could keep a fixed source port.

Symptoms

  • VLAN 40 clients receive a lease, correct gateway, and correct DNS servers.
  • ping 10.40.0.1 (the firewall) succeeds. The GUI loads.
  • nslookup www.example.com returns an answer.
  • ping 1.1.1.1 times out. curl https://example.com hangs then fails.
  • No ICMP unreachable, no TCP reset - the traffic simply disappears.
  • LAN and VLAN 20 are unaffected.

Available evidence

  • Live log, filtered on the VLAN 40 source, shows the flows being permitted:

    vlan40  pass  10.40.0.51:41022  1.1.1.1:443  tcp  let out anything from firewall host itself
vlan40  pass  10.40.0.51        1.1.1.1      icmp echoreq

  • The outbound NAT rule set as pf sees it:

    root@fw01:~ # pfctl -sn
nat on igb0 inet from 192.168.10.0/24 to any -> (igb0) round-robin
nat on igb0 inet from 192.168.20.0/24 to any -> (igb0) round-robin

  • A capture on the WAN interface while a VLAN 40 host pings:

    root@fw01:~ # tcpdump -ni igb0 host 1.1.1.1
09:14:02.118455 IP 10.40.0.51 > 1.1.1.1: ICMP echo request, id 4211, seq 1, length 64
09:14:03.121004 IP 10.40.0.51 > 1.1.1.1: ICMP echo request, id 4211, seq 2, length 64

  • State table entries for the client:

    root@fw01:~ # pfctl -ss | grep 10.40.0.51
all icmp 1.1.1.1:8 <- 10.40.0.51:4211       0:0
all tcp 93.184.216.34:443 <- 10.40.0.51:41022   SYN_SENT:CLOSED

  • Firewall > NAT > Outbound shows mode Hybrid outbound NAT rule generation with an empty manual rule list above the generated rules.

  • The firewall’s own ping -c 2 1.1.1.1 succeeds with 12 ms round trip.

Student investigation

Work the packet’s path, not the client’s complaint.

  1. The log says pass. What is the next stage after the filter decision, and does the log tell you anything at all about it?
  2. Compare the source address in the WAN capture with the source address you would expect a translated packet to carry. What does that single line rule out?
  3. Enumerate what pf actually loaded. What does pfctl -sn list, and which internal subnets are missing from it?
  4. Ask why DNS behaved differently from ICMP. Which host actually sent the query that reached the internet, and from which interface?
  5. Look at the outbound NAT mode. What is the practical difference between Automatic and Hybrid when a new interface is created afterwards?
  6. Before changing anything, predict the exact line you expect to see in the WAN capture after the fix. If your prediction and the result disagree, your model is wrong, not the firewall.

Validation

Reproduce the failure deliberately so you know you understand it:

  • With the fix in place, temporarily disable the new outbound NAT rule and confirm the WAN capture immediately shows the private source again.
  • Re-enable it, flush only the affected states, and confirm recovery without touching any other subnet.
  • Check that VLAN 20 traffic was never affected during either step, using its own capture rather than an assumption.

Root cause

Reveal the root cause after you have formed a hypothesis

Outbound NAT is in Hybrid mode. Hybrid keeps the automatically generated rules and lets you add manual ones, but the generated set in this configuration only covers the internal subnets that existed when the mode was applied. VLAN 40 was created later, so no rule - manual or generated - matches source 10.40.0.0/24.

pf’s filter stage and pf’s translation stage are separate. The filter said pass, which is faithfully logged. The translation stage then found no matching nat rule and left the packet untouched, so it left igb0 with source 10.40.0.51. The ISP’s first-hop router drops RFC 1918 sources silently, which is why there is no ICMP error to diagnose from.

DNS looked healthy because the client’s resolver is Unbound on the firewall. The client-to-firewall leg never needs NAT, and the firewall-to-upstream leg is sourced from the WAN address, which needs no translation either. That asymmetry is the tell: name resolution working while IP connectivity fails points at the egress path, not at the client.

Remediation

  1. Firewall > NAT > Outbound > Add.

  2. Interface WAN, TCP/IP version IPv4, protocol any.

  3. Source: 10.40.0.0/24 - or, preferred, an alias RFC1918_Internal containing every internal subnet.

  4. Destination: any. Translation target: Interface address.

  5. Description: Outbound NAT for VLAN 40 - CHG-2291.

  6. Order it below any static-port or one-to-one rules, above the generated block.

  7. Apply, then clear only the broken flows:

    root@fw01:~ # pfctl -k 10.40.0.0/24

  8. Raise a follow-up ticket to evaluate returning to Automatic mode if the original reason for Hybrid no longer applies.

Verification

# From a VLAN 40 client
$ ping -c 4 1.1.1.1
$ curl -s https://ifconfig.io      # must return the WAN public address

# On the firewall
root@fw01:~ # pfctl -sn | grep 10.40
root@fw01:~ # pfctl -vsn           # evaluation counter must increment
root@fw01:~ # tcpdump -ni igb0 host 1.1.1.1
root@fw01:~ # pfctl -ss | grep 10.40.0.51

The check that can actually fail is the evaluation counter: a rule that exists but never matches produces exactly the original symptom. Confirm the counter moves under live test traffic, and repeat the client test from a second host so the result is not one machine’s cache.

Prevention

  • Add “create outbound NAT rule” to the VLAN provisioning checklist so the translation ships with the subnet.
  • Source outbound NAT from an alias of internal subnets; adding a network then touches one object, not several rules.
  • Default to Automatic outbound NAT. If Hybrid or Manual is required, record why in the rule description and review it at each rule-hygiene cycle.
  • Add a post-change probe that curls an external echo service from every internal VLAN and asserts the expected public address. One test catches missing NAT, wrong NAT, and policy-routing errors together.