OPNsense Production Network & Security Administrator — Final Assessment

This assessment covers all forty-eight parts of the course. It has two halves.

• Part A is the auto-scored question set above this page’s body: sixty-two questions spread across packet flow, rules and state, NAT, segmentation, DNS and DHCP, multi-WAN, VPN, PKI and identity, high availability, IDS/IPS, logging, monitoring, performance, capture, troubleshooting, backup, upgrades, automation, change management, hardening and capacity.
• Part B is the written scenario set below. Each scenario carries the actions a passing answer must contain, followed by a model answer. Mark yourself one point per scenario whose required actions you produced before reading the answer.

The pass bar is 75% overall on Part A and 6 of 8 scenarios passed. Any scenario answer that omits a REQUIRED data-loss or HA-safety action fails that scenario regardless of the rest.

Part A is closed-book. Part B is open-book and open-shell: what is being marked is the evidence you collect and the order you collect it in. Every scenario answer should name the symptom and its impact, the evidence in the order you would gather it, the most likely cause with a justification, the remediation, the verification, and the rollback you kept ready. An answer that names a subsystem without naming the command that would prove it is incomplete; an answer that names a command without saying what the two possible outputs would mean is also incomplete.

Section 1 — Packet flow and PF internals

PF registers with pfil in both directions on every interface, so a forwarded packet crosses two hooks: inbound on the interface it arrived on, then — after the routing decision has chosen an egress — outbound on the interface it leaves by. Each translation is bound to one interface and fires on that interface’s hook immediately before the filter rules there: rdr on the ingress interface, nat on the egress interface. pf.conf(5) is unambiguous about the order within a hook: translation occurs before filtering, so filter rules see the addresses as they look after translation. That single sentence explains why a port forward’s pass rule names the internal target rather than the public address, and why a capture on WAN shows the translated source while a capture on LAN shows the real one.

Crossing a hook is not the same as walking the ruleset. The inbound pass creates state; the outbound hook matches that state — floating by default, so a different interface does not matter — and passes the packet without evaluating rules. Policy therefore lives on the inbound hook of the interface traffic enters by. Where the ruleset is walked outbound, the generated ruleset has an answer ready: a non-quick “Default deny / state violation rule” that names no direction, and immediately after it a non-quick pass out labelled “let out anything from firewall host itself”. Among non-quick rules the last match wins, so the inbound default is deny and the outbound default is pass. A passing student can name every drop point on the path — NIC, ARP, IP layer, PF on either hook, routing, NAT, egress NIC — and the command that proves each one.

Section 2 — Rules, aliases and state

OPNsense generates every GUI rule with quick, so the first match wins and a broad permit placed above a specific deny silently shadows it. The GUI order is not necessarily the compiled order: pfctl -s rules is the only order that matters, and the rule id in field four of a filter-log record is the durable pointer back to a configured rule, where the rule number in field one is only a position that renumbers whenever a rule is inserted above it. Block drops silently and the client times out; reject answers with a TCP RST or an ICMP unreachable and the client fails immediately. Drop on the WAN, reject on internal interfaces.

State is the other half of the ruleset. A state entry records the decision made when the flow was created, so a rule change applies to new flows at once and to existing flows never — the trap that wastes hours. pfctl -k kills targeted states and forces re-evaluation; pfctl -F state flushes everything and is service-affecting. The defaults are 24 hours for TCP established, 60 seconds for UDP and 20 seconds for ICMP, read back with pfctl -st. The state ceiling is a pf runtime limit — set limit states — read with pfctl -sm, set from Firewall → Settings → Advanced → Firewall Maximum States, and derived from RAM at roughly 100 states per MiB when that field is left blank. There is no sysctl for it. Exhaustion has one signature no other failure produces: new connections fail while established ones keep working.

Section 3 — NAT, port forwarding and multi-WAN

Outbound NAT rewrites the source on the egress interface’s outbound hook, which is why it cannot run until routing has chosen that interface. Inbound NAT rewrites the destination on the ingress interface’s inbound hook, before the rules there. Automatic mode generates the rules; Hybrid keeps them and adds manual rules on top; Manual deletes them and makes the operator responsible for every internal subnet — the mode a multi-WAN estate needs, because the routing decision and the NAT decision are independent and have to be made to agree. When they disagree, the remote host answers the address it was given and the reply arrives on a WAN the flow never left by.

A port forward is two objects: the rdr that rewrites the destination and the pass rule that permits the rewritten packet. The auto-generated associated rule names the redirect target; a hand-written rule that names the public address matches nothing. Three other failure points recur: the internal server whose default gateway is not the firewall, the ISP that filters the port upstream, and NAT reflection, which is needed only because an internal client resolved the public name to the public address — split-horizon DNS through an Unbound host override is the better fix. Multi-WAN adds dpinger: the monitor address must be reachable through the gateway being monitored, or the failover it triggers is false.

Section 4 — Segmentation, DMZ, IPv6, DHCP and DNS

An 802.1Q trunk carries a 4-byte tag that the kernel strips before PF sees the frame, so a VLAN mismatch between the firewall and the switch is invisible in the ruleset and obvious in a capture on the parent interface. Each VLAN interface is a connected route and the firewall is the router for every inter-VLAN packet; inter-VLAN traffic is not NATted, so the source address is preserved. Segmentation is the default deny at the bottom of every VLAN’s rule set, not the VLAN itself. A DMZ earns its name only when DMZ-to-LAN is denied; a “permit for convenience” rule turns the buffer zone back into part of the LAN.

IPv6 is a second, independent address family in PF: rules apply only to the family they declare, every subnet is a /64 because SLAAC requires it, and ICMPv6 carries Neighbor Discovery, Router Advertisements and PMTUD, so blocking it breaks IPv6 rather than securing it. DHCP is a broadcast protocol, so a server on another VLAN needs a relay, which converts the broadcast to a unicast and sets giaddr to its own address for scope selection; DHCPv6 keys reservations on DUID, not MAC. Unbound answers as a validating resolver by default and as a forwarder when the trust model puts an upstream in the path. Its access lists match most-specific-first and deny unmatched sources — which is why a query from a subnet nobody added comes back REFUSED, while a broken chain of trust comes back SERVFAIL.

Section 5 — VPN: WireGuard, IPsec and the failures that look alike

A tunnel is one packet wrapped inside another, and encryption makes the inner packet unreadable on the path; the outer header, the packet length and the timing stay visible. WireGuard’s AllowedIPs is two things at once — the list of inner source addresses a peer is authorised to claim, and the list of destinations the firewall routes to that peer. A site-to-site peer carries the other site’s subnet; a remote-access peer carries a unique /32. IPsec splits the work between Phase 1, which builds the IKE SA, and Phase 2, which builds the ESP SAs that carry traffic; route-based VTI is the production default because it makes the tunnel a first-class interface that firewall rules, NAT, multi-WAN and monitoring can all see.

The failures look alike and the logs tell them apart. NO_PROPOSAL_CHOSEN is a proposal mismatch; INVALID_KE_PAYLOAD is a DH group mismatch; AUTHENTICATION_FAILED is a PSK or certificate problem; TS_UNACCEPTABLE is a Phase 2 traffic-selector mismatch. A WireGuard handshake within the last two minutes proves key exchange and nothing else. Above the tunnel sit three more layers that fail silently: the firewall rules on the tunnel interface, which see the decapsulated inner addresses; the routing table, which is the final arbiter; and the MTU, whose signature is a TCP session that completes its handshake and then hangs on the first large payload. MSS clamping and a lowered VTI MTU are the cure, because the ICMP that PMTUD needs is routinely dropped somewhere on the path.

Section 6 — PKI and identity

A certificate binds a public key to a name, and modern clients match the name in the SAN, not the CN — a certificate without a SAN fails verification no matter how correct the chain is. A CSR carries the public key, the subject, the SAN and a self-signature proving possession of the private key, which never leaves the requester. Run an intermediate CA and keep the root offline; distribute the trust anchor to every verifier and check the fingerprint out of band. ACME automates issuance and renewal — HTTP-01 needs port 80 reachable from the Internet, DNS-01 works for private names and wildcards at the cost of a high-privilege DNS API token — and expiry monitoring at 60, 30, 14 and 7 days is what keeps the renewal pipeline honest.

Identity has the same shape. Local users and groups are the firewall’s first store, external directories are how an estate scales, and group membership is what carries privilege. LDAP, RADIUS and TACACS+ each fail in their own way — an unreachable server, a rotated shared secret, an expired directory certificate, a missing attribute mapping — and every one of those failures locks operators out of the firewall at the moment they most need it. A local break-glass account that works regardless of the external store is not optional, and an untested break-glass path is not a break-glass path.

Section 7 — High availability: CARP, pfsync and configuration sync

HA reduces MTTR; it does not touch MTBF. It answers device failure and nothing else — not the switch, not the WAN, not the ISP, not a configuration mistake that synchronises to both nodes. CARP is the arbitration protocol: advertisements on multicast 224.0.0.18 (ff02::12 for IPv6) with MAC 01:00:5e:00:00:12, IP protocol 112, on each VIP’s own parent segment — never on the dedicated sync link. The election compares advertisement intervals, advbase + (min(advskew + demotion, 240) / 256) seconds, and the shorter interval wins; the backup takes over after three missed advertisements, so failover is up to about three seconds plus a gratuitous ARP. Demotion is a runtime knob that adds to net.inet.carp.demotion rather than assigning it, which is why it is cleared by writing the negative of what was added.

pfsync replicates state so that established sessions survive a failover. It rides directly on IP as protocol 240 with no TCP or UDP port, so a rule or a capture filter for it selects a protocol; it is neither authenticated nor encrypted, so the only real control is an isolated sync segment that no other host can transmit on. Configuration sync is a third, independent mechanism — XMLRPC over HTTPS, one-directional from primary to backup — and it deliberately does not copy the node-specific values: interface addresses, hostnames, CARP skews and logs. Rolling maintenance follows from all three: demote the backup, work on it, validate CARP, pfsync and config sync, then demote the primary and repeat. A pair that has never been failed over on purpose is an untested pair.

Section 8 — IDS/IPS, TLS inspection and threat intelligence

An IDS inspects a copy and an IPS sits inline. On OPNsense that is the difference between netmap in passive mode and netmap in inline mode, and inline costs latency and CPU on every packet. Suricata’s rule actions are alert, drop, reject and pass; an IPS whose rules are all alert is an IDS with extra latency. The promotion discipline is fixed: run in IDS mode with the intended ruleset, triage the alerts, disable or suppress the rules that produce unacceptable false positives, convert the survivors to drop, and only then switch the interface to IPS. HOME_NET has to match the actual subnets or the signatures never fire, and capture.kernel_drops is the canary that says the engine is behind the traffic.

TLS inspection is the sharpest tool in the box. It requires a forged certificate, which means the intercept CA must be trusted by every client — including Firefox’s own NSS store — and it must be a separate CA from the production one, because the blast radius of a compromised intercept key is the entire inspected traffic stream. The legal frame comes first: notice, consent, proportionality, purpose limitation, retention, and exemptions for banking, healthcare and personal communications defined before interception is switched on. Threat-intelligence feeds arrive through URL Table aliases that replace their table atomically; monitor the last refresh and the entry count, because a feed that silently stops updating looks perfectly healthy in the configuration.

Section 9 — Logging, monitoring and performance

OPNsense logs through syslog-ng into plain text — one directory per application, one file per day, plus a latest.log symlink. The filter log lives at /var/log/filter/filter_<YYYYMMDD>.log and each record is a positional, comma-separated payload behind an RFC 5424 header: rule number, sub-rule, anchor, rule id, interface, reason, action, direction, IP version, then the IP and transport headers. Reason says why the packet was logged and action says what pf did; match is a reason and never an action. Retention is Maximum preserved files, defaulting to 31, and it bounds the file count rather than the disk footprint. clog belongs to pfSense and has no part in reading an OPNsense log.

Monitoring is triaged firewall-health first, network-health second: state-table occupancy, CPU and interrupt share, memory, disk, daemon health. The exporter’s trap is label cardinality — per-flow or per-address labels destroy a Prometheus server far faster than the firewall destroys itself. Performance is measured in packets per second, not bits per second, because the CPU cost is per packet; climbing Idrops on an input interface is the canary that says the CPU cannot keep up. AES-NI is what makes IPsec and WireGuard reach line rate; TSO and LRO make forwarding fast and must be disabled on any interface feeding an IDS or a capture, because they hide the wire-sized segments. A capture is a witness and not an actor: it proves a frame was on the wire, the filter log proves what pf decided, and the state table proves what was matched. Disagreement among the three is the most useful evidence an operator can hold.

Section 10 — Troubleshooting method and lockout recovery

The method is the deliverable: separate the user-observed symptom from the firewall-observed symptom from the cause, collect evidence before changing anything, and when the two symptoms disagree believe the firewall. Seven commands cover most incidents — arp -an, netstat -rn and route -n get, tcpdump -nei, pfctl -s state, pfctl -s rules, pfctl -s nat, and a log search — and each one is run to distinguish between named candidates rather than to see what turns up. “No output” is ambiguous until the filter has been broadened. Routing is checked before rules, because a missing route produces a symptom no rule change can fix, and FRR’s learned routes shadow static ones for any prefix FRR carries.

Lockout is the failure mode unique to a firewall. The anti-lockout rule permits TCP 80 and 443 from the LAN subnet to the firewall’s LAN address and is generated on LAN only — an operator who moves management to an OPT interface has no safety net there. The safe rule change procedure is identify, back up, peer review, verify the recovery path, apply, test from a second session on a different source address, and document. The console — serial, BMC SOL, or the hypervisor’s virtual console — is the path that always works, and it is verified at provisioning time rather than during the incident.

Section 11 — Backup, upgrades and automation

/conf/config.xml is the single source of truth; every generated file under /var and /etc is derived from it and is overwritten on the next apply. The backup is that XML, it contains private keys and shared secrets, and it must be treated as a credential wherever it is stored. What it does not contain matters as much: DHCP leases, RRD graphs, logs, SSH host keys and the firmware itself. A backup that has never been restored onto disposable hardware is an assumption, not an asset, and interface naming drift across hardware revisions is the reason the restore drill exists.

Upgrades create a ZFS boot environment, so the previous release is always available as a rollback target; plugins are versioned against a release line and an ABI mismatch is what turns an upgrade into an outage. On an HA pair the sequence is secondary first, validate, fail over, then the original master. Automation rides the same API the GUI does: authentication is HTTP Basic with the API key as the username and the secret as the password, endpoints follow /api/<module>/<controller>/<command>, writes are staged and then made real by a separate apply or reconfigure, and firewall rules written by automation live in the automation rule set under /api/firewall/filter/*. Ansible adds fleet discipline on top: a pinned collection version, match_fields that define idempotency, a single reload from a handler rather than a reload per task, and serial batches that bound the blast radius.

Section 12 — Change management, hardening and capacity

Every change answers seven questions before it is applied: what, why, expected effect, worst-case effect, backup, recovery path, and window. The worst-case question is the one that catches lockouts, and it is answered by naming the failure mode rather than by saying “the firewall breaks”. Rules are reviewed on a cadence because rulesets rot: rules shadowed by an earlier quick match, duplicates that match the same traffic, and dead rules that have not matched a packet in months all survive indefinitely unless somebody looks. Removal is a change and gets the same procedure as any other change.

Hardening is organised by plane — management, control and data — and each plane has its own threats and its own controls. Bogon and anti-spoofing rules, rate-limited rather than blanket-blocked ICMP, source-restricted management, key-only SSH, MFA, scoped API credentials and a default-deny data plane are the baseline; anything that looks secure and produces no evidence is theatre. Capacity closes the loop. Throughput is sized from application throughput and not from wire rate; state-table occupancy is Little’s Law, the new-flow rate times the mean state lifetime, times a headroom factor; VPN throughput is bounded by whichever is smaller, the link or the encryption; and log volume is a disk, network and retention budget that has to be planned before the syslog target starts dropping records.

Part B — Written scenarios

Mark yourself one point per scenario whose required actions you produced before reading the model answer.

Scenario 1 — LAN clients cannot reach the Internet

A user on the Users VLAN cannot reach example.com. They can reach 10.0.0.1, the firewall’s Users interface. Both WAN gateways show online in the gateway group, and drill example.com @10.0.0.1 from the firewall itself answers correctly.

Required actions

1. Reproduce from the VLAN, not from the firewall: drill example.com from a host on Users, and a direct connection attempt to a literal address, so DNS and forwarding are separated before anything else.
2. Capture on the Users interface and on the selected WAN at the same time, for the same flow, and compare. If the packet is on Users and not on WAN, PF or routing dropped it; if it is on both, look at the source address it carries on WAN.
3. Read pfctl -s nat and confirm a rule covers this source subnet on the interface the flow actually leaves by.
4. Read the filter log for the flow, taking the interface and direction fields first, so the addresses in the record are interpreted against the right hook.
5. Confirm the routing decision with route -n get for the destination, and check whether a policy-routing rule with a Gateway set is overriding it.

Model answer: the capture is the discriminator. A packet on Users and not on WAN means PF blocked it on the Users inbound hook or the routing table had nowhere to send it — read the filter log and netstat -rn. A packet on WAN still carrying the client’s private source address means outbound NAT did not cover this subnet, which is the classic result of switching to Manual mode and missing a subnet, or of adding a VLAN and not adding its NAT rule. A packet on WAN with the wrong WAN’s address is the multi-WAN NAT-and-routing disagreement: the reply is delivered to the other WAN and dropped. The fix is a manual outbound NAT rule pinning this subnet to the WAN the firewall rule sends it out by.

Scenario 2 — Tunnel up, no traffic

A WireGuard site-to-site tunnel between HQ and Branch reports a handshake 30 seconds old at both ends. A user at HQ cannot reach 10.50.0.50 at Branch. HQ-local resources work normally.

Required actions

1. State explicitly that the handshake proves key exchange and nothing about the inner flow, so the investigation starts at the next layer.
2. Read AllowedIPs on both peer entries. HQ’s entry for Branch must contain 10.50.0.0/24; Branch’s entry for HQ must contain HQ’s subnet. Check for overlap with any other peer.
3. Read the firewall rules on the WireGuard interface at both ends — they see the decapsulated inner addresses, and the default is deny.
4. Run wg show at both ends while the user generates traffic and watch whether the transfer counters move, and in which direction.
5. Capture on the WireGuard interface at HQ. If the inner packet is not there, it never entered the tunnel and the fault is local; if it is, the fault is at Branch — AllowedIPs, rules, or the return route.

Model answer: AllowedIPs and the tunnel-interface rules account for most of these incidents, and the two are distinguishable by where the counters move. AllowedIPs is both the authorisation list and the routing entry, so a missing subnet means packets never enter the tunnel at all and HQ’s transmit counter does not rise for that flow. If HQ transmits and Branch does not receive, look at the path between the endpoints. If Branch receives and nothing reaches 10.50.0.50, the fault is Branch’s tunnel-interface rules or the return route on the server itself. Check the routing table last and believe it: it is the final arbiter and no rule change substitutes for a wrong route.

Scenario 3 — Failover did not happen

FW-A loses power. Hosts at the site lose connectivity immediately. CARP was expected to move the VIPs to FW-B within about three seconds and did not. When FW-A returns, both nodes are healthy and traffic flows.

Required actions

1. Read FW-B’s logs across the failure window: did it see the advertisements stop, and did it attempt to promote?
2. Verify FW-B’s interfaces were up at the time. A node cannot win an election on a segment it cannot transmit on.
3. Check the switching layer for anything that suppresses link-local multicast on the VLANs carrying VIPs — multicast suppression, IGMP snooping without a querier, port isolation, port security.
4. Compare the CARP configuration on both nodes: VHID, password, advbase and advskew, on each VIP’s parent interface.
5. Read sysctl net.inet.carp.demotion on FW-B — a demotion left over from earlier maintenance keeps a healthy node from taking over.
6. Capture with tcpdump -n -i <parent> proto carp on the VIP’s parent segment, not on the sync link, and confirm advertisements are seen.

Model answer: the two dominant causes are Layer 2 and demotion. If the switch does not carry CARP advertisements between the peers, neither node hears the other and failover cannot be triggered by an advertisement gap — the same fault also produces split brain in the opposite failure mode. A leftover demotion is the quieter cause: the node is healthy, advertising and deliberately losing, and the tell is in the demotion sysctl or on the Virtual IPs status page. Note that the sysctl adds rather than assigns, so it is cleared by writing the negative of what was added. A healthy pfsync link proves nothing here; CARP and pfsync are independent subsystems on different segments.

Scenario 4 — State table exhaustion

New connections are failing intermittently across the estate while existing sessions continue to work. The firewall has been logging warnings for an hour.

Required actions

1. Confirm the signature before acting: new flows failing while established flows work is state exhaustion and almost nothing else.
2. Read the ceiling with pfctl -sm and the occupancy with pfctl -si, and note whether the memory counter is climbing.
3. Identify the source of the pressure: group the state table by source address and look for one address holding a disproportionate share.
4. Decide between two remedies on evidence — address the source (block or rate-limit a misbehaving or malicious host) or raise the ceiling under Firewall → Settings → Advanced → Firewall Maximum States with a written justification for the new figure against the RAM available.
5. Verify with pfctl -sm that the new ceiling is in force, and set monitoring on the occupancy ratio and the memory counter.

Model answer: the required distinction is between a firewall that is too small and a firewall that is under attack, and it is made by looking at the distribution of the state table rather than at its size. A flat distribution across thousands of clients means the deployment outgrew its ceiling — size it with Little’s Law, arrival rate times mean lifetime times a headroom factor, and confirm the new figure is affordable against RAM. One address holding hundreds of thousands of entries is a DoS or a malfunctioning host, and raising the ceiling only buys time. Do not reach for a sysctl: the state limit is a pf runtime limit read with pfctl -sm, not a kernel tunable. Alert at 80% and 95% occupancy afterwards so the next occurrence is seen before users report it.

Scenario 5 — A rule change breaks production

A rule change made ten minutes ago has taken a mission-critical application offline. Users are escalating.

Required actions

1. Restore service before diagnosing: identify the change from the ticket or the configuration revision history and revert it through the GUI or the API. Do not edit generated files.
2. Clear the affected states with pfctl -k so that flows created under the broken rule are re-evaluated against the reverted one.
3. Verify from a second session on a different source address, not from the workstation that is already connected.
4. Capture the evidence for the review before the window closes: the compiled ruleset before and after, the filter-log records for the affected flows, and the state counts.
5. Write the post-incident record: what the rule was intended to do, what it actually matched, and which step of the pre-change checklist would have caught it.

Model answer: the rollback comes first and it goes through the supported path, because a hand-edited generated file is overwritten by the next apply and will re-break production at a time nobody chooses. Clearing state is what makes the rollback take effect for sessions that already exist — without it the operator sees a partial recovery and concludes the revert failed. The two-session test is what proves the recovery rather than assuming it. The review usually lands on the same place: the worst-case-effect question on the pre-change checklist was answered as “the firewall breaks” instead of naming the flow that would be shadowed.

Scenario 6 — A backup nobody has restored

Daily configuration backups have been taken for six months and stored off-device. None has ever been restored. The business asks whether the firewall could be rebuilt from them.

Required actions

1. Retrieve a backup and decrypt it, confirming that the passphrase is held somewhere other than the storage system itself.
2. Restore onto disposable hardware or a VM — never onto production — and record how long each stage takes.
3. Validate the restored device end to end: it boots, the GUI is reachable, the interfaces come up with the right assignments, and DNS, DHCP and the VPNs are healthy.
4. Check specifically for what the XML does not carry: DHCP leases, RRD data, logs, SSH host keys, and the firmware and plugin set, and note what has to be reinstalled before a restore is usable.
5. Verify interface mapping against the destination hardware — driver names and NIC ordering are not stable across appliance revisions.
6. Document the result, fix whatever failed, and put the drill on a schedule.

Model answer: an untested backup is an assumption. The drill is run on disposable hardware because the failure modes it uncovers are exactly the ones that would extend an outage — a plugin that is no longer available, a certificate whose private key was redacted out of the copy that was stored, an interface assignment that lands WAN on the wrong NIC. The timing data from the drill is what makes the DR runbook honest, and the difference between a cold spare on site and a procurement lead time is the difference between a four-hour recovery and a day.

Scenario 7 — IPS blocking legitimate traffic

An IPS rule is dropping traffic to an internal service the operator knows is legitimate. Users are affected now.

Required actions

1. Identify the exact rule from the alert: signature id, revision, category, and the flow it matched, with the packet detail from the EVE record.
2. Decide whether the match is a true positive on unexpected traffic or a false positive on legitimate traffic — the packet detail in the alert is the evidence, not the operator’s expectation.
3. If it is a false positive, suppress the rule for the specific source and destination, or disable it, rather than switching the interface out of IPS mode.
4. If it is a true positive, keep the block and investigate the source and the destination host.
5. Record the tuning change with the reason, the scope and the date, and put the rule back on the review list.

Model answer: “enable every rule” is the anti-pattern the course names, and its consequence is exactly this incident. The narrow fix is preferred over the broad one: suppression for the specific flow keeps the signature working everywhere else, where disabling the rule estate-wide or dropping the interface out of IPS mode discards detection the operator has already paid for. The deeper correction is process: rules reach drop only after being observed in IDS mode with their false-positive rate measured and accepted, and the tuning record is what stops the same rule being re-enabled by the next ruleset update without anyone noticing.

Scenario 8 — An HA upgrade that took the site down

An operator upgraded both nodes of an HA pair at the same time. Both rebooted together and the site lost connectivity for twenty minutes. Service was restored by hand.

Required actions

1. Restore service first: bring one node up on a known-good boot environment and confirm it holds the VIPs before touching the other.
2. Validate the recovered node fully — CARP state on each VIP’s parent interface, pfsync replication, configuration sync, then the services.
3. Record the incident with the actual root cause: the procedure exists and was not followed.
4. Re-state the correct sequence: demote the backup, upgrade it, validate it, demote the master so the upgraded node takes over, validate again, upgrade the original master, then decide deliberately whether to fail back — and clear the demotion.
5. Update the change-management record and the upgrade runbook so the next operator meets an explicit prohibition on simultaneous upgrades.

Model answer: an HA pair provides availability only while one node is available. Upgrading both at once removes the property the pair exists to provide, and the twenty-minute outage is the entirely predictable consequence. The recovery is boot-environment rollback on one node, because that is the fastest path back from a firmware-level problem, followed by full validation of the three HA mechanisms separately — a node can be up, and holding VIPs, while pfsync has never resumed. The post-incident output is a change to the runbook rather than a note about operator care.

End of assessment

A passing result is 75% on Part A and 6 of 8 scenarios in Part B. If you did not pass, the section that carries each question is named in its explanation; re-read the corresponding lessons and the matching runbooks and break/fix scenarios in the course toolkit before retaking. Give it a week of study rather than a second attempt on the same day — the questions that reward judgement do not become easier by being seen twice.