high riskservice affecting~60 min

Runbook: Configure a Firewall Policy

1 · Prerequisites

Confirm every item is in place before any state change.

Stateful vs stateless filtering — conntrack, NEW/ESTABLISHED/RELATED, the performance argument
Zones and chains — base chains, named rule-sets, `firewall zone`, and jump targets
Rule ordering — sequence numbers, first-match, action accept/drop/reject
Default deny — WAN-IN posture, established accept, INVALID log, the production reference architecture
Firewall troubleshooting — log, conntrack, packet capture, the diagnostic method
commit-confirm — the rollback safety net for remote changes

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Out-of-band access is confirmed by USING it, not by believing it exists: log in over serial/IPMI/LOM now and leave that session open. A change to input filter, or to any rule-set the management path traverses, can end the SSH session you are typing into.
· The image version is established with show version, because the firewall tree moved in 1.4. On 1.4/1.5 the nodes are set firewall ipv4 name ... and set firewall zone ...; the older set firewall name ... and set zone-policy ... forms are rejected at commit.
· The chain that actually carries this traffic is identified from counters, not from the network diagram: show firewall statistics, then the specific chain. Traffic TO the router is input filter; traffic THROUGH the router is forward filter. A rule in the wrong one never fires and reads as "the firewall is broken".
· Whether the estate is zone-based is established with show firewall zone-policy. If zones are in use, policy lives in named rule-sets reached from the zone matrix, and a rule added directly to forward filter may never be reached.
· A management permit rule already exists in the chain being changed AND has a non-zero packet counter. An unproven management rule is not a management rule.
· The pre-change configuration is on disk as a file you can load back: save /config/pre-change-TICKET.conf, plus show configuration commands | match firewall captured to the ticket.
· The candidate diff has been peer-reviewed for rule ORDER, not only rule content. Evaluation is first-match, so the sequence number is the policy.
· The commit-confirm window is chosen and the operator understands the semantics: if confirm is not typed within that many minutes, the router reverts to the previous configuration on its own.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Open and keep an out-of-band console session. Do not proceed without it if the change touches input filter or a rule-set on the management path.
2Capture the pre-change baseline: show configuration commands | match firewall and show firewall statistics into the ticket, then save /config/pre-change-TICKET.conf.
3Identify the target chain by evidence. Run show firewall ipv4 input filter and show firewall ipv4 forward filter and read the counters. The chain whose counters move when the traffic in question flows is the chain to change.
4If zones are in use, resolve the rule-set name from the matrix: show firewall zone-policy. Remember the direction: set firewall zone DEST from SRC firewall name RULESET names the DESTINATION zone first.
5Confirm the management permit fires. Generate management traffic and re-read the rule counter. If it does not move, stop: the rule you believe is protecting you is not the rule being consulted.
6Enter configure and write the rule. Choose a sequence number that places it correctly against the rules already there; leave a gap (steps of 10) so a later insert does not need a renumber.
7Run compare and read the diff aloud against intent. Confirm the new rule sits ABOVE anything it must take precedence over and BELOW anything that must still win.
8Apply with a timer, not with a bare commit: commit-confirm 10. The router now reverts by itself if you stop typing.
9While the timer runs, verify: the new rule counter is non-zero for the traffic it targets, and the default-action counter is NOT climbing for traffic you just permitted.
10Test the permitted flow end to end from the real source, in the real direction. A test from the router itself proves output/local behaviour, not forward.
11Only after verification passes, type confirm to keep the change. If any check failed, do nothing and let the window expire.
12Persist and record: save, then show configuration commands | match firewall again and diff it against the pre-change capture in the ticket.

4 · Verification

Confirm the procedure actually fixed the problem.

✓show firewall ipv4 name RULESET rule N (or the equivalent base-chain form) shows a non-zero packet counter for the new rule after the target traffic has been generated. A zero counter is an unproven rule, not a working one.
✓The chain default-action counter does not increase while the newly permitted flow is running — proof the packet is matching the intended rule rather than falling to the bottom.
✓For a new deny, the blocked flow fails from the real source AND at least one adjacent flow that must keep working is retested and still works.
✓The management session survives, and the out-of-band console still reaches a logged-in prompt.
✓show log firewall ipv4 name RULESET rule N shows the expected entries when the rule carries log.
✓compare saved is empty after save, confirming the running and boot configurations agree.
✓The post-change show configuration commands | match firewall differs from the pre-change capture by exactly the reviewed diff and nothing else.

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶Not yet committed: discard in configuration mode drops the candidate. Nothing reached the kernel. This is the cheapest exit and it is available right up to the commit.
↶Committed under commit-confirm N and something is wrong: do nothing. Do not type confirm. The router reverts to the previous configuration when the window expires — this is the whole reason the timer exists.
↶Committed and confirmed: reload the pre-change file and commit it — load /config/pre-change-TICKET.conf then commit. load replaces the running configuration, so this is the in-window rollback that does not depend on revision numbering.
↶Do NOT reach for rollback N as the in-window undo. The VyOS documentation states that rollback to a revision currently requires a reboot, so on a router carrying traffic it is a break-glass action with an outage attached, not a quick revert.
↶If the management path is gone, the rollback happens on the out-of-band console. This is the step the console was opened for; if there is no console, there is no rollback.
↶After any rollback, re-verify on counters exactly as after the change. A rollback that was not verified is a second unverified change.

6 · Escalation

When the runbook isn't enough, contact:

· Escalate immediately if the router is unreachable from BOTH the management network and the out-of-band console — that is a site visit, not a configuration problem, and the sooner it is called the sooner someone is walking.
· Escalate to network engineering before touching the second router of a redundant pair. Applying the same firewall change to both halves inside one window removes the working half you were going to compare against.
· Escalate to the security owner if the change widens exposure — a new inbound permit, a source restriction removed, or a default-action moved from drop toward accept. That is a policy decision with an owner, not an operator judgement call.
· Escalate rather than proceed when the pre-checks cannot be satisfied: no working console, no proven management permit, or a chain whose counters do not match the diagram. Holding is a legitimate outcome — record the hold with an owner and an end time, and hand back the change window.
· Escalate if rule counters and observed behaviour disagree after the change. That combination points below the VyOS layer (a stale rendered ruleset, conntrack state, or an offload path) and needs someone who can read nft list ruleset against the config.

A firewall change on a router is the change class where the tool you would use to fix the mistake is the thing the mistake just broke. Everything in this runbook is arranged around that one fact: evidence before the edit, a timer around the commit, and verification that reads counters rather than intentions.

When this runbook applies

Use it for a rule-level change to an existing firewall policy on a router that is carrying traffic: adding a permit for a new service, tightening or removing a permit, inserting a deny, or changing a source restriction.

Do not use it for:

Building the initial policy on a router with no policy. That is a deployment, done before the router carries traffic, and it does not need the lockout scaffolding here.
A change whose real subject is NAT. If the traffic is translated, the firewall sees the post-translation address and the two changes have to be reasoned about together; go to the NAT runbook first.
An incident where traffic is already blocked and nobody knows why. That is a diagnosis, not a change. Diagnose first — a change made while the cause is still unknown adds a variable to an investigation.

The version fork, before anything else

The firewall configuration tree moved in VyOS 1.4 and 1.5 kept the new form. This is not cosmetic: the old node names are rejected at commit, and a runbook written against the wrong generation fails in the middle of a change window rather than at review time.

Subject	1.3	1.4 / 1.5
Named rule-set	`set firewall name NAME ...`	`set firewall ipv4 name NAME ...`
Zones	`set zone-policy zone Z ...`	`set firewall zone Z ...`
Connection state	`state established enable`	`state established`
Interface binding	per-interface `firewall in`/`out`	rules in `firewall ipv4 forward filter`, `action jump`

Several lessons in Part XXXVII of this course are still written in the 1.3 form and say so in a banner at the top. The commands in this runbook are the 1.4/1.5 form. Establish which generation you are on before you write anything.

Read-only / Safeestablish the generation

show version
show configuration commands | match firewall | head -20

If the captured configuration contains set firewall name, you are on 1.3 and this runbook’s syntax does not apply to your box. Stop and get the 1.3-era reference.

Evidence before the edit: find the chain that actually carries the traffic

The most common wasted hour on a VyOS firewall is a correct rule in the wrong chain. The rule commits cleanly, the counter never moves, the traffic keeps doing what it was doing, and the operator concludes the firewall is misbehaving.

There are three base chains and they are not interchangeable:

input filter — traffic destined for the router itself. SSH to the router, BGP sessions the router terminates, ICMP to a router address. This is the chain that locks you out.
forward filter — traffic passing through the router. Everything your users care about. This is the chain that breaks the site.
output filter — traffic the router originates. Rarely the answer, frequently the wrong guess.

Named rule-sets (set firewall ipv4 name NAME) are not chains in their own right. They are reached from a base chain by an action jump rule, or from the zone matrix. A rule-set nothing jumps to is dead code that commits successfully.

Read-only / Saferead the counters, not the diagram

show firewall
show firewall summary
show firewall statistics
show firewall ipv4 input filter
show firewall ipv4 forward filter
show firewall zone-policy

Generate the traffic in question — one connection is enough — and run the chain commands again. The chain whose counters moved is the chain that carries it. That is a two-minute check that removes the single largest source of firewall change failures, which is why it is a procedure step and not a suggestion.

On a zone-based router, the matrix names the destination first

show firewall zone-policy tells you whether zones are in play. If they are, the policy for a flow lives in a named rule-set selected by the matrix, and the binding reads:

set firewall zone LAN from WAN firewall name WAN-TO-LAN

The zone named immediately after zone is the destination. zone LAN from WAN is traffic arriving into LAN from WAN. Reading it as source-then-destination inverts every policy decision you are about to make, and the resulting rule-set is not wrong so much as pointed the other way.

The lockout rule

Before any deny rule exists, and before default-action moves anywhere near drop, the permit that keeps you connected must already be in the chain and must have a non-zero counter.

Read-only / Safeprove the management permit before you rely on it

show firewall ipv4 input filter rule 10
# generate management traffic - open a second SSH session, or run a
# command over the existing one - then read it again
show firewall ipv4 input filter rule 10

If the counter does not move, one of three things is true: the rule is in the wrong chain, an earlier rule is matching first, or a global state policy is admitting your session before the chain is consulted. All three mean the rule you are about to trust is not the rule protecting you.

Ordering: the sequence number is the policy

Evaluation is first-match. A rule does not have a priority, it has a position, and the first rule that matches ends the walk. This means a broader rule placed above a narrower one silently makes the narrower one dead code — it commits, it displays, and it never fires.

The review question is therefore not “is this rule correct” but “what is above it”. Two placements to get right every time:

A deny must sit above every permit it is meant to override. A brute-force drop placed after the SSH permit never sees a packet.
A permit must sit below any deny that must still win, and above the chain’s fall-through.

Use steps of 10 so the next person can insert without renumbering, and never renumber an existing rule as part of an unrelated change — renumbering is a policy change wearing a formatting change’s clothes.

Configuration changewrite the rule, then read the diff

configure

set firewall ipv4 name WAN-TO-LAN rule 40 description "HTTPS to web server"
set firewall ipv4 name WAN-TO-LAN rule 40 action accept
set firewall ipv4 name WAN-TO-LAN rule 40 protocol tcp
set firewall ipv4 name WAN-TO-LAN rule 40 destination address 192.0.2.10
set firewall ipv4 name WAN-TO-LAN rule 40 destination port 443
set firewall ipv4 name WAN-TO-LAN rule 40 state new

compare

Read the compare output against the intent sentence from the ticket. If the diff contains anything you cannot account for, discard and start again; a candidate configuration is cheap to throw away and expensive to commit by accident.

Apply with a timer

Service impact possiblecommit-confirm, then verify, then confirm

commit-confirm 10

# ... verification happens here, in operational mode via run ...
run show firewall ipv4 name WAN-TO-LAN rule 40

# only when verification has passed:
confirm
save

commit-confirm 10 applies the change and starts a ten-minute timer. If confirm is not entered before it expires, the router reverts to the previous configuration by itself. The default window when no number is given is ten minutes.

This is the single most valuable habit on a router you administer remotely, and its value is exactly in the case where you cannot act: the change cuts your session, you cannot type anything, and the router undoes it anyway.

`rollback N` is not the undo button — it reboots

The obvious-looking revert is rollback 1, and it is the wrong reflex on a production router. The VyOS documentation is explicit: rollback “will apply the selected revision and trigger a system reboot”, and the command itself asks for exactly that.

vyos@vyos# rollback 1
Proceed with reboot? [confirm][y]

On a box carrying traffic that prompt is an outage, not a revert — every session, tunnel and routing adjacency on the router goes with it.

The three real rollbacks, cheapest first:

Before commit — discard. Nothing reached the kernel.
After commit-confirm, before confirm — do nothing and let the window expire.
After confirm — load /config/pre-change-TICKET.conf then commit. load replaces the running configuration wholesale, which is why the pre-change file is saved as a pre-check rather than improvised afterwards.

Keep rollback N for the case where the configuration is unusable and a reboot is already acceptable.

Verification that can fail

A verification step that cannot come out negative is not verification. Two that can:

The new rule’s counter is non-zero. Generate the traffic the rule targets and read the counter. Zero means the packet is not reaching this rule — wrong chain, wrong interface direction, or an earlier match.

The default-action counter is not climbing while the newly permitted flow runs. If both the new rule and the fall-through are counting, the flow is partly matching and partly falling through, which usually means the rule covers one direction and not the other.

Read-only / Safecounters and logs

show firewall ipv4 name WAN-TO-LAN
show firewall ipv4 name WAN-TO-LAN rule 40
show firewall statistics
show log firewall ipv4 name WAN-TO-LAN rule 40

Test from the real source, in the real direction. A test run from the router itself exercises the output path and tells you nothing about forward.

When the config and the behaviour disagree, read the rendered ruleset

VyOS renders the configuration tree into an nftables ruleset. Almost always the two agree, and when they do not — the rule is in show firewall, the counter is zero, and the traffic demonstrably arrives — the question is whether the kernel has what you think it has.

sudo nft list ruleset

Read the rendered chains for the rule you just added and for anything above it in the same chain. A rule present in the configuration but absent from the ruleset means the commit did not fully apply; a rule present in both with a zero counter means the packet is not reaching the chain at all, and the question moves back to interfaces and hooks.

This is a diagnostic of last resort, not a routine step. Reach for it when counters and observed behaviour disagree, and escalate with its output rather than making another change on top.

What each step costs if it is wrong

Step	Cost of getting it wrong
Wrong base chain	Rule never fires. Traffic unchanged. Time lost, and a second change stacked on the first.
Deny placed below the permit it overrides	Dead code. The policy reads as tightened and is not. Discovered by an audit, or by an incident.
`default-action drop` before a proven management permit	Immediate lockout of every remote session, including the one applying it. Recovery is the console.
Bare `commit` instead of `commit-confirm`	The lockout has no expiry. Recovery is the console or a site visit.
`rollback N` used as the undo	Reboot. Every session, tunnel and adjacency on the router drops.
Verified by reading the config instead of the counters	A rule that commits but never matches is indistinguishable from one that works.

Escalation and the option to hold

Holding is a first-class outcome. If the out-of-band console cannot be reached, or the management permit will not produce a counter, or the chain counters contradict the network diagram, the correct action is to stop and hand the window back. Record the hold with an owner and a time by which it is revisited — an unowned hold becomes a permanently deferred change and then a surprise.

Escalate rather than continue when:

The router answers on neither the management network nor the console.
The change is the second half of a redundant pair inside the same window.
The change widens exposure and no security owner has signed the diff.
Counters and behaviour disagree after the change.