medium riskservice affecting~40 min

Runbook: Troubleshoot Blocked Traffic

1 · Prerequisites

Confirm every item is in place before any state change.

Stateful vs stateless filtering — conntrack, NEW/ESTABLISHED/RELATED, the performance argument
Zones and chains — base chains, named rule-sets, `firewall zone`, and jump targets
Rule ordering — sequence numbers, first-match, action accept/drop/reject
State tracking — connection marks, recent, and the limits of the conntrack matches
Default deny — WAN-IN posture, established accept, INVALID log, the production reference architecture
Firewall troubleshooting — log, conntrack, packet capture, the diagnostic method
NAT with firewall — rule placement, established match, conntrack interaction
Return-path — forward and reverse, asymmetric routing, return-path sanity

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· The failing flow is written down as a five-tuple plus two interfaces: source address, destination address, protocol, destination port, the interface it enters this router on and the interface it should leave by. A firewall rule is written in exactly those terms and the incident cannot be discussed in any others
· The failure reproduces on demand. Every piece of evidence in this runbook is a counter delta or a log line matched to a timestamp, and none of that works against a fault that happens twice a day. If it is genuinely intermittent, say so and treat it as a different investigation
· You know whether the packet is transit traffic or traffic destined for the router itself. Those are different hooks with different rule sets, and reading the wrong one is the single most common wrong turn in this incident
· Whether NAT sits in this path has been established BEFORE any rule is read. NAT decides which addresses the rules are written in, and a rule written against the wrong side of a translation never matches and never logs
· The change history has been read: show configuration commit. Traffic that stopped within minutes of a commit is a rollback, and vyos-rb-rollback-bad-config is faster and more certain than a diagnosis
· You have confirmed which configuration tree this router uses. VyOS 1.3 puts rule sets under firewall name; 1.4 and 1.5 put them under firewall ipv4 name and bind them through the forward, input and output filters. The chain names in the log and the command that prints the counters both differ
· Out-of-band access is confirmed if the investigation will touch the input chain - that is the chain admitting your own SSH session, and an existing connection survives edits that would stop a new one from ever being made
· The counters have a baseline. A default-deny rule with 847,000 hits proves nothing on its own; the only reading that matters is whether it advances by your packet count while you reproduce the failure

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Reproduce the failure and write down the exact time. Every counter delta, log line and conntrack entry from here on is matched against that timestamp, and an unrepeatable symptom cannot be attributed to a rule.
2Decide which chain SHOULD fire before reading any rules: transit traffic or router-destined, which zone pair or hook, which direction. Naming the chain first stops you from finding a plausible rule in the wrong rule set and believing it.
3Prove the packet reaches this router. Capture on the ingress interface with a filter tight enough to see only your flow. If nothing arrives, this is not a firewall incident at all - it is routing, addressing or the link, and the runbook is vyos-rb-troubleshoot-missing-route.
4Prove whether the packet leaves. Capture on the expected egress interface during the same reproduction. Arrives and does not leave means this router stopped it, and only now is the firewall a suspect rather than an assumption.
5Take a two-sample counter reading of the chain while reproducing: read the rule counters, reproduce, read again. The rule whose counter advances by your packet count is the rule that fired. Absolute totals since boot say nothing.
6Read the firewall log for the reproduction timestamp. The bracketed tag names the chain, the rule number and the action, and the IN and OUT fields name the interfaces, which together identify the rule without guessing.
7If nothing logs, treat that as the finding rather than as a dead end: you have a drop you cannot see. Turn on logging for the default action or the suspect rule, reproduce once, read the entry, and then take the logging back off - a permanent debug log on a WAN chain is its own incident.
8Check whether the drop is a state judgement rather than a rule match. An invalid-state counter advancing in step with the failure means this router is seeing one direction of a flow whose other direction took a different path. That is asymmetric routing, and no firewall rule fixes it.
9Read the conntrack entry for the flow. Absent when the traffic should be established is a state problem; present but carrying different addresses from the ones your rule matches is the NAT problem, and they need different fixes.
10Where NAT is in the path, compare the addresses in the conntrack entry against the addresses in the rule. The filter hooks see the post-DNAT destination and the pre-SNAT source, so a rule written against the public address of a port-forwarded service never matches and never logs.
11Decide whether the drop was DELIBERATE before changing anything. A default-deny doing its job is indistinguishable from a fault to the person whose traffic it is dropping, and the difference is a policy decision with a named owner rather than a rule edit.
12If a rule change is genuinely the fix, change ONE rule, with compare and commit-confirm, then reproduce. Adding a rule is small; renumbering or reordering is not, because it changes which rule fires for flows you never tested.
13Run the negative test as well as the positive one: confirm the flow that should now pass does pass, AND that a flow which should still be denied is still denied. Then remove every diagnostic artefact, save, and record the chain, the rule and the decision.

4 · Verification

Confirm the procedure actually fixed the problem.

✓The flow that failed now succeeds, tested with the same five-tuple on the same interfaces rather than with a convenient approximation of it
✓The rule you intended to fire is the rule whose counter advances, proved by a two-sample reading taken across a reproduction rather than by reading the rule and finding it plausible
✓The negative test still fails. A change that admits what you wanted and also admits what you did not looks exactly like a success from the reporter side, and is discovered by somebody else later
✓The default-deny counter is still advancing. A catch-all that has stopped counting means something above it now matches everything, which is a far larger change than the one you meant to make
✓The firewall log carries no new drops for this flow at the reproduction timestamp, and the invalid-state counter is not advancing
✓Conntrack holds an entry for the flow, in the expected state, carrying the expected post-NAT addresses
✓Every diagnostic artefact is gone: trace rules removed, temporary logging switched back off, captures stopped. A trace rule left behind adds work to every packet on that chain and log volume to every operator after you
✓show configuration commit shows the archive entry, diff /config/config.boot <(show configuration commands) is empty, and the policy decision - what was opened, for whom, and who approved it - is written in the ticket

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶To undo one added rule, delete that rule number, compare, commit-confirm, and reproduce. Deleting the specific rule is nearly always the correct revert and has a blast radius of exactly that rule
↶Do NOT roll the whole chain back to undo one rule. rollback N reverts every configuration change since that archive entry, and on a firewall during an incident that regularly includes somebody else responding to the same incident
↶A renumbering is not undone by deleting the new rule. Reordering changed which rule fires for flows nobody tested, so revert it through the archive with compare N read in full, and re-run the negative tests afterwards
↶Trace rules and counter resets applied directly with nft are NOT in the configuration tree. They will not be removed by rollback, they will not appear in show configuration, and they do not survive a reboot. Write down every one you make, because the next operator cannot see them
↶If the change was to the input chain and you have lost access, stop issuing remote commands and use the console. If the change was made with commit-confirm, doing nothing is the recovery and it needs no working path back to the router
↶POINT OF NO RETURN: traffic admitted while a rule was widened has already been delivered. Narrowing the rule again stops the next packet; it does not recall the session that is already inside, and for an inbound rule that is a security event rather than a routing one
↶POINT OF NO RETURN: conntrack entries deleted during diagnosis are gone. The flows that were using them were dropped at that moment and their applications had to reconnect - the deletion is a small outage, not a read

6 · Escalation

When the runbook isn't enough, contact:

· Escalate to the policy owner as soon as the drop turns out to be deliberate. Widening a rule to make a page stop is a change to the statement of intent the firewall exists to make, and it belongs to whoever wrote the intent rather than to whoever is on call
· Escalate to security when the request is to admit inbound traffic from an untrusted network, regardless of how small the change looks. A single accept rule on a WAN-facing chain is a permanent exposure and it will outlive the ticket that created it
· Escalate to whoever owns routing when the evidence is an invalid-state drop caused by asymmetry. The firewall is reporting a routing design problem accurately, and rules added to work around it hide the problem rather than solving it
· Escalate to the application owner when the flow depends on secondary connections the firewall cannot associate with the first one. That is a question about how the application works before it is a question about the rule set
· Escalate to the upstream or peer network when the ingress capture shows the packet never arriving. Give them the five-tuple, the timestamp and the capture, not the conclusion that they are blocking it
· Escalate rather than continue after two rule changes have failed to move the result. Two failures usually means the chain was misidentified at step two, and a third change on the same assumption leaves a rule set nobody can read

“The firewall is blocking it” is almost never an observation. It is a conclusion somebody reached because the firewall is the part of the path they can name, and at least four completely different faults arrive wearing it.

The packet may never have reached this router at all. It may have reached it, been forwarded correctly, and had its reply dropped somewhere else. It may have been dropped here by a rule that is doing precisely what it was written to do. Or it may have been dropped by a rule that could never have matched it, because NAT rewrote the address between the rule and the packet.

Those four have nothing in common except the symptom, and only one of them is fixed by editing a rule. So the order of this runbook is: establish where the packet stopped, then who stopped it, and only then whether that was a mistake.

When to use this runbook

Traffic that is expected to pass through a VyOS router, or to reach the router itself, does not arrive, and the failure is reproducible.

When not to use it

Some traffic gets through and some does not, proportionally, across all sizes. That is loss, and it is vyos-rb-investigate-packet-loss.
Small packets pass and large ones fail. That is vyos-rb-investigate-mtu-issue, and no firewall rule will fix it.
The destination has no route at all. Start at vyos-rb-troubleshoot-missing-route; a routing failure looks like a firewall drop from the far end.
The translation itself is wrong rather than the filtering. That is vyos-rb-troubleshoot-nat.
You are deliberately changing what the firewall permits. That is vyos-rb-configure-firewall-policy, at planned-change pace.

Blast radius

Reading counters, logs and conntrack is free. Three things in this runbook are not.

A packet capture on a busy interface competes with the packet path it is observing. Filter tightly and bound the count.

Deleting conntrack entries to force re-evaluation drops the flows that were using them. It is a small outage, not a read, and on a router carrying sessions people are using it will be noticed.

Changing a rule is the largest of the three, and its size depends entirely on which kind of change it is. Adding a rule affects the traffic that rule matches. Renumbering or reordering changes which rule fires for every flow the chain sees, including all the ones nobody thought to test.

Which configuration tree is this router on?

VyOS 1.3 keeps rule sets under firewall name and binds them to interfaces directly. VyOS 1.4 restructured this and 1.5 kept the new form: rule sets live under firewall ipv4 name, and they are reached through the forward, input and output filters rather than through a per-interface binding.

This matters here for a practical reason rather than a stylistic one. It changes the command that prints the counters, and it changes how a chain name in a log line maps back to something you can read. An operator running the 1.3-era command on a 1.5 router gets an error or an empty result and concludes the chain does not exist.

Confirm the tree with show configuration commands | match firewall before you start reading rules, and note which one you are on in the incident record.

Step 1: prove where the packet stopped

This is the step that decides whether the rest of the runbook applies, and it is the one most often skipped in favour of reading the rule set — which is always readable, always contains something plausible, and does not tell you whether the packet ever arrived.

Read-only / Safeingress and egress, during one reproduction

# Substitute your own values before running:
SRC=198.51.100.50
DST=203.0.113.10
PORT=443

sudo tcpdump -nn -c 20 -i eth0 "host $SRC and host $DST and port $PORT"

Run the same capture on the interface the traffic should leave by, during the same reproduction, and read the pair together:

Ingress	Egress	What it means
Nothing	Nothing	The packet never got here. Routing, addressing, or the link. Not a firewall incident
Request seen	Request seen	This router forwarded it. The fault is downstream, or it is the reply that is being lost
Request seen	Nothing	This router stopped it. Now the firewall is a suspect
Request seen, reply never returns	Request seen	The return path, not this direction. Different owner, different runbook

Only the third row makes this a firewall investigation. The first row is the one that saves the most time, because it ends the investigation in ninety seconds and moves it to the right team.

Step 2: read the counters as a delta

Read-only / Safethe rule that fires is the one that moves

show firewall statistics
show firewall ipv4 forward filter
show firewall ipv4 input filter

The per-rule view is a row per rule carrying the rule number, the action, a packet counter, a byte counter and the description — so a healthy WAN-facing chain reads roughly like this:

Rule	Action	Packets	Description
10	accept	1,845,234	Established and related
20	drop	12	Invalid state, logged
30	accept	432	SSH from the jump host
40	accept	234,982	HTTPS to the web server
9999	drop	847,213	Default deny

None of those numbers are evidence by themselves. A default-deny rule with 847,213 hits on an internet-facing chain is a router doing its job against background scanning, and it looks alarming to somebody reading it for the first time at three in the morning.

Read it, reproduce the failure, read it again, and work on the difference. The rule whose counter advances by your packet count is the rule that fired. If nothing advances anywhere in the chain, the packet is not reaching the chain — go back to step 1, because you have the wrong hook or the wrong interface.

Step 3: the log names the rule

Read-only / Safethe drop, in its own words

show log firewall

Aug 19 14:23:45 router1 kernel: [WAN-IN-9999-D] IN=eth0 OUT=eth1
SRC=198.51.100.50 DST=203.0.113.10 PROTO=TCP SPT=52012 DPT=443
WINDOW=65535 RES=0x00 SYN URGP=0

Illustrative output

The bracketed tag is the whole answer in one field: the chain, the rule number that fired, and the action. IN and OUT give the interfaces, which is how you confirm you were looking at the right hook. The rest is the packet.

Step 4: is this a rule, or a state judgement?

Some drops are not decided by a rule matching your traffic. They are decided by the router’s opinion of the flow, formed before any rule was consulted.

The one that matters here is the invalid state. A packet judged invalid is one the router has no connection-tracking entry for and cannot place in an existing conversation. On a healthy path that is background noise: scanners, stale sessions, packets arriving after their flow timed out.

When an invalid-state counter advances in step with your reported failure, it usually means something quite specific and quite different from a firewall misconfiguration: this router is seeing one direction of a flow whose other direction takes a different path. The request goes out through here and the reply comes back somewhere else, so from this router’s point of view the reply is a packet belonging to no conversation.

That is asymmetric routing. It is a routing design problem being reported accurately by the firewall, and the rules are not where it is fixed. Adding an accept rule to admit invalid packets does stop the symptom, and it also switches off the stateful behaviour that the rest of the policy depends on.

Step 5: conntrack, and the address the rule actually sees

Read-only / Safewhat the router thinks the conversation is

# Substitute your own values before running:
SRC=198.51.100.50
DST=203.0.113.10

sudo conntrack -L -s "$SRC" -d "$DST"
cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max

Two different findings come out of this and they need different fixes.

No entry where there should be one. The flow is not being tracked, so every packet after the first is judged as if it were new. Check whether the table is full — the count against the maximum — before assuming a rule is responsible.

An entry whose addresses are not the ones your rule matches. This is the NAT case, and it is the most reliably confusing failure in this runbook.

The filter hooks see the post-DNAT destination and the pre-SNAT source

The kernel applies destination NAT in PREROUTING, before the routing decision and before the forward filter runs. It applies source NAT in POSTROUTING, after the filter has already seen the packet.

For a port-forwarded inbound service, that means the firewall never sees the public address. By the time the forward filter runs, the destination has already been rewritten to the private one. A rule written against the public address — which is the address in the ticket, in the DNS entry and in the head of everyone discussing the incident — cannot match, will never fire, and will never appear in the log.

The traffic then falls through to the default action and is dropped silently, while a rule that reads exactly like the fix sits in the chain with a counter of zero.

Write the filter rule against the private address. Use conntrack -L to read both halves of the translation, and compare them against what the rule matches, before concluding the rule is wrong in any other way.

Step 6: deliberate, or accidental?

This is the decision the runbook exists for, and it comes before any edit.

A default-deny chain dropping traffic nobody explicitly permitted is not a fault. It is the firewall being correct. To the person whose traffic is being dropped it is indistinguishable from an outage, and the pressure in the incident channel is identical, which is exactly why the decision needs to be made explicitly rather than by reflex.

What the evidence shows	What it is	What to do
Default-deny fired, no rule permits this flow	Working as configured	A policy decision. Name the owner and get approval
An explicit rule permits it but a broader deny sits above it	Rule ordering	A real defect. Fix the order, then re-run the negative tests
The permitting rule exists and its counter is zero	It cannot match. Usually NAT, sometimes the wrong interface or zone	Fix what the rule matches, not what it permits
Invalid-state drops climbing with the failure	Asymmetric routing	Routing owns it. Do not disable state tracking to hide it
The drop began at a commit timestamp	Somebody changed it	Revert first, diagnose after

The one to be most careful with is the first row. Widening a rule so a page stops is a change to the statement of intent that the firewall exists to make. It is frequently the right answer, it is never the on-call operator’s answer alone, and the difference between those two sentences is what stops a temporary exception from becoming a permanent one that nobody can justify.

Step 7: change one rule, and test both directions of the question

Service impact possibleone rule, a timer, and two tests

configure
compare
commit-confirm 5
exit

Add the narrowest rule that admits the flow, with compare read against the approved change and commit-confirm sized to cover the reproduction. Then run both tests, not one:

the flow that failed now succeeds, with the same five-tuple on the same interfaces;
a flow that should still be denied is still denied.

The second test is the one that gets skipped, and it is the one that catches the change that fixed the reported problem by opening considerably more than the reported problem. From the reporter’s side those two outcomes are identical. They are told apart by somebody else, later, under worse circumstances.

Then check that the default-deny counter is still advancing. A catch-all that has stopped counting means something above it is now matching everything, and that is a much larger change than the one you intended.

Finally, remove the diagnostics. Turn the temporary logging back off, delete any trace rule, stop the captures. Trace rules and counter resets applied directly with nft are not in the configuration tree: rollback will not remove them, show configuration will not show them, and a reboot will silently discard them. Whatever you did outside the configuration tree, write it down.

Common patterns

Symptom	Likely cause	Where to look
Nothing in the ingress capture	The packet never arrived	Routing, addressing, the link. Not the firewall
Counters advance, log is silent	A rule fired without logging	Enable logging on the default action, reproduce, disable
Permitting rule exists, counter is zero	It cannot match the packet as it arrives	NAT translation, wrong interface, wrong zone or hook
Works from one source, not another	An earlier rule matches one and not the other	Two-sample counter delta for both sources
Works outbound, fails inbound	Only one direction of the zone matrix was written	The chain for the other direction
First packet passes, the rest do not	Connection tracking is not holding the flow	conntrack entry, table count against the maximum
Invalid-state drops track the failure	One direction of the flow goes elsewhere	The return path, then routing
Broke at a commit timestamp	The change, not the policy	`show configuration commit`, then `compare N`
Fixed by widening a rule, nobody can say why	The negative test was never run	Re-run it now, before the exception becomes permanent

Escalation

Escalate when:

The drop turns out to be deliberate — that is the policy owner’s decision.
The request is to admit inbound traffic from an untrusted network.
Invalid-state drops point at asymmetric routing.
The application depends on secondary connections the firewall cannot associate with the first.
The ingress capture shows the packet never arriving.
Two rule changes have failed to move the result.

Why this matters

Firewalls attract blame out of all proportion to how often they are wrong, and the reason is structural rather than technical: the firewall is the component in the path that has a name and an owner. Routing is diffuse, the application is somebody else’s, the far end is a company. The firewall is right there, it is configurable, and somebody on this call can change it.

That produces a specific and expensive failure. The operator reads the rule set — which is always readable and always contains something that looks relevant — finds a plausible rule, widens it, and the traffic starts working for a reason nobody has established. Sometimes the widened rule was the fix. Sometimes the reporter retried at the same moment. Either way the chain now carries a rule with no reason attached to it, and the next person to read it will either preserve it out of caution or remove it at a bad moment.

The counter-discipline is two questions asked in order, and neither requires knowing anything about the rule set.

Where did the packet stop? Two captures answer it, and a meaningful fraction of these incidents end there, because the packet never reached the router at all.

Was the drop deliberate? A default-deny chain doing exactly what it was written to do is not a defect, however much it looks like one from the far end. Treating it as a defect and widening the rule is how a firewall stops being a statement of intent and becomes a record of which outages were noisy enough to get an exception.

Everything else in this runbook — the counter deltas, the log tag, the conntrack tuple, the NAT ordering — exists to answer those two questions without guessing.