Validating actual firewall behaviour

The firewall ruleset on disk is one view. The actual behaviour on the wire is another. Configuration mistakes, rule ordering bugs, and stale rules can all cause a firewall to behave differently than intended. The only way to be sure is to test.

Test from outside the host

# Substitute your own values before running:
HOST=server01.example.com

# Quick check
nc -vz "$HOST" 22
nc -vz "$HOST" 80
nc -vz "$HOST" 443
nc -vz "$HOST" 8080

# Comprehensive port scan
nmap -p- "$HOST"
nmap -sS -p 1-65535 "$HOST"

-p- scans all 65535 ports. -sS is a SYN scan (requires root on the scanning host).

Read the nmap output

PORT      STATE    SERVICE
22/tcp    open     ssh
80/tcp    open     http
443/tcp   open     https
3306/tcp  filtered mysql
8080/tcp  closed   http-proxy

States:

• open: SYN-ACK received. Port is listening.
• closed: RST received. Port is reachable but not listening.
• filtered: No response. Either firewall dropped or host is unreachable.

The goal of a default-deny policy is “filtered” for everything except explicitly allowed ports.

Service version detection

nmap -sV -p 22,80,443 <host>

Output:

PORT    STATE SERVICE VERSION
22/tcp  open  ssh     OpenSSH 9.6
80/tcp  open  http    nginx 1.24
443/tcp open  https   nginx 1.24

-sV probes the service to determine the version. Useful for verifying that the right service is exposed and for identifying outdated software.

Script scan

# Substitute your own values before running:
HOST=server01.example.com

nmap --script=ssl-enum-ciphers -p 443 "$HOST"
nmap --script=http-title -p 80,443 "$HOST"
nmap --script=vuln -p 80,443 "$HOST"

Nmap’s scripting engine (NSE) can probe for specific vulnerabilities and configuration issues. The vuln category checks for known CVEs.

Confirm rule precedence

If a rule is not behaving as expected, capture the wire traffic:

Two hosts, two roles, and getting them the wrong way round is what makes this test inconclusive. Capture on the host whose firewall you are validating; generate traffic from the host you are testing access from.

On the target — the host with the firewall:

# Substitute the address of the host you will connect FROM:
SOURCE=192.0.2.25

sudo tcpdump -i eth0 -nn -s0 "tcp port 22 and host $SOURCE"

From the source, in another terminal:

# Substitute the address of the host you are capturing ON:
TARGET=192.0.2.10

nc -vz "$TARGET" 22

Now read the capture. The pattern names the verdict:

That last row is the reason to capture on the target rather than the source. From the source, a DROP by the local firewall and a DROP three hops upstream look identical — both are a timeout. Only the target’s capture distinguishes “my rules did this” from “it never got here”, and those have completely different fixes.

-s0 captures full packets, which matters if you go on to inspect payloads; for this test the headers are enough.

Common validation scenarios

SSH should be allowed from management only:

# From management (should succeed)
nc -vz <host> 22

# From outside (should fail)
nc -vz <host> 22

If both succeed, the firewall is too permissive.

Database port should not be exposed:

# Substitute your own values before running:
HOST=server01.example.com

nmap -p 3306 "$HOST"    # MySQL
nmap -p 5432 "$HOST"    # PostgreSQL
nmap -p 27017 "$HOST"   # MongoDB

Should show “filtered” or “closed” for all.

Outbound should be allowed:

nc -vz 8.8.8.8 53
nc -vz example.com 443

Both should succeed from any host.

Build a regression test

For production, define a set of expected results and run them after every firewall change:

#!/bin/bash
# firewall-regression-test.sh

HOST=$1
FAILED=0

check_open() {
    if nc -z -w 3 $HOST $1 2>/dev/null; then
        echo "PASS: $HOST:$1 is open"
    else
        echo "FAIL: $HOST:$1 is open"
        FAILED=$((FAILED+1))
    fi
}

check_closed() {
    if nc -z -w 3 $HOST $1 2>/dev/null; then
        echo "FAIL: $HOST:$1 is closed (but is open)"
        FAILED=$((FAILED+1))
    else
        echo "PASS: $HOST:$1 is closed"
    fi
}

check_open 22
check_closed 23
check_open 80
check_closed 3306
check_closed 5432

if [ $FAILED -gt 0 ]; then
    echo "FAILED: $FAILED checks failed"
    exit 1
fi
echo "All checks passed"

Run this in CI after every firewall change. A failed test is a real failure.

Validate from inside the host

The host’s own firewall rules apply to incoming traffic. Inside the host:

ss -tlnp                                 # what is listening, and on which address
ss -tnp state syn-sent                   # our SYNs unanswered -> outbound drop
ss -tnp state syn-recv                   # their SYNs arrive but never complete
ss -tan state all '( sport = :443 )'     # every socket on one port, any state

There is no new state. ss filters on kernel TCP states, and the kernel has no such state — ss -tnp state new exits with ss: wrong state name: new. The half-open states are the ones that answer a firewall question, and they answer different halves of it:

• syn-sent accumulating means this host is sending SYNs that get no reply. The drop is outbound, or on the return path.
• syn-recv accumulating means SYNs are arriving and the handshake is not completing. The peer is reaching you; something is eating the SYN-ACK or the final ACK.

If the firewall is dropping packets but the service is listening, the issue is the firewall. If the service is not listening, the issue is the application. ss -tlnp distinguishes those two in one line, which is why it comes first.

Knowledge check