Security groups and the host firewall - two layers that fail differently

There are at least two firewalls between a client and a process on a cloud instance, and they are enforced in different places by different teams with different tooling.

The platform firewall - security group, network security group, VPC firewall rule - is enforced in the virtual network, outside the guest. The guest kernel never sees a packet the platform dropped, and nothing inside the guest can show you the rule that dropped it.

The guest firewall - nftables, or iptables through a compatibility layer - is enforced by the kernel on the instance. It is fully visible from inside, and completely invisible from the cloud console.

Almost all cloud network triage is the work of deciding which of the two is responsible, and the fastest way to decide is not to read either rule set.

The failure signatures differ

The two layers do not fail the same way, and the difference is audible from the client.

A timeout is the ambiguous one, and it is also the common one, because the platform firewall drops silently by design. So the question becomes: did the packet reach the guest kernel at all?

The one test that splits the layers

Run a capture on the instance and try the connection from the client.

# On the instance, on the interface facing the client
sudo tcpdump -ni any -c 20 'tcp port 8080'

The result decides everything:

• No packets at all. The platform never delivered them. The problem is the security group, a network ACL, a route table, or the client being somewhere you did not expect. The guest firewall is irrelevant and reading it is wasted time.
• SYN arrives, nothing goes back. The packet reached the kernel and the kernel dropped it. Now the guest firewall - or the absence of a listener - is the suspect.
• SYN arrives, RST goes back. Nothing is listening on that port, or a reject rule answered.
• SYN, SYN-ACK, and then the client still reports a timeout. The reply is being dropped on the way back. On a cloud network that is nearly always a stateless rule missing its return path, or asymmetric routing.

That last case is worth its own section.

Stateful and stateless rules

Security groups are stateful: allow an inbound flow and the return traffic is allowed automatically, because the platform tracks the connection. This matches how nftables conntrack works in the guest, and it is why most people never think about return traffic.

Network ACLs on AWS - and the equivalent subnet-level filters elsewhere - are stateless. Every packet is evaluated on its own, in both directions.

The classic failure: a subnet ACL allows inbound TCP 443, and its outbound rules allow only TCP 443. The request arrives. The reply leaves from source port 443 to the client’s ephemeral port, somewhere in the high range, and the outbound rule does not match it. The connection establishes far enough for tcpdump to show a SYN-ACK leaving, and then dies.

# The kernel's own ephemeral range, on the instance
cat /proc/sys/net/ipv4/ip_local_port_range

$ cat /proc/sys/net/ipv4/ip_local_port_range

32768	60999

That range is the client’s concern, not the server’s, and the client may be anything - a browser, another cloud, a load balancer. A stateless outbound rule must therefore allow the whole registered ephemeral range, conventionally 1024-65535, for return traffic. If that feels like it defeats the purpose, that is the correct reaction: stateless subnet filters are a coarse blast-radius control, not a substitute for a stateful rule set.

The guest side

The guest firewall is the layer you can actually read.

# Is anything listening, and on which address?
ss -ltnp

# The full rule set, as the kernel holds it
sudo nft list ruleset

# Counters, to see whether a rule is matching at all
sudo nft list ruleset -a

$ ss -ltn

State  Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0      4096       127.0.0.1:4222      0.0.0.0:*
LISTEN 0      4096       127.0.0.1:5432      0.0.0.0:*
LISTEN 0      4096         0.0.0.0:22        0.0.0.0:*

Read the Local Address column before you read a single firewall rule. In that output, port 5432 is bound to 127.0.0.1 - it is not reachable from anywhere else on the network no matter what any firewall says. A service bound to loopback is the most common cause of a “firewall problem” that has no firewall in it, and it is a one-line configuration fix rather than a rule change.

Prove the local path works before you blame the network:

curl -sS -m 3 -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/health

A base rule set for a cloud instance, where the platform firewall is doing the coarse work and the guest firewall is the backstop:

sudo nft -f - <<'EOF'
table inet filter {
  chain input {
    type filter hook input priority 0; policy drop;

    ct state established,related accept
    ct state invalid drop
    iif lo accept

    ip protocol icmp accept
    ip6 nexthdr ipv6-icmp accept

    tcp dport 22 accept
    tcp dport { 80, 443 } accept

    # counts everything the default policy is about to drop
    counter
  }
}
EOF

ct state established,related accept is what makes the rule set stateful; without it, outbound connections from the instance work but nothing ever comes back, and the symptom is an instance that cannot reach its package repository while ping works fine.

Rules that survive replacement

Instances are replaced, and replacement changes the address. Any rule that names an instance address is a rule with an expiry date on it, and the expiry is “the next deploy”.

• Reference a group, not an address. Every platform can express “allow from members of the application security group” rather than “allow from 192.0.2.17”. Membership follows the instance through its whole lifecycle; the address does not.
• Reference the subnet when you must use addresses. 198.51.100.0/24 for the application tier is stable in a way that 198.51.100.42 is not.
• Health checks do not come from your subnet. A managed load balancer probes from the platform’s own address ranges. Tighten a security group to “the application subnet only” and the health checks start failing, the load balancer marks every instance unhealthy, and the group replaces them - all of them, because they are all equally unhealthy. Allow the documented health-check source, or reference the load balancer’s own security group.

Knowledge check