Firewall Observability

At 04:00 the application on host-a cannot reach the database on host-b. The TCP retransmits are rising on host-a. The database is up. The application is up. The firewall between them is dropping the connection and not telling anyone. The host metrics do not show a network problem; the connection is not getting past the kernel’s netfilter layer.

This is the failure shape that firewall observability exists to catch. The drops are real. The signals are absent.

What it is

Firewall observability, in this lesson, is the combination of three distinct telemetry streams:

• Per-rule counters from nftables or iptables. Each rule in the active ruleset has a packet and byte counter. The counters are the ground truth for “is this rule actually doing what the runbook says it does?”
• Conntrack table state. The kernel’s connection tracker counts current entries and a configured maximum. The metric is the saturation gauge for stateful firewalls.
• Drop logs from nf_log or ulogd. When a packet is dropped, the kernel can emit a structured log event. Shipped to Loki, the events become a search index for “what was dropped, by which rule, at what time.”

The three streams are not interchangeable. The counters tell you what is being dropped. Conntrack tells you whether the kernel can keep state. The logs tell you which packet was dropped, with which rule, at what time. Production monitoring uses all three.

The canonical alternative is “read the ruleset when an incident happens.” Reading the ruleset is the right answer when investigating. It is the wrong answer as monitoring, because nothing is recorded, nothing is alerted, and the operator has to be awake.

Why a sysadmin cares

Three production failure classes appear as green dashboards without these streams:

• Silent drop. A new rule is added that blocks the application’s outbound connection. The TCP connection is reset; the application logs a “connection refused” error. The runbook says the rule allows the traffic. Symptom: the per-rule counter for the drop rule rises; the per-rule counter for the allow rule is unchanged.
• Asymmetric path. Return traffic is dropped because the firewall does not see the original packet on its table. Conntrack creates an entry for the original; the return packet matches the conntrack entry and is allowed. If the return packet arrives at a different firewall (because of a routing change), it is dropped. The metric that catches this is a rise in DROP counters on the firewall that should have allowed the return.
• Conntrack exhaustion. A misconfigured application opens thousands of connections per second and never closes them. The conntrack table fills. The kernel starts dropping new connections. The application sees “no buffer space available.” The metric that catches this is node_nf_conntrack_entries_percent_used rising toward 100.

Each of these failures is operationally important and silent in the standard host metrics. The cost of exposing them is a textfile collector script and a node_exporter flag. The value is minutes of investigation per incident.

How it works

iptables and nftables are user-space front-ends for the kernel’s netfilter subsystem. The kernel maintains counters per rule (and per chain, per table) in kernel memory. The counters are read by iptables -L -nvx and nft list ruleset style commands, not by /proc.

The conntrack table is exposed via:

• /proc/sys/net/netfilter/nf_conntrack_count — current number of entries.
• /proc/sys/net/netfilter/nf_conntrack_max — configured maximum.
• /proc/net/netfilter/nf_conntrack — per-entry state (large, usually avoided).

   nftables / iptables          script                 textfile collector
   -------------------          ------                 -------------------
   per-rule counters   --->    nft list ruleset   --->  /var/lib/node_exporter/
   in kernel memory              iptables -L -nvx       firewall.prom
                                 every 15 s
                                       |
                                       v
                                 node_exporter reads the .prom file
                                 at scrape time and emits the metrics
                                       |
                                       v
                                 nftables_rule_packets_total{chain="input",rule="allow_ssh"} 412
                                 iptables_rule_packets_total{chain="INPUT",rule="DROP"} 8

   conntrack table             node_exporter
   --------------               -------------
   /proc/sys/net/netfilter/ --->  conntrack collector
   nf_conntrack_count             node_nf_conntrack_entries
   nf_conntrack_max               node_nf_conntrack_entries_limit
                                  node_nf_conntrack_entries_percent_used

Two things to internalise:

1. The counters are kernel-side. Neither iptables nor nftables resets them between reads. The exporter side must track the previous value and emit Prometheus counters (with _total suffix), or use the gauge value with a separate rate computation in PromQL.
2. The conntrack collector is built into node_exporter. node_nf_conntrack_entries and friends are exposed directly without an external script.

How to configure it

Three layers matter: the conntrack collector on the exporter, the textfile collector for the per-rule counters, and the recording rules for alerting.

1. The node_exporter textfile collector

# /etc/default/node_exporter
ARGS="--collectors.enabled=conntrack,textfile \
      --collector.textfile.directory=/var/lib/node_exporter"

The textfile directory is read-only from node_exporter’s perspective. A cron job (or systemd timer) writes the metrics there.

2. The script that emits the per-rule counters

#!/usr/bin/env bash
# /usr/local/bin/firewall_metrics.sh
# Emit nftables per-rule counters as Prometheus metrics.
set -euo pipefail

OUT=/var/lib/node_exporter/firewall.prom
TMP=$(mktemp)
trap 'rm -f "$TMP"' EXIT

{
  echo '# HELP nftables_rule_packets_total Packets matched by nftables rule.'
  echo '# TYPE nftables_rule_packets_total counter'

  # nftables: chain, handle, comment (used as rule label)
  nft -j list ruleset \
    | jq -r '
        .nftables[]?
        | select(.rule != null)
        | .rule as $r
        | ($r.family // "ip") as $family
        | ($r.table // "filter") as $table
        | ($r.chain // "unknown") as $chain
        | ($r.expr | map(select(.counter)) | .[0].counter.packets) as $packets
        | ($r.expr | map(select(.counter)) | .[0].counter.bytes) as $bytes
        | ($r.comment // "uncommented") as $comment
        | "nftables_rule_packets_total{family=\"\($family)\",table=\"\($table)\",chain=\"\($chain)\",comment=\"\($comment)\"} \($packets // 0)\n
           nftables_rule_bytes_total{family=\"\($family)\",table=\"\($table)\",chain=\"\($chain)\",comment=\"\($comment)\"} \($bytes // 0)"
      '

  echo '# HELP iptables_rule_packets_total Packets matched by iptables rule.'
  echo '# TYPE iptables_rule_packets_total counter'

  # iptables: chain, rule number, policy
  iptables -L -nvx \
    | awk 'NR>2 && $1 != "" {
        chain=$2; packets=$1; bytes=$3;
        printf("iptables_rule_packets_total{chain=\"%s\"} %s\n", chain, packets);
        printf("iptables_rule_bytes_total{chain=\"%s\"} %s\n", chain, bytes);
      }'

} > "$TMP"
mv "$TMP" "$OUT"

The script writes a .prom file in the textfile directory. node_exporter picks it up at the next scrape.

3. The systemd timer

# /etc/systemd/system/firewall-metrics.service
[Unit]
Description=Export firewall counters for node_exporter
[Service]
Type=oneshot
ExecStart=/usr/local/bin/firewall_metrics.sh

# /etc/systemd/system/firewall-metrics.timer
[Unit]
Description=Run firewall-metrics every 15 seconds
[Timer]
OnBootSec=15s
OnUnitActiveSec=15s
AccuracySec=1s
[Install]
WantedBy=timers.target

The 15-second cadence is independent of the scrape interval. node_exporter reads whatever file is current at scrape time.

4. The Prometheus scrape job

# /etc/prometheus/prometheus.yml
scrape_configs:
  - job_name: node
    scrape_interval: 30s
    scrape_timeout: 10s
    static_configs:
      - targets: ['host-a.internal:9100']
        labels:
          role: app

The textfile collector and the conntrack collector are picked up automatically once the flags are passed.

5. The recording rules

# /etc/prometheus/rules/firewall.yml
groups:
  - name: firewall.saturation
    interval: 30s
    rules:
      - record: host:conntrack_entries:percent_used
        expr: |
          100 * node_nf_conntrack_entries
          / clamp_min(node_nf_conntrack_entries_limit, 1)

      - record: host:nftables_drop_packets:increase5m
        expr: |
          sum by (chain, comment) (
            increase(nftables_rule_packets_total{comment=~".*[Dd]rop.*"}[5m])
          )

      - record: host:iptables_drop_packets:increase5m
        expr: |
          sum by (chain) (
            increase(iptables_rule_packets_total{chain="INPUT"}[5m])
            + increase(iptables_rule_packets_total{chain="OUTPUT"}[5m])
            + increase(iptables_rule_packets_total{chain="FORWARD"}[5m])
          )

The clamp_min guards against divide-by-zero when the limit file is unreadable. The increase over five minutes is the operationally interesting window; a single dropped packet is the signal worth surfacing.

How to validate it

Three layers must be confirmed: the conntrack collector emits, the textfile collector ingests, and the ruleset is actually changing.

# 1. The conntrack collector emits.
curl -sf http://host-a.internal:9100/metrics | grep -E '^node_nf_conntrack_'
# node_nf_conntrack_entries 412
# node_nf_conntrack_entries_limit 65536
# node_nf_conntrack_entries_percent_used 0.6

# 2. The textfile collector ingests the firewall metrics.
curl -sf http://host-a.internal:9100/metrics | grep -E '^nftables_rule_'
# nftables_rule_packets_total{chain="input",comment="allow_ssh"} 412
# nftables_rule_packets_total{chain="input",comment="drop_invalid"} 8

# 3. The rule counters agree with nft list ruleset.
nft list ruleset | grep -A 1 'allow_ssh'
#     counter packets 412 bytes 28900

If the exporter value and the nft output agree, the metric is correct. If they disagree by a small amount, the script ran during a packet and the difference is real. If they disagree by a large amount, the script is not running or the counter has been reset.

How it can fail

Six failure modes appear regularly. Each one is recognisable in the data.

1. Script never runs. The textfile collector sees no file and emits no metrics. Symptom: nftables_rule_* is absent from /metrics; only node_nf_conntrack_* is present. Check the systemd timer: systemctl status firewall-metrics.timer.

2. Script runs as the wrong user. The script cannot read nftables state or iptables counters. Symptom: the script fails silently; the .prom file is empty or missing metrics. Check the systemd journal for “permission denied”.

3. Counter reset on rule change. A rule is reloaded; the counters reset. Symptom: a sharp drop in nftables_rule_packets_total; increase() over a window that crosses the reload produces a negative value. Prometheus detects the reset and emits a marker.

4. Conntrack table fills silently. nf_conntrack_max is reached; new connections are dropped with no metric indicator. Symptom: node_nf_conntrack_entries_percent_used sits at 100 for several minutes; the application reports “no buffer space available.”

5. DROP log not shipped. nf_log is enabled but the structured events are not collected. Symptom: an incident shows the drop in counters but the operator cannot find which rule dropped which packet. Check Loki: {job="nf_log"} for the affected host.

6. Asymmetric path drops. A return packet arrives at a different firewall than the original. Symptom: the DROP counter on the receiving firewall rises for traffic that was emitted by a host on the same VPC. The originating host sees a TCP RST; the application reports a connection failure.

How to troubleshoot it

Order matters. Start at the boundary where evidence is most concrete.

1. Is the conntrack collector running? curl http://host:9100/metrics | grep nf_conntrack. If absent, the collector is not enabled.
2. Does the textfile collector see the file? ls -l /var/lib/node_exporter/. A missing or zero-byte firewall.prom means the script did not run or failed.
3. Did the script run recently? systemctl status firewall-metrics.timer and journalctl -u firewall-metrics for the last execution.
4. Do the counters match nft list ruleset? The two must agree within the script’s run interval. A large disagreement means the script is not running.
5. Is the conntrack table full? node_nf_conntrack_entries_percent_used > 90 is the signal; dmesg | grep conntrack for the kernel’s own message.
6. Are the drop logs being shipped? {job="nf_log"} in LogQL for the affected host. A missing entry means the collection pipeline is broken.

Security implications

The per-rule counters reveal which rules are matching traffic. On a multi-tenant host, that can reveal which services are running and which ports are in use. Treat the metrics endpoint as operationally sensitive; bind it on the monitoring network.

The textfile script runs as root (or with CAP_NET_ADMIN) to read the ruleset. The output is written to a directory node_exporter reads. The directory must be writable only by the script’s user and readable only by the exporter. A misconfigured permission allows an unprivileged user to inject metrics.

The drop logs contain source and destination IPs, ports, and sometimes payload fragments. Ship them to Loki with redaction. A drop log that contains user data is a privacy incident; configure the log format to omit payload bytes.

The conntrack table itself can be a privacy concern. Each entry maps a tuple to a state. A dump of /proc/net/netfilter/nf_conntrack is a record of every connection the host has made. Do not expose this file beyond the exporter’s user.

Performance implications

The textfile collector reads a file at scrape time. The cost is proportional to the size of the file. A ruleset with a thousand rules produces a few hundred lines; the cost is sub-millisecond.

The script runs every 15 seconds and reads the kernel’s ruleset. On a busy firewall with thousands of rules, the script can take 100 ms; this is a small fraction of the host CPU. Run it at a cadence slower than the scrape interval.

The conntrack collector reads two small files per scrape. The cost is sub-millisecond.

Cardinality is the dominant cost. The textfile metrics are labelled by chain, comment, and table. A ruleset with many rules and many comments produces many label values. Whitelist the labels that matter for alerts.

Production guidance

• Treat the firewall ruleset as deployed code. The same review and CI process that covers application code covers firewall rules.
• Use the conntrack collector directly; do not write a script to duplicate the metrics.
• Alert on node_nf_conntrack_entries_percent_used > 90. The table fills before the kernel starts dropping connections; the alert gives time to drain.
• Use rate() and increase() on the per-rule counters. A rule that drops 100 packets a day for months is the baseline; the alert reads the spike.
• Ship drop logs to Loki. The counters tell you what is dropping; the logs tell you which packet and which rule.
• Document the textfile script convention. The directory, the file naming, and the run cadence must be consistent across hosts; otherwise dashboards cannot aggregate.

Verification

You should now be able to answer:

• Which two telemetry sources expose per-rule drop counts? (Hint: counters and logs.)
• What is the operational difference between per-rule and per-chain counters?
• How do you detect conntrack saturation before the kernel starts dropping connections?
• Why does a textfile collector script need to handle counter resets across rule reloads?
• Which three firewall failure shapes are most common in production, and what metric catches each?

Quiz