Network Metrics

A database cannot reach its replication peer. The application panel says “connection refused” half the time. The host panel shows the eth0 throughput at 4 Gbps on a 10 Gbps NIC, no errors, no drops. The on-call closes the panel. An hour later, the same outage. The cause is a misconfigured MTU on the peer side: packets larger than the path MTU are being silently dropped, and node_network_* does not surface the drop counter for ICMP “needs frag” replies. This lesson is about reading node_network_* so that this kind of intermittent failure is not invisible.

What it is

The netdev collector reads /proc/net/dev and emits a gauge for each (device, counter) pair. The full set in node_exporter 1.8.x:

# Byte and packet counters (RX = receive, TX = transmit).
node_network_receive_bytes_total{device="eth0"}      9.87e+11
node_network_transmit_bytes_total{device="eth0"}     4.5e+10
node_network_receive_packets_total{device="eth0"}    7.6e+08
node_network_transmit_packets_total{device="eth0"}   4.3e+07

# Errors, drops, frame errors.
node_network_receive_errs_total{device="eth0"}       0
node_network_transmit_errs_total{device="eth0"}      0
node_network_receive_drop_total{device="eth0"}        0
node_network_transmit_drop_total{device="eth0"}       0
node_network_receive_fifo_total{device="eth0"}        0
node_network_transmit_fifo_total{device="eth0"}       0
node_network_receive_frame_total{device="eth0"}       0
node_network_receive_compressed_total{device="eth0"}  0
node_network_transmit_compressed_total{device="eth0"} 0
node_network_receive_multicast_total{device="eth0"}   1.2e+06
node_network_transmit_carrier_total{device="eth0"}    0
node_network_receive_csum_errors_total{device="eth0"} 0
node_network_mtu_bytes{device="eth0"}                 1500

The cardinalities scale with the number of interfaces. A host with 4 physical NICs, 1 bond, 2 VLANs, and 1 bridge has about 8 interfaces * 16 metrics = 128 series. Container hosts have hundreds of veth interfaces — the reason the device- exclude flag exists.

The netclass collector (separate, also enabled by default) emits:

node_network_info{device="eth0",address="aa:bb:cc:dd:ee:ff",
  broadcast="ff:ff:ff:ff:ff:ff",duplex="full",ifindex="2",
  operstate="up",speed="10000"} 1
node_network_speed_bytes{device="eth0"} 1.25e+09
node_network_mtu_bytes{device="eth0"} 1500
node_network_up{device="eth0"} 1

The operstate label carries up, down, unknown, dormant, notpresent, lowerlayerdown. The speed is in bytes per second (the kernel reports megabits; node_exporter multiplies by 125000).

Throughput, errors, drops

Three derivations cover most operational questions:

# Throughput per device, bytes per second.
rate(node_network_receive_bytes_total[5m])
  + rate(node_network_transmit_bytes_total[5m])

# Error rate (any kind), per second.
rate(node_network_receive_errs_total[5m])
  + rate(node_network_receive_drop_total[5m])
  + rate(node_network_transmit_errs_total[5m])

# Per-interface utilisation as a fraction of link speed.
(rate(node_network_receive_bytes_total[5m])
  + rate(node_network_transmit_bytes_total[5m]))
  / node_network_speed_bytes

The third is the saturation gauge. A value of 0.7 means the NIC is at 70% of its rated link speed.

Excluding lo, veth, and other noise

A host with a default node_exporter configuration exposes counters for every interface in /proc/net/dev. On a container host that is hundreds of veth, docker, br-*, and lo interfaces. The dashboard becomes unreadable. The fix is the same pattern as for diskstats:

--collector.netdev.device-exclude=^(veth.*|docker.*|br-.*|lo)$
--collector.netclass.ignored-devices=^(veth.*|docker.*|br-.*|lo)$

The regex drops virtual interfaces. The result on a Kubernetes node is that the dashboard shows the two physical NICs (or the bond), the VLAN interfaces, and the bridge — the interfaces the operator actually configures.

A common typo is ^(veth*)$ (missing the dot). The result is that veth interfaces come back and the panel is unreadable. Verify with a quick check:

# READ-ONLY
# After the systemd unit reloads, the device list should be small.
curl -sf http://localhost:9100/metrics \
  | grep '^node_network_receive_bytes_total' \
  | awk -F'device="' '{print $2}' | awk -F'"' '{print $1}' | sort

A production host should expose 1–5 interfaces. If the list has dozens, the regex is wrong.

Link state and carrier transitions

node_network_up{device="eth0"} returns 1 if the interface is in the up operstate, 0 otherwise. A flapping link shows the metric bouncing between 1 and 0.

The node_network_info metric carries the operstate label and the speed label. A speed of 0 on a wired NIC means the link is down or the driver did not negotiate a speed.

node_network_transmit_carrier_total increments on each carrier-loss event. A non-zero value on a stable link is a signal of a flaky cable, switch port, or SFP module. The counter is monotonic — to find recent activity, use increase(...).

# Carrier transitions in the last hour.
increase(node_network_transmit_carrier_total[1h]) > 0

# Interfaces currently up.
node_network_up == 1

The first is the alertable signal. The second is the panel.

MTU mismatches

node_network_mtu_bytes returns the MTU configured on the interface. The metric itself does not detect a mismatch between two endpoints — that requires comparing the metric on both ends. The mismatch manifests as silent drops:

• A packet larger than the path MTU cannot be forwarded.
• The sender sends an ICMP “needs frag” back to the source.
• The source reduces the packet size and retries.

The retry cycle adds latency. node_network_* does not expose the drop counter for ICMP “needs frag” silently dropped at the next hop. The signal that catches this:

• node_network_receive_frame_total increases — frames failed CRC on receive (often link-quality issues).
• node_network_mtu_bytes shows different values on two hosts that should match.
• The application latency rises while the throughput gauge is unchanged.

NIC ring buffer drops

The NIC ring buffer is a small FIFO inside the NIC itself. When the kernel cannot drain the ring fast enough, packets are dropped at the hardware level — not by the kernel. The metric that catches this is node_network_receive_fifo_total (receive FIFO overruns).

The ethtool collector (opt-in, requires --collector.ethtool) exposes more detail:

node_ethtool_rx_bytes_total{device="eth0"} 9.87e+11
node_ethtool_tx_bytes_total{device="eth0"} 4.5e+10
node_ethtool_rx_packets_total{device="eth0"} 7.6e+08
node_ethtool_rx_crc_errors_total{device="eth0"} 0
node_ethtool_rx_missed_total{device="eth0"} 0
node_ethtool_rx_frame_errors_total{device="eth0"} 0

rx_missed_total is the NIC-level miss counter (often the ring-buffer drop). rx_crc_errors_total is the CRC error counter (link quality). These are more granular than the netdev metrics.

The trade-off: ethtool shells out to ethtool -S for every NIC on every scrape. On a host with 20 NICs (many virtual functions), this is hundreds of milliseconds. Use it where the operational value justifies the cost — usually on critical-path hosts.

Bonded NICs

A bonded interface combines two physical NICs into one logical interface (bond0). Two metrics must be reconciled:

• node_network_* on eth0, eth1 — the physical NICs.
• node_network_* on bond0 — the logical interface.

The counters on eth0 and eth1 are the actual throughput. The counters on bond0 are the aggregated throughput after the bonding driver has decided which slave gets each frame.

The aggregation rules depend on the bonding mode:

• mode=0 (balance-rr). Frames are round-robined across slaves. bond0 throughput equals the sum of slave throughput. The drops on slaves are the per-slave drops, not aggregated drops.
• mode=1 (active-backup). Only one slave is active; the others are standby. bond0 throughput equals the active slave’s throughput. The other slaves should show zero or near-zero counters.
• mode=4 (802.3ad, LACP). Frames are load-balanced by hash. bond0 throughput equals the sum of slave throughput (assuming both slaves are active). Drops on slaves are independent.

The operational rule:

• Panel the slave interfaces (eth0, eth1) for per-NIC errors and drops. Drops at the slave level are not visible on the bond.
• Panel the bond (bond0) for the aggregate throughput.
• Alert on drops at the slave level, not at the bond level — the bond hides them.
• For mode=1, alert when the active slave changes (the bond_slave state in the kernel).

The PromQL patterns:

# Per-slave drops in the last 5 minutes.
increase(node_network_receive_drop_total[5m]) > 0
  or increase(node_network_transmit_drop_total[5m]) > 0

# Bond throughput.
rate(node_network_receive_bytes_total{device="bond0"}[5m])
  + rate(node_network_transmit_bytes_total{device="bond0"}[5m])

# Slave utilisation, which should match bond throughput in LACP.
rate(node_network_receive_bytes_total{device=~"eth[01]"}[5m])
  + rate(node_network_transmit_bytes_total{device=~"eth[01]"}[5m])

The third query returns the aggregate slave throughput. In LACP mode this should be close to the bond throughput. A large difference indicates the bonding driver is dropping frames or the slaves are not all active.

How to configure it

The netdev and netclass collectors are enabled by default. The flags to tune:

--collector.netdev.device-exclude=^(veth.*|docker.*|br-.*|lo)$
--collector.netclass.ignored-devices=^(veth.*|docker.*|br-.*|lo)$
--collector.ethtool   # opt-in; required for NIC-level counters

The recording rules:

# /etc/prometheus/rules/network.yml
groups:
  - name: network
    interval: 30s
    rules:
      - record: instance:nic_throughput_bytes:rate5m
        expr: sum by (instance, device) (
            rate(node_network_receive_bytes_total[5m])
          + rate(node_network_transmit_bytes_total[5m])
        )

      - record: instance:nic_utilisation_ratio:rate5m
        expr: instance:nic_throughput_bytes:rate5m
          / on (instance, device) node_network_speed_bytes

      - record: instance:nic_errors:rate5m
        expr: sum by (instance, device) (
            rate(node_network_receive_errs_total[5m])
          + rate(node_network_receive_drop_total[5m])
          + rate(node_network_receive_fifo_total[5m])
          + rate(node_network_transmit_errs_total[5m])
        )

      - record: instance:nic_carrier_transitions:increase1h
        expr: increase(node_network_transmit_carrier_total[1h])

The PromQL patterns the on-call uses:

# Top 10 NICs by utilisation.
topk(10, instance:nic_utilisation_ratio:rate5m)

# NICs with errors in the last hour.
instance:nic_errors:rate5m > 0

# Carrier transitions in the last hour.
instance:nic_carrier_transitions:increase1h > 0

# Currently down interfaces.
node_network_up == 0

How to validate it

# READ-ONLY
# 1. The collector is enabled.
curl -sf http://localhost:9100/metrics | grep '^# HELP node_network_receive_bytes_total'

# 2. Only real interfaces are exposed.
curl -sf http://localhost:9100/metrics \
  | grep '^node_network_receive_bytes_total' \
  | awk -F'device="' '{print $2}' | awk -F'"' '{print $1}' | sort

# 3. Cross-check against /proc/net/dev.
awk '/:/ {gsub(/:/, ""); print $1}' /proc/net/dev | sort

# 4. The netclass info is exposed.
curl -sf http://localhost:9100/metrics | grep '^node_network_info'

# 5. The operstate label is correct.
curl -sf http://localhost:9100/metrics \
  | grep '^node_network_info' \
  | awk -F'operstate="' '{print $2}' | awk -F'"' '{print $1}'

Expected for a 10-Gbps database host:

$ curl -sf http://localhost:9100/metrics | grep '^node_network_receive_bytes_total'
node_network_receive_bytes_total{device="eth0"} 9.87e+11
node_network_receive_bytes_total{device="bond0"} 9.87e+11
$ curl -sf http://localhost:9100/metrics | grep '^node_network_info'
node_network_info{device="bond0",operstate="up",speed="10000"} 1

If bond0 shows speed="0", the bonding driver has not negotiated a link speed. Check the slaves.

How it can fail

1. veth interfaces dominate the panel. Symptom: 200 series per host, mostly container traffic. Cause: device-exclude regex missing. Fix: add ^(veth.*|docker.*|br-.*|lo)$ to both netdev and netclass.
2. Bond drops invisible. Symptom: a slave NIC has drops, but the bond shows zero. Cause: the drops are at the slave level; the bonding driver does not aggregate them. Fix: panel the slaves; alert on slave drops.
3. MTU mismatch undetectable from a single host. Symptom: connection is slow to a peer; both endpoints show healthy NICs. Cause: the MTU values differ between the two hosts. Fix: panel node_network_mtu_bytes and compare across hosts that should match.
4. Carrier transitions buried in uptime. Symptom: a cable is flaky; the link stays “up” most of the time; node_network_up rarely shows 0. Cause: short outages are missed by a 15s scrape. Fix: alert on increase(node_network_transmit_carrier_total[1h]) > 0.
5. Speed reported as 0. Symptom: node_network_speed_bytes = 0 on an active interface. Cause: the driver does not expose the speed, or the link is down. Fix: check operstate; if up and speed=0, the driver is misconfigured.
6. VLAN interface drops invisible on the parent. Symptom: a VLAN is dropping frames; the parent NIC shows clean. Cause: VLAN drops are at the VLAN sub-interface. Fix: panel the VLAN interface (eth0.100).

Security implications

Network metrics are moderate-risk:

• They reveal interface names, MAC addresses, IP addresses (in some configurations), and link speeds. On a shared host that is reconnaissance.
• The node_network_info carries the device MAC address in the address label. MAC addresses are not secrets but they are identifying.
• The node_network_mtu_bytes and operstate reveal network configuration. Low risk.

The standard mitigations apply: bind to a private interface, restrict with a firewall, enable auth on the listener.

A deeper risk is that an attacker on the same network can correlate the throughput metrics to infer workload patterns. The fix is operational, not technical — restrict who can read the metrics endpoint.

Performance implications

The netdev collector reads /proc/net/dev once per scrape. It is cheap — single milliseconds.

The ethtool collector (opt-in) shells out to ethtool -S and ethtool -i for every NIC. On a host with 20 NICs this is hundreds of milliseconds. The trade-off: richer metrics at higher cost.

The netclass collector walks /sys/class/net/. On a container host with thousands of veth interfaces, this is the dominant cost in node_exporter scrape duration. The ignored-devices regex bounds it.

Production guidance

• Canonical excludes in version control.
• Recording rules for throughput, utilisation, errors, and carrier transitions.
• Alerts:
  • nic_utilisation_ratio > 0.85 for 5m — early warning.
  • nic_utilisation_ratio > 0.95 for 1m — page.
  • nic_errors > 0 for 5m — page; errors indicate a link or driver problem.
  • increase(nic_carrier_transitions[1h]) > 0 — page; flaky cable or SFP.
• For bonded NICs, panel the slaves and the bond. Drops at the slave level are the operational signal.
• For MTU, panel node_network_mtu_bytes and document the expected MTU per network segment.
• The ethtool collector is opt-in for a reason. Enable it on critical-path hosts; leave it off on the rest.

Verification

You should now be able to answer:

• How do you derive NIC throughput from node_network_* counters, and why is rate() required?
• What does node_network_transmit_carrier_total measure, and what does a non-zero value indicate?
• Why does a bonded interface hide per-slave drops, and how do you reconcile the slave and bond counters?
• What is the relationship between MTU mismatches and network drops, and how do you detect a mismatch?

Quiz