Network Metrics

A page load takes 4.2 seconds. The dashboard reports 40 percent CPU, 12 percent memory, healthy disks. The application panel says “request OK.” The user is staring at a loading spinner.

This is the failure shape node_network_* exists to catch. CPU, memory, disk, and application metrics can all be green while a host is in network distress. The interface counters are the ground truth.

What it is

node_network_* metrics are per-interface counters and gauges exposed by the node_exporter netdev collector. The collector reads /proc/net/dev on Linux once per scrape and emits each counter from the kernel’s struct net_device as a Prometheus counter. Each series carries a device label (eth0, ens5, bond0, vlan100) and the standard scrape labels (instance, job).

The metric family covers four operations: receive, transmit, errors, and drops. The pair a sysadmin should learn to read first is the receive and transmit bytes per second and the corresponding error and drop counters. Bytes-per-second tells you the volume. Errors and drops tell you whether the volume arrived intact.

The canonical alternative is SNMP polling on switches and routers (covered in the SNMP lesson). For hosts, node_network_* is the right approach because it is free, requires no agent on the device, and integrates with every other host metric in the same scrape job.

Why a sysadmin cares

Three production failure classes appear as green dashboards without these metrics:

• Ring buffer overruns. A NIC can receive packets faster than the kernel can drain them. The kernel ring buffer fills; packets are dropped before any socket sees them. The application reports nothing wrong because no socket read failed. The metric that catches this is node_network_receive_drop_total rising while node_network_receive_packets_total rises normally.
• Driver-level errors. A faulty SFP, a flaky cable, a misconfigured MTU. The driver reports frame, FIFO, or length errors back to the kernel. The application sees timeouts. node_network_receive_errs_total catches this.
• Bond or bridge saturation. A 10 Gbps link bonded from two 10 Gbps slaves can saturate if a single flow lands on one slave. The application sees microbursts of latency. The metric that catches it is per-slave traffic, which node_network_* exposes when the slaves are physical NICs.

These are not exotic failures. They are the most common network degradation shapes on a long-running Linux host. The metrics that catch them cost one scrape interval of CPU on the exporter.

How it works

The netdev collector reads /proc/net/dev once per scrape. The file is a snapshot of the kernel’s per-interface counter struct. The exporter formats those counters with the standard Prometheus counter naming convention (_total suffix) and emits them.

   /proc/net/dev         node_exporter        Prometheus
   -----------------     -------------        ----------
   Inter-|   Receive     netdev collector     node_network_
   face  |bytes packets  --->  /metrics ----->  _receive_bytes_total
   eth0  |  X       Y    one read per scrape    _transmit_bytes_total
   ens5  |  X       Y    no kernel calls        _receive_errs_total
   bond0 |  X       Y    no per-packet cost     _receive_drop_total
                                                 _transmit_errs_total
                                                 _transmit_drop_total

Two things to internalise:

1. Counters are monotonic. Each _total only goes up. The rate of change (rate(node_network_receive_bytes_total[1m])) is what dashboards plot. A flat line means a quiet interface, not a broken exporter, unless up{job="node"} is zero too.
2. The labels are limited. The default collector emits device and the scrape labels. Anything more (VLAN, VRF, container) needs the netclass collector or a custom script feeding the textfile collector.

How to configure it

Three decisions matter: which collector is enabled, which interface fields are exposed, and which labels are added.

1. The node_exporter collector set

node_exporter enables collectors via CLI flags. The default build enables netdev. Confirm with:

node_exporter --collectors.enabled=netdev,netclass \
  --collector.netclass.ignored-devices="^(veth|docker|br-).*"

The --collector.netclass.ignored-devices flag is the most frequently missed. By default, netclass emits a metric per /sys/class/net entry. On a host with two hundred veth* peer interfaces from containers, that is two hundred label values per metric. Whitelist or blacklist explicitly:

node_exporter \
  --collector.netclass.ignored-devices="^(veth|docker|br-).*" \
  --collector.netclass.ignore-infiniband=true

2. The scrape job

# /etc/prometheus/prometheus.yml
scrape_configs:
  - job_name: node
    scrape_interval: 30s
    scrape_timeout: 10s
    static_configs:
      - targets: ['node-a.internal:9100']
        labels:
          env: prod
          role: app

No special configuration is required. The exporter is the source of truth. Prometheus scrapes it. The metric set is what the binary exposes.

3. The recording rules

Two recording rules separate steady-state monitoring from alerting noise:

# /etc/prometheus/rules/network.yml
groups:
  - name: network.interface
    interval: 30s
    rules:
      - record: instance:node_network_receive_bytes:rate1m
        expr: rate(node_network_receive_bytes_total[1m])

      - record: instance:node_network_transmit_bytes:rate1m
        expr: rate(node_network_transmit_bytes_total[1m])

      - record: instance:node_network_drops:increase5m
        # Drops are rare. Alert only on a sustained non-zero increase.
        expr: |
          increase(node_network_receive_drop_total[5m])
          + increase(node_network_transmit_drop_total[5m])

The increase form on drops is deliberate. A counter that records zero increases should not fire an alert. Recording a single value across the fleet lets the alert rule read instance:node_network_drops:increase5m > 0.

How to validate it

Three layers must be confirmed: the exporter exposes the metrics, the metrics reflect real kernel state, and the recording rules produce sensible values.

# 1. The metrics are exposed. /metrics from the exporter.
curl -sf http://node-a.internal:9100/metrics \
  | grep -E '^node_network_(receive|transmit)_(bytes|errs|drop)_total'
# node_network_receive_bytes_total{device="eth0"} 8.741e+10
# node_network_transmit_bytes_total{device="eth0"} 4.123e+09
# node_network_receive_errs_total{device="eth0"} 0
# node_network_transmit_errs_total{device="eth0"} 0
# node_network_receive_drop_total{device="eth0"} 0
# node_network_transmit_drop_total{device="eth0"} 0

# 2. The values reflect kernel truth.
cat /proc/net/dev | awk 'NR>2 {gsub(":","",$1); print $1, $2, $3, $4, $5, $11}'
# eth0 87410000000 54120000 0 0 0
# ens5 12345678901 8000100 0 0 0

# Compare against the exporter values. They must agree within
# the scrape interval's worth of traffic.

# 3. PromQL produces sensible rate values.
up{job="node", instance="node-a.internal:9100"}
# 1
rate(node_network_receive_bytes_total{device="eth0"}[1m])
# {device="eth0", instance="node-a.internal:9100", job="node"} 1.842e+07

If the values disagree by more than the rate over one scrape interval, the scrape is missing samples or /proc/net/dev is being read on a different CPU than the one updating the counter. Both are rare. The simpler diagnosis is a stale exporter.

How it can fail

Six failure modes appear regularly. Each one is recognisable in the data.

1. Ring buffer overruns. node_network_receive_drop_total rises while bytes and packets also rise. The kernel ring cannot keep up with the wire rate. Symptom: TCP throughput plateaus even though the link is not at utilisation; ip -s link show eth0 shows overruns in the RX column.

2. Driver errors from a bad cable or SFP. node_network_receive_errs_total rises steadily. The link is up but frames are corrupt. Symptom: TCP retransmits spike (see the TCP lesson), ethtool -S eth0 | grep -i err shows CRC or symbol errors.

3. Bond imbalance. A bond0 made of two 10 Gbps slaves carries 14 Gbps. One slave carries 13 Gbps; the other carries 1 Gbps. Per-slave node_network_* shows the asymmetry; the bond itself looks fine. Symptom: latency brownouts at high traffic; the bond’s hash policy is wrong.

4. Netclass cardinality storm. --collector.netclass enabled with no filter on a container host. Two thousand veth* devices each emit a label. TSDB head block churns. Symptom: Prometheus memory grows steadily; the prometheus_tsdb_head_series metric rises by tens of thousands per hour.

5. Counter wrap silently ignored. A 64-bit counter wraps at 16 EiB. On a normal interface this never happens. On tun or dummy interfaces with synthetic traffic it can. Symptom: rate() produces a sharp negative spike; the Prometheus counter reset detection catches it but the dashboard spikes.

6. Interface disappears after scrape. A NIC is hot-plugged or rebonds. The exporter no longer emits node_network_* for that device. Prometheus times the series out after staleness_delta (default 5 minutes). Symptom: the panel reports “no data” rather than red; the alert on absent(node_network_receive_bytes_total) catches the disappearance if configured.

How to troubleshoot it

Order matters. Start at the boundary where evidence is most concrete.

1. Is the exporter up? curl http://host:9100/metrics. A missing node_exporter_build_info series means the exporter is not running. Check systemd or container logs.
2. Does the scrape flow? up{job="node"} in PromQL. If up=0, the problem is at the scrape boundary, not at the device.
3. Compare /proc/net/dev to the exporter. If the values disagree, the exporter is on a different version or the scrape interval is dropping samples. journalctl -u node_exporter for “scrape too slow” warnings.
4. Look at the rate, not the absolute value. rate() and irate() convert counters into evidence. A counter at 4.8e9 is meaningless; the rate tells you whether traffic is moving.
5. Inspect the device directly. ethtool -S eth0, ip -s link show eth0, tc -s qdisc show dev eth0. The kernel’s own counters are the source of truth. The exporter is a reader of those counters.
6. Check the label cardinality. count by (__name__) (node_network_*) shows the active series count. If it is in the millions, a collector is misconfigured.

Security implications

The netdev collector is read-only and exposes no credential. The risk surface is the same as any other node_exporter metric: network reachability to port 9100. Bind the exporter on the monitoring network, not the public interface.

The --collector.netclass flag walks /sys/class/net. On a container host, that walks every container’s veth. A careless whitelist exposes every peer’s MAC address as a label. Do not enable netclass on multi-tenant container hosts without an explicit blacklist.

The metrics themselves are not sensitive; interface names and traffic volumes are operational, not confidential. Treat firewall and DNS logs as sensitive (covered in the firewall and DNS lessons).

Performance implications

The netdev collector reads one /proc file per scrape. The cost is proportional to the number of interfaces and the scrape interval, not to the traffic volume. A 100-interface host scraped every 15 seconds spends a fraction of a millisecond per scrape in /proc/net/dev.

netclass is more expensive because it walks /sys/class/net. A 2,000-interface container host with netclass enabled pays a few milliseconds per scrape and emits 2,000 labels per metric. Whitelist aggressively.

Cardinality is the dominant cost. Each interface emits roughly ten metrics. A thousand interfaces is ten thousand series per scrape job. The label budget is bounded by the device list, not by traffic.

Production guidance

• Use the recording rule form instance:node_network_*:rate1m once. Query the recording rule from dashboards and alerts. Querying rate() on every panel is wasteful at fleet scale.
• Pair drops and errs with the TCP retransmit metric covered in the next lesson. The two together tell a coherent story; neither alone is sufficient evidence.
• Whitelist --collector.netclass.ignored-devices on container hosts before the first deploy. The series storm is silent until Prometheus runs out of memory.
• Alert on increase(..._drop_total[5m]) > 0, not on the rate directly. A burst of one packet is operationally interesting; a sustained rate below one per second is noise.
• Document the device label convention for the fleet. Bond names, VLAN tags, and bridge names diverge quickly without one.

Verification

You should now be able to answer:

• Which node_network_* metric catches kernel ring buffer overruns, and which catches driver-level errors?
• Why is a counter at value 4.8e9 by itself not a useful signal?
• What is the operational cost of enabling --collector.netclass on a 2,000-interface container host without a whitelist?
• When would SNMP polling be a better choice than node_network_*? (Hint: see the SNMP lesson.)
• What does an increase in node_network_receive_drop_total with no change in bytes-per-second typically indicate?

Quiz