Network Observability

A backend service is timing out on requests to a peer service. The network “panel” on the host overview shows throughput at 12% of the link capacity. The on-call engineer concludes the network is fine. The application is in fact waiting on TCP retransmits caused by a duplex mismatch on the switch port. The link is up, the counters are incrementing, but the driver is dropping packets at the rate of one per second. The throughput metric is the wrong question; the drop and error counters are the right ones.

Network observability on Linux is the discipline of measuring four quantities per interface: bytes (the throughput question), packets (the load question), errors and drops (the quality question), and carrier (the “is the link actually up” question). The metrics are emitted by node_exporter for every interface the kernel sees, which is hundreds on a Kubernetes node and the reason the production deployment filters the interface label aggressively.

What it is

Network observability on Linux is the practice of exposing the per-interface counters published by /proc/net/dev, augmented by sysfs-derived carrier and operational-state gauges:

• Throughput - node_network_receive_bytes_total and node_network_transmit_bytes_total, divided by the scrape interval.
• Packet rate - node_network_receive_packets_total and node_network_transmit_packets_total.
• Errors - node_network_receive_errs_total, node_network_transmit_errs_total.
• Drops - node_network_receive_drop_total, node_network_transmit_drop_total.
• Carrier - node_network_carrier (read from /sys/class/net/<iface>/carrier).
• Operational state - node_network_up (read from /sys/class/net/<iface>/operstate).

The speed metric is read from sysfs and is the link’s nominal capacity. The percentage utilisation is the byte rate divided by the link speed. The interface label is the axis on which the operator pivots.

Why a sysadmin cares

The silent interface is the most operationally expensive network failure mode. The kernel keeps the interface “up” while the driver buffers and drops packets. The TCP stack retransmits. The application sees latency. The dashboard reports the link as up and the throughput as nominal. The on-call engineer is forced into the error and drop counters, which are typically not on the host overview.

A second reason: per-interface filtering is the difference between a dashboard that loads in 200 ms and a dashboard that times out. A Kubernetes node has hundreds of veth interfaces; a docker host has dozens. The node_network_* metrics are emitted per interface; without a label filter, the Prometheus query scans thousands of series. The discipline is to filter on the production interface (eth0, ens5, eno1, bond0) and to exclude the virtual ones.

A third reason: the network metrics answer three questions the host overview does not answer. Drops and errors are the question of “is the link healthy?” Carrier is the question of “is the link physically connected?” Operational state is the question of “does the kernel agree the link is usable?” The three answers together are the production-graded network picture.

How it works

The Linux kernel publishes per-interface counters in /proc/net/dev. The fields and the node_exporter mappings:

  /proc/net/dev field        node_exporter metric
  -----------------------    ----------------------------------------
  receive bytes              node_network_receive_bytes_total
  receive packets            node_network_receive_packets_total
  receive errs               node_network_receive_errs_total
  receive drop               node_network_receive_drop_total
  receive fifo               node_network_receive_fifo_total
  receive frame              node_network_receive_frame_total
  receive compressed         node_network_receive_compressed_total
  receive multicast          node_network_receive_multicast_total
  transmit bytes             node_network_transmit_bytes_total
  transmit packets           node_network_transmit_packets_total
  transmit errs              node_network_transmit_errs_total
  transmit drop              node_network_transmit_drop_total
  transmit fifo              node_network_transmit_fifo_total
  transmit colls             node_network_transmit_colls_total
  transmit carrier           node_network_transmit_carrier_total

  /sys/class/net             node_exporter metric
  -----------------------    ----------------------------------------
  carrier                    node_network_carrier
  operstate                  node_network_up
  speed                      node_network_speed_bytes
  mtu                        node_network_mtu_bytes
  duplex                     node_network_duplex

The interface label is the axis on which the operator pivots. The fields are cumulative counters; the application of rate() produces the per-second metric.

The four most important production metrics:

                Network Health (USE: errors + saturation)
                ==========================================

   rx_errors   = rate(node_network_receive_errs_total[1m])
               - the rate of receive errors. Non-zero means
                 frame errors, CRC errors, or driver-side
                 failures. Threshold: 0.1 errors per second.

   rx_drops    = rate(node_network_receive_drop_total[1m])
               - the rate of receive drops. The kernel or the
                 driver gave up on the packet. Threshold: 0.1
                 drops per second. Drops during traffic spikes
                 indicate a saturated ring buffer.

   tx_carrier  = rate(node_network_transmit_carrier_total[1m])
               - the rate of carrier loss. Non-zero means the
                 link went down. Always an incident.

   speed       = node_network_speed_bytes
               - the link's nominal capacity. The byte rate
                 divided by speed is the utilisation percentage.
                 100% util on a 1 Gbps link is the saturation
                 warning; 100% util on a 100 Gbps link is fine.

The node_network_up value is 1 if the kernel considers the interface operational and 0 otherwise. The metric is a gauge that increments on every state change. The Prometheus rate() of an up/down gauge is the link flap rate.

Under the hood

How to configure it

The network collector is enabled by default. The flags that matter are the interface ignore regex and the netclass collector:

# /etc/systemd/system/node_exporter.service.d/override.conf
[Service]
ExecStart=
ExecStart=/opt/node_exporter/node_exporter \
  --web.listen-address=0.0.0.0:9100 \
  --collector.netclass.ignored-devices=^(veth.*|docker.*|br-.*|flannel.*|calico.*|cilium.*|cni.*|tunl.*|kube-ipvs.*)$ \
  --collector.netdev.device-include=^(eth.*|ens.*|eno.*|enp.*|bond.*|team.*)$ \
  --collector.netstat \
  --collector.softnet

The --collector.netdev.device-include regex is the inverse of the ignore regex. The include form is safer for production because the explicit allow-list catches new CNI interfaces that the team did not anticipate.

The --collector.netstat collector exposes Linux network stack metrics: node_netstat_Tcp_Established, node_netstat_Tcp_ActiveOpens, and so on. The metrics are the kernel-wide TCP connection counters.

The --collector.softnet collector exposes per-CPU softirq processing metrics: node_softnet_dropped_total, node_softnet_processed_total. The metric is the answer to “is the kernel dropping packets on the receive softirq?” The dropped counter is the saturation metric for the softirq processing path.

Reload the unit:

# SEVERITY: SERVICE-IMPACT
sudo systemctl daemon-reload
sudo systemctl restart node_exporter

How to validate it

The first check is that the metrics are present for the production interfaces:

# SEVERITY: READ-ONLY
curl -s http://localhost:9100/metrics | grep '^node_network_receive_bytes_total' | head -5

Expected output (illustrative):

node_network_receive_bytes_total{device="eth0"} 1.4e+10
node_network_receive_bytes_total{device="lo"} 4.2e+07

If the host reports only device="lo" and not the physical interface, the device-include regex is too restrictive. Adjust the regex to include the right prefix.

The second check is the carrier and up gauges:

# SEVERITY: READ-ONLY
curl -s http://localhost:9100/metrics | grep -E 'node_network_(carrier|up|mtu_bytes|speed_bytes)\{device="eth0"\}'

Expected output (illustrative):

node_network_carrier{device="eth0"} 1
node_network_up{device="eth0"} 1
node_network_speed_bytes{device="eth0"} 1.25e+08
node_network_mtu_bytes{device="eth0"} 1500

A node_network_up value of 0 on a production interface is a hard incident. A node_network_speed_bytes value of 1.25e+08 (1 Gbps) on a host that should be 10 Gbps is a switch port configuration issue.

The third check is the dashboard query. The production-grade network panel is split into four sub-panels:

# Panel 1: receive bytes per second
rate(node_network_receive_bytes_total{device=~"eth.*|ens.*|eno.*|enp.*|bond.*"}[1m])

# Panel 2: transmit bytes per second
rate(node_network_transmit_bytes_total{device=~"eth.*|ens.*|eno.*|enp.*|bond.*"}[1m])

# Panel 3: receive errors per second
rate(node_network_receive_errs_total{device=~"eth.*|ens.*|eno.*|enp.*|bond.*"}[1m])

# Panel 4: receive drops per second
rate(node_network_receive_drop_total{device=~"eth.*|ens.*|eno.*|enp.*|bond.*"}[1m])

The label selector is the per-device filter. The expression returns the rate over the last minute.

The fourth check is the alert rule:

# /etc/prometheus/rules/network.rules.yml
groups:
- name: network.errors
  interval: 30s
  rules:
  - alert: NetworkInterfaceRxErrors
    expr: |
      rate(node_network_receive_errs_total{device=~"eth.*|ens.*|eno.*|enp.*|bond.*"}[1m]) > 0.1
    for: 5m
    labels:
      severity: ticket
      team: platform
      resource: network
    annotations:
      summary: 'Receive errors on {{ $labels.instance }}/{{ $labels.device }} above 0.1/s for 5m'
      description: 'Check cable, switch port, and duplex settings.'
      runbook_url: 'https://runbooks.example.com/host/network-errors'

The threshold of 0.1 errors per second is the production baseline. A non-zero rate for more than five minutes is a slow failure.

How it can fail

Six failure modes appear repeatedly in production.

1. The interface label is not filtered. The host overview scans thousands of veth interfaces; the panel times out. The on-call engineer sees “no data” instead of the answer. The fix is the --collector.netdev.device-include regex.
2. The duplex mismatch is invisible. The link is up, the throughput is nominal, the TCP retransmits are killing the application. Symptom: the application sees latency; the host metrics are green. The fix is to alert on node_network_carrier changes and on TCP retransmits from the netstat collector.
3. The link speed is wrong. The switch port is hard-coded to 100 Mbps; the NIC auto-negotiates and reports 100 Mbps. The application is at 5% of the link’s actual capacity and the operator does not know. The fix is to alert on node_network_speed_bytes unexpected values.
4. The carrier is flapping. A bad cable or a misconfigured switch port causes the link to go up and down every few seconds. The node_network_up gauge transitions between 0 and 1. Symptom: TCP connections reset; the kernel log shows link is not ready. The fix is to replace the cable and configure the switch port.
5. The driver buffer is saturated. The softirq processing path cannot keep up with the packet rate; the kernel drops packets at the softirq layer. Symptom: drops accumulate on node_softnet_dropped_total but not on node_network_receive_drop_total. The fix is to enable RPS (Receive Packet Steering) or to increase the softirq CPU budget.
6. The VLAN or bond is misconfigured. The configured interface is up but the underlying physical is down. The metrics report the parent interface as up and the child as down. Symptom: the application is unreachable; the host metrics are green. The fix is to inspect the kernel log and the bond/VLAN state.

How to troubleshoot it

The diagnostic order when a host is slow and network is the suspect:

1. Inspect node_network_up for the production interface. Is the kernel state up?
2. Inspect node_network_carrier. Is the link physically connected?
3. Inspect node_network_speed_bytes and node_network_duplex. Is the link at the expected speed and duplex?
4. Inspect rate(node_network_receive_errs_total[1m]). Is the rate above 0.1/s?
5. Inspect rate(node_network_receive_drop_total[1m]). Is the rate above 0.1/s?
6. Inspect rate(node_network_transmit_carrier_total[1m]). Has the carrier been lost?
7. Inspect node_softnet_dropped_total rate. Is the kernel dropping on the softirq path?
8. Inspect node_netstat_Tcp_OutSegs and node_netstat_Tcp_RetransSegs. Is the TCP retransmit rate above 1% of outbound segments?
9. Inspect journalctl -k. Is the kernel logging link state changes?

Each step confirms or rules out a layer. The first three answer the “is the link up?” question. The next three answer the “is the link healthy?” question. The next two answer the “is the kernel keeping up?” question. The last asks the kernel directly.

Security implications

The network metrics do not contain payload data; they are aggregate counters. The PII surface is low.

The node_network_carrier and node_network_up values expose the link’s physical state. The information is operationally useful but not security-sensitive. A network attacker who can read the metrics learns the host’s network state; the threat model assumes the scraper is trusted.

The --collector.netdev collector walks /sys/class/net/. On a host with high-cardinality interfaces (CNI, veth), the collector emits a series per interface. The cardinality is the operational risk; the fix is the include regex.

The kernel log may contain the link’s MAC address and the switch port’s MAC address. The information is operational metadata; the journal forwarding should be configured to ship the kernel line without alteration.

Performance implications

The network metrics on node_exporter are per-interface counters and gauges. The cardinality is the number of interfaces per host. A host with two physical interfaces and a docker bridge has three relevant interfaces; the metric is cheap. A Kubernetes node with hundreds of veth interfaces has hundreds of series; the metric is the bottleneck.

The cost of reading /proc/net/dev and /sys/class/net/ is a few hundred microseconds per scrape. The scrape interval is not a performance concern.

The dominant cost is the absolute size of the per-interface label set in Prometheus. A query that scans all interfaces takes longer than a query that filters to the production interfaces. The fix is the include regex.

Production guidance

• Filter the interface label. The default ignore regex is a starting point; production fleets tune it per host class.
• Alert on errors and drops, not on throughput. The throughput is the planning question; the errors and drops are the real-time question.
• Alert on node_network_carrier and node_network_up transitions. The link flap is the silent-interface failure shape.
• Alert on TCP retransmit rate. The TCP layer is the application-visible signal.
• Coinvestigate network metrics with the application-side histogram. The two together answer the question.
• For per-interface SLOs, use the application-side histogram. The kernel counters do not emit per-request latency.

Verification

You should now be able to answer:

• What is the difference between node_network_receive_bytes_total rate and node_network_receive_errs_total rate, and what question does each answer?
• Why is the “silent interface” failure shape not visible on a throughput-only dashboard?
• How does the right per-interface filter change for a docker host versus a Kubernetes node?
• What is the diagnostic order when a host is slow and network is the suspect?
• Why is TCP retransmit rate the right application-visible signal for network health?

Quiz