TCP Observability

The application’s request log shows 35 ms median, 380 ms p99. The TCP retransmit counter on the host rises from 14 to 9,400 per second. The user p99 is real; it is also a story about a host whose kernel is doing work the application cannot see.

The metric family that surfaces this is node_netstat_Tcp_*. It is the layer that turns a working application with a brown network into a panel someone is willing to investigate.

What it is

TCP observability, in this lesson, is the set of counters the Linux kernel exposes about TCP sockets and segments, formatted as Prometheus metrics by node_exporter. The counters come from three distinct kernel tables:

• /proc/net/snmp — the standard TCP MIB. Counters like Tcp_ActiveOpens, Tcp_PassiveOpens, Tcp_CurrEstab, Tcp_InSegs, Tcp_OutSegs, Tcp_RetransSegs, Tcp_InErrs, Tcp_OutRsts, Tcp_AttemptFails, Tcp_EstabResets. node_exporter exposes these as node_netstat_Tcp_*.
• /proc/net/netstat — extended TCP statistics. Counters like ListenOverflows, ListenDrops, TCPBacklogDrop, TCPDirectCopyFromBacklog, TCPDirectCopyFromPrequeue. The exporter exposes these as node_netstat_TcpExt_*.
• /proc/net/sockstat — socket counts by protocol and state. TCP_alloc, TCP_inuse, TCP_orphan, TCP_tw, TCP_mem. The exporter exposes these as node_sockstat_TCP_*.

These three sources are not interchangeable. The MIB counters count segments. The extended counters count kernel-level events that the MIB does not surface. The sockstat counters count sockets in particular states. A useful investigation reads all three.

The canonical alternative is eBPF-based tracing (bcc, bpftrace, Pixie). For most production sysadmins, the proc-based counters are the right approach because they are already on the host, require no kernel compilation, and integrate with every other metric in the same scrape job.

Why a sysadmin cares

Three production failure classes appear as green application metrics without these counters:

• SYN backlog exhaustion. An application accepts connections but its listen() backlog fills because accept is slow. New SYNs are dropped at the kernel level. The application never sees them. The metric that catches this is node_netstat_TcpExt_ListenOverflows rising.
• Packet loss on the path. The network is dropping one in two hundred packets. TCP retransmits the lost segments. The application sees latency spikes; throughput is otherwise fine. The metric that catches this is node_netstat_Tcp_RetransSegs rising relative to Tcp_OutSegs.
• Socket leak. A service opens a connection per request, closes the wrong end, and accumulates sockets in TIME_WAIT or CLOSE_WAIT. Eventually the host exhausts ephemeral ports or file descriptors. The metric that catches this is node_sockstat_TCP_tw rising, or node_sockstat_TCP_orphan holding steady at a non-zero value.

These failures cost minutes of investigation per incident without these metrics and seconds with them. The counters are already on the host. The exporter is already running. The cost of exposing them is a config flag.

How it works

The netstat collector reads /proc/net/snmp and /proc/net/netstat once per scrape. The sockstat collector reads /proc/net/sockstat once per scrape. The exporter formats each line as Prometheus counters or gauges and emits them.

   /proc/net/snmp           netstat collector
   -----------------        ----------------
   Tcp: ActiveOpens ...     --->  node_netstat_Tcp_*
   Tcp: PassiveOpens ...    one read per scrape
   Tcp: RetransSegs ...     no kernel calls

   /proc/net/netstat        netstat collector
   -----------------        ----------------
   TcpExt: ListenOver ...   --->  node_netstat_TcpExt_*

   /proc/net/sockstat       sockstat collector
   -----------------        -----------------
   sockets: used 412        --->  node_sockstat_TCP_*
   TCP: inuse 12 tw 89

Three things to internalise:

1. MIB counters are segments; extended counters are events. Tcp_RetransSegs counts every retransmitted segment. A burst of fast retransmits after a single drop produces several increments. ListenOverflows counts the number of times the kernel refused a SYN. Both are counters, but the unit is different. Read rates, not absolutes.
2. CurrEstab is a gauge, not a counter. It is the instantaneous count of sockets in ESTABLISHED. Read the value, not the rate.
3. RTT is not in /proc. The kernel does not expose a cumulative RTT counter. Per-socket RTT lives in the TCP_INFO socket option, surfaced by ss -tin. To get RTT into Prometheus, an exporter must run ss periodically and parse the output. node_exporter does not do this by default; an ss exporter or tcp_info eBPF program is the usual path.

How to configure it

Three decisions matter: which collectors are enabled, which fields are exposed, and which recording rules summarise the data.

1. The node_exporter collector set

# The field filter is one regex. Keep it in a single-quoted
# variable so the shell does not try to interpret ( ) | $.
NETSTAT_FIELDS='^Tcp_(Active|Passive)Opens$,^Tcp_(Curr|AttemptFails|OutRsts|EstabResets)$,^Tcp_(In|Out)Segs$,^Tcp_RetransSegs$,^Tcp_InErrs$,^TcpExt_Listen(Overflows|Drops)$,^TcpExt_(TCPBacklogDrop|TCPDirectCopyFromBacklog)$'

node_exporter \
  --collectors.enabled=netstat,sockstat \
  --collector.netstat.fields="$NETSTAT_FIELDS"

The --collector.netstat.fields flag is a regex over the field names. The default emits everything. Whitelisting the fields you alert on keeps the metric count bounded and reduces the risk of a future kernel adding a counter that breaks a dashboard label set.

2. The scrape job

# /etc/prometheus/prometheus.yml
scrape_configs:
  - job_name: node
    scrape_interval: 30s
    scrape_timeout: 10s
    static_configs:
      - targets: ['node-a.internal:9100']
        labels:
          env: prod
          role: app

No special configuration is required on the Prometheus side.

3. The recording rules

# /etc/prometheus/rules/tcp.yml
groups:
  - name: tcp.retransmits
    interval: 30s
    rules:
      - record: instance:tcp_retransmits:ratio5m
        # Retransmits as a fraction of outbound segments. A 1%
        # retransmit rate on a healthy link is a brownout; 5% is
        # an outage. Bound the ratio to ignore idle periods.
        expr: |
          rate(node_netstat_Tcp_RetransSegs[5m])
          /
          clamp_min(rate(node_netstat_Tcp_OutSegs[5m]), 1)

      - record: instance:tcp_listen_overflows:rate5m
        expr: rate(node_netstat_TcpExt_ListenOverflows[5m])

      - record: instance:tcp_established:gauge
        expr: node_netstat_Tcp_CurrEstab

The retransmit ratio is the headline. A 1 percent retransmit rate on a 100 Mbps link costs roughly 1 percent of throughput and a great deal more of latency variance. Alerting on absolute retransmits gives the wrong answer for a quiet host and the right answer for a busy one; the ratio normalises that.

How to validate it

Three layers must be confirmed: the exporter exposes the metrics, the metrics reflect real kernel state, and ss confirms what the counters summarise.

# 1. The metrics are exposed.
curl -sf http://node-a.internal:9100/metrics \
  | grep -E '^node_(netstat_Tcp|sockstat_TCP)_'
# node_netstat_Tcp_ActiveOpens{instance="node-a.internal:9100"} 412334
# node_netstat_Tcp_PassiveOpens{instance="node-a.internal:9100"} 1872341
# node_netstat_Tcp_CurrEstab{instance="node-a.internal:9100"} 87
# node_netstat_Tcp_RetransSegs{instance="node-a.internal:9100"} 9412
# node_netstat_TcpExt_ListenOverflows{instance="node-a.internal:9100"} 0
# node_sockstat_TCP_tw{instance="node-a.internal:9100"} 124

# 2. The values reflect /proc truth.
cat /proc/net/snmp | grep -E '^Tcp:'
# Tcp: ActiveOpens PassiveOpens AttemptFails EstabResets ...
# Tcp: 412334 1872341 0 12 ...
cat /proc/net/sockstat | grep -E '^TCP:'
# TCP: inuse 12 orphan 0 tw 124 alloc 144 mem 3

# 3. Confirm with ss and read RTT for a live connection.
ss -s
# TCP:   144 (estab 87, closed 12, orphaned 0, timewait 124)

ss -tin 'dst :443'
# ESTAB  0  10.0.0.4:443  10.0.4.17:52412
#  cubic wscale:7,7 rtt:3.4/4.1 mss:1448 ...

If the exporter values and /proc values agree within the scrape interval, the metric is correct. If ss -s reports a socket count that disagrees with node_sockstat_TCP_inuse, the exporter was scraped during a connection churn and the difference is real, not a bug.

How it can fail

Six failure modes appear regularly. Each one is recognisable in the data.

1. SYN backlog exhausted. TcpExt_ListenOverflows rises. The application’s listen() backlog is full. Symptom: client logs show ECONNREFUSED or connection timeouts that the server’s application log does not.

2. Receive path overrun. Tcp_OutSegs rises; Tcp_InSegs falls. The host is sending faster than the peer can ACK. Symptom: round-trip time climbs; throughput plateaus; ss -tin shows a non-zero retrans count.

3. Packet loss on the path. Tcp_RetransSegs rises sharply while Tcp_OutSegs is steady. The kernel is retransmitting lost segments. Symptom: latency tail widens; the application reports SLO breach without CPU or memory changes.

4. TIME_WAIT accumulation. node_sockstat_TCP_tw rises to the tens of thousands. The service opens and closes many short-lived connections. Symptom: ephemeral port exhaustion on a client that cannot bind a new socket; ss -s reports timewait in the tens of thousands.

5. Orphan accumulation. node_sockstat_TCP_orphan holds at a non-zero value. A service is closing its end of a socket without telling the peer. Symptom: Tcp_OutRsts rises as the peer eventually gives up.

6. RTT absent from metrics. node_exporter does not emit a per-socket RTT. Dashboards show TCP throughput but no latency. Symptom: the on-call engineer relies on application metrics to find latency, then must pivot to ss -tin or eBPF during the incident.

How to troubleshoot it

Order matters. Start at the boundary where evidence is most concrete.

1. Is the exporter up? curl http://host:9100/metrics. A missing node_exporter_build_info series means the exporter is not running. Check systemd or container logs.
2. Does the scrape flow? up{job="node"} in PromQL. If up=0, the problem is at the scrape boundary, not at the kernel.
3. Compare /proc to the exporter. If the values disagree, the exporter is on a different version or the scrape interval is dropping samples.
4. Look at the rate, not the absolute value. rate() and increase() turn retransmit counters into evidence. A retransmit count of 9412 is meaningless; the rate over a window is the signal.
5. Inspect the kernel with ss. ss -s, ss -tin, and ss -tlp 'sport = :PORT'. These confirm what the counters summarise and surface per-socket RTT, congestion window, and retransmit counts that the counters aggregate.
6. Check the listening backlog. ss -ltn and the application’s listen() configuration. A small Recv-Q with TcpExt_ListenOverflows rising is the textbook signature.

Security implications

The counters are read-only and expose no credential. The risk surface is the same as any other node_exporter metric: network reachability to port 9100. Bind the exporter on the monitoring network.

The counters themselves are operational. They do not reveal content or endpoints. The exception is CurrEstab and per- socket ss output, which can be correlated with a known internal IP set to infer connection patterns. Treat any TCP-level metric endpoint as a reconnaissance target for an attacker who already has a foothold on the monitoring network.

Performance implications

The netstat collector reads /proc/net/snmp and /proc/net/netstat. The cost is two file reads per scrape and two regex parses. On a busy host the proc files are a few kilobytes. The sockstat collector reads one small file. The total CPU cost is sub-millisecond per scrape interval.

Cardinality is low. Each metric is one series per host. The netstat.fields whitelist is more about hygiene than memory.

The performance cost of an ss-based RTT exporter is higher because it must walk every TCP socket on the host. A hundred-thousand-socket host pays real time in /proc walks or netlink dumps. Choose the polling interval and the source of RTT carefully.

Production guidance

• Use the ratio recording rule. A host moving gigabits per second with one percent retransmits is in distress; a host moving kilobits per second with one percent retransmits is fine. The ratio captures both.
• Pair Tcp_RetransSegs with node_network_* drops. A retransmit spike with no corresponding drop is a peer-side loss; a retransmit spike with a drop is local.
• Alert on ListenOverflows increasing, not on a threshold. One SYN drop is operationally interesting; ten per second is an outage.
• Document the sockstat interpretation for the team. A TCP_tw of 200 is normal; 20,000 is a leak. Without the context, the alert is meaningless.
• Where RTT matters, deploy a tcp_info exporter or an eBPF program and feed the metric to Prometheus. Do not rely on node_exporter for RTT.

Verification

You should now be able to answer:

• Which three /proc files do node_netstat_Tcp_*, node_netstat_TcpExt_*, and node_sockstat_TCP_* come from?
• What is the operational difference between Tcp_RetransSegs and TcpExt_ListenOverflows?
• Why is alerting on absolute retransmit count wrong, and what is the right denominator?
• How do you confirm a retransmit spike with ss?
• Why does node_exporter not expose RTT, and what is the usual workaround?

Quiz