Memory Observability · Observability

The application on host app-prod-12 is slow. The on-call engineer opens Grafana. The memory panel reports “55% used.” Nothing is red. The user complaint is real. The dashboard is asking the wrong question: it is reporting used memory, not available memory. A host can be at 95% used and still perfectly healthy because the kernel is reclaiming clean pages for disk cache; the same host can be at 55% used and thrashing because the dirty set is pinned and swap is full.

Memory observability is the discipline of measuring the available memory (the kernel’s own estimate of how much a new workload could allocate without swapping), the pressure on the memory subsystem (how often something is waiting for memory), and the errors (OOM kills, swap exhaustion, allocation failures). The metrics that answer these questions are the production baseline.

What it is

Memory observability on Linux is the practice of exposing four classes of measurement from the kernel:

Capacity - MemTotal, the physical RAM.
Allocated - MemFree, Buffers, Cached, SReclaimable, AnonPages.
Available - MemAvailable, the kernel’s estimate of free memory plus memory that can be reclaimed without swapping.
Pressure - PSI memory lines, which report the fraction of time at least one task is waiting for memory.
Errors - OOM kills, page-steal, swap-in/swap-out rate, allocation failures.

The naive dashboard reports “used” as MemTotal - MemFree. The production-graded dashboard reports “available” as MemAvailable. The difference is the most common single error in memory observability.

Why a sysadmin cares

Memory exhaustion is the most operationally expensive host-level failure mode. The recovery is slow: the kernel starts swapping, the disk fills, the page cache thrashes, the application freezes and the user gives up. The OOM killer fires long after the saturation has begun, and the postmortem has to reconstruct the seconds before the kill from logs that may also be stuck in the same memory pressure.

The production pattern is: a host has been at 95% used for six months. The team looked at the panel and said “this is fine, the kernel is just using memory for cache.” Two days before the incident, a deployment increased the application’s resident set size beyond the cache-reclaimable zone. The host went from 95% used to 95% used and 95% MemAvailable exhausted, in one release. The OOM killer fired. The on-call engineer learned the difference between used and available at 03:00.

The lesson learned: capacity is not the same as availability. The metric you alert on is the metric that has to mean what you think it means.

How it works

The Linux kernel publishes its memory state through /proc/meminfo and /proc/vmstat. The relevant fields and how node_exporter maps them:

  /proc/meminfo field     node_exporter metric
  ---------------------   ----------------------------------------
  MemTotal                node_memory_MemTotal_bytes
  MemFree                 node_memory_MemFree_bytes
  MemAvailable            node_memory_MemAvailable_bytes
  Buffers                 node_memory_Buffers_bytes
  Cached                  node_memory_Cached_bytes
  SwapTotal               node_memory_SwapTotal_bytes
  SwapFree                node_memory_SwapFree_bytes
  AnonPages               node_memory_AnonPages_bytes
  Dirty                   node_memory_Dirty_bytes
  Slab                    node_memory_Slab_bytes

  /proc/vmstat field      node_exporter metric
  ---------------------   ----------------------------------------
  pswpin                  node_vmstat_pswpin
  pswpout                 node_vmstat_pswpout
  pgmajfault              node_vmstat_pgmajfault
  oom_kill                node_vmstat_oom_kill

  /proc/pressure/memory   node_exporter metric
  ---------------------   ----------------------------------------
  some avg10=0.00         node_pressure_memory_waiting_seconds_total
  full avg10=0.00         node_pressure_memory_stalled_seconds_total

MemAvailable is the kernel’s own estimate of memory that can be made available to a new workload without swapping. It is calculated as MemFree + reclaimable cache + reclaimable slab. The exact formula is in the kernel source; the value is conservative.

PSI (/proc/pressure/memory) is the kernel’s stall instrumentation. Some line reports the fraction of time at least one task is waiting for memory; Full line reports the fraction of time the entire system is waiting. PSI is the only metric that directly answers “is something in the system memory-starved right now?”

                Memory Pressure (USE: saturation)
                =================================

   mem.total = 64 GiB
   mem.avail = 4.2 GiB     - the kernel can reclaim 4.2 GiB
                              without swapping. Below this,
                              new allocations are at risk.

   psi/some  avg10 = 0.4   - 40% of the last 10s, at least one
                              task was waiting for memory.
                              This is the on-host signal.

   vmstat.pswpin = 12/s    - the host is swapping in pages from
                              swap. This is the saturation metric.

   vmstat.oom_kill = 0     - the killer has not fired yet
                              (this counter is cumulative).

Under the hood

MemAvailable is calculated by the kernel as an estimate of memory that can be made available without performing I/O. The calculation, in mm/page_alloc.c and updated over kernel releases, considers MemFree, the reclaimable slab, the page cache, and the active vs inactive lists. The value is not the same as MemFree + Cached; it is more conservative.

The PSI mechanism, added in Linux 4.20 and incremented since, reports the fraction of time the system spent waiting on a resource. The some line is the time at least one task was waiting; the full line is the time every task was waiting. The values are sampled every two seconds and reported as averages over 10s, 60s, and 300s windows. PSI is the only kernel-level metric that directly answers “is something stalled right now?”

The OOM killer is invoked by the kernel when it cannot reclaim memory fast enough to satisfy an allocation. The counter is incremented in /proc/vmstat (oom_kill) and surfaced through the kernel ring buffer; the dmesg line is Out of memory: Killed process <pid> (<name>) total-vm:<N> anon-rss:<N> file-rss:<N>. Both sources are authoritative; the correlation between the counter and the log line is how the postmortem identifies the victim.

The vm.overcommit_memory sysctl changes the kernel’s policy: 0 (heuristic, default), 1 (always overcommit), 2 (never overcommit, capped by vm.overcommit_ratio). On a host that runs untrusted workloads, 2 is the safe choice. On a host that runs a database, the developer often sets 1 to avoid the OOM killer firing on a legitimate allocation peak; the trade-off is that there is no early warning of memory exhaustion.

How to configure it

node_exporter collects memory metrics by default. The flags that matter are the filesystem excludes (to avoid scraping tmpfs and overlay as memory consumers) and the processes collector (which captures process-level RSS):

# /etc/systemd/system/node_exporter.service.d/override.conf
[Service]
ExecStart=
ExecStart=/opt/node_exporter/node_exporter \
  --web.listen-address=0.0.0.0:9100 \
  --collector.systemd \
  --collector.processes \
  --collector.filesystem.mount-points-exclude=^/(dev|proc|sys|run|var/lib/docker/.+)($|/) \
  --collector.filesystem.fs-types-exclude=^(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ \
  --collector.vmstat.fields=^(oom_kill|pgmajfault|pswpin|pswpout|nr_free_pages|nr_inactive_anon|nr_active_anon|nr_inactive_file|nr_active_file|nr_dirty|nr_writeback)$

The collected vmstat.fields are the production baseline. The ones above cover OOM kills, page faults, swap I/O, and the per-zone page counts. The full list is longer; the bare minimum is oom_kill, pgmajfault, pswpin, pswpout.

Reload the unit:

# SEVERITY: SERVICE-IMPACT
sudo systemctl daemon-reload
sudo systemctl restart node_exporter

For OOM kills, the kernel ring buffer is the source of truth. node_exporter does not collect it; the production path is journalctl -k plus a log shipper (Loki, etc.) for retention.

To make OOM kills generate a syslog line, set the kernel printk level on RHEL 9 (the default is sufficient) and ensure the systemd journal is forwarded:

# SEVERITY: READ-ONLY
journalctl -k --since '-1h' | grep -i 'out of memory'

Expected output (illustrative):

Jan 14 02:51:13 app-prod-12 kernel: Out of memory: Killed process 18421 (java)
  total-vm:268435456kB, anon-rss:18350080kB, file-rss:5242880kB

How to validate it

The first check is that the metrics are present:

# SEVERITY: READ-ONLY
curl -s http://localhost:9100/metrics | grep -E 'node_memory_MemAvailable_bytes|node_vmstat_oom_kill|node_pressure_memory_waiting_seconds_total'

Expected output (illustrative):

node_memory_MemAvailable_bytes 4.2e+09
node_vmstat_oom_kill 0
node_pressure_memory_waiting_seconds_total 0.0008

The second check is the dashboard query. The production-grade panel is:

100 * (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)

This is the used fraction by the MemAvailable definition. The common variant, 100 * (1 - node_memory_MemFree_bytes / node_memory_MemTotal_bytes), is the dashboard error this lesson warns against.

The third check is the alert rule. The SLO-class rule for memory pressure is:

# /etc/prometheus/rules/memory.rules.yml
groups:
- name: memory.pressure
  interval: 30s
  rules:
  - alert: HostMemoryPressureHigh
    expr: |
      (
        node_pressure_memory_waiting_seconds_total
        /
        node_pressure_memory_waiting_seconds_total + 0.0001
      ) > 0.20
    for: 10m
    labels:
      severity: ticket
      team: platform
    annotations:
      summary: 'Memory pressure on {{ $labels.instance }} above 20% for 10m'
      description: 'PSI some avg10 is {{ $value | humanizePercentage }}. Host is stalling on memory. Inspect MemAvailable and OOM count.'
      runbook_url: 'https://runbooks.example.com/host/memory-pressure'

The fourth check is the kernel log for any recent OOM kills:

# SEVERITY: READ-ONLY
journalctl -k --since '-24h' | grep -c 'Out of memory'

A non-zero count is a hard incident. The number should match node_vmstat_oom_kill. If they diverge, the metric is being overcounted or the journal is being rotated before the metric captures it.

How it can fail

Six failure modes appear repeatedly in production.

Used-vs-available confusion. The dashboard reports MemTotal - MemFree as “used.” A host can be at 75% used by that calculation and at 12% used by MemAvailable. Symptom: the team calls the host “fine” while the kernel is reclaiming cache and the application is slowing down. The fix is to add MemAvailable to the panel and to subtract it from MemTotal.
The OOM killer fires, but the metric was not captured. The metric node_vmstat_oom_kill is present but the journal line is not. Symptom: the metric says zero kills, journalctl says one. The fix is to verify the journal is forwarded to the log platform before relying on the metric.
PSI is missing. The kernel is older than 4.20 or the sysctl kernel.psi_enabled=0 is set. Symptom: the node_pressure_* series are absent. The fix is to enable PSI or substitute node_load5 and node_vmstat_pswpin as saturation proxies.
Swap is enabled when it should not be. A database host has swap on a fast SSD; the kernel swaps instead of OOM killing. Symptom: latency spikes during period memory peaks, no OOM kill. The fix is to disable swap (swapoff -a and remove from /etc/fstab) for hosts whose memory budget is sized correctly.
Memory cgroup limits are wrong. A container host runs containers with memory.limit_in_bytes below the application’s working set. Symptom: the cgroup kills processes, but the host reports plenty of free memory. The fix is to size the cgroup to the application’s working set + headroom.
The kernel overcommit is set aggressively. vm.overcommit_memory=1 allows all allocations; the system never reports pressure until the page allocator itself fails. Symptom: a sudden OOM kill on a host that was at 30% used by the panel. The fix is to set vm.overcommit_memory=0 (default) and vm.overcommit_ratio= to a value the host can back with swap.

How to troubleshoot it

The diagnostic order when a host is slow and memory is the suspect:

Inspect MemAvailable and MemTotal. Is the host at 10% available or 1% available?
Inspect node_pressure_memory_waiting_seconds_total. Is the PSI “some” avg10 above 5%?
Inspect node_vmstat_pswpin and node_vmstat_pswpout. Is the host swapping?
Inspect node_vmstat_oom_kill. Has the OOM killer fired?
Inspect journalctl -k --since '-1h' | grep -i 'out of memory' for the most recent kill. Correlate with the application logs.
Inspect the per-process RSS via top -o %MEM or ps -eo pid,rss,comm --sort -rss | head -20.
Inspect the cgroup if the host is a container host: cat /sys/fs/cgroup/memory/memory.usage_in_bytes and memory.limit_in_bytes.

Each step confirms or rules out a layer. Step 1 is the capacity question. Step 2 is the pressure question. Step 3 is the saturation question. Step 4 is the error question. The distinction is the USE method applied to memory.

What if the host is at 4% MemAvailable and the application is slow?

A host is at 4% MemAvailable and the application is slow. The on-call engineer restarts the application, which frees the page cache, and the host returns to 40% available. Six hours later, the host is back at 4% available. The root cause is a leak in the application; the restart only delays the symptom. The fix is to inspect the application’s RSS over time, not to script a restart-and-pray.

A cgroup OOM kills the application’s main process. The host metrics show plenty of free memory, and the node_vmstat_oom_kill counter does not increment. The on-call engineer sees the host dashboards as green and the application as broken. The fix is to inspect the cgroup memory.events file and the application’s own log line, which on systemd shows the cgroup OOM kill.

A database host with swap enabled on a fast SSD starts swapping during a query. The kernel pages out file cache, then pages out anonymous memory, then page-faults it back. The latency is catastrophic. The fix is to disable swap on the host and to size the database memory to the actual working set.

Security implications

/proc/meminfo and /proc/vmstat are world-readable. The metrics node_exporter exposes do not contain process names, process owners, or memory contents. The PSI values are aggregate fractions. The PII surface is low.

The cgroup path is more sensitive. Per-container memory usage is a useful metric for the operator but can leak information about which tenants are running and how much they consume. The production baseline is to expose aggregate metrics to the scraper and to require authentication for the per-cgroup breakdown.

The kernel ring buffer is not rate-limited. A misconfigured log shipper can DoS the host by writing OOM lines to a slow disk. The fix is to ship the kernel ring via the journal and to use the journal’s rate-limiting.

Performance implications

The memory metrics on node_exporter are counters and gauges on a finite, bounded set of fields. The cardinality is small (tens of series per host). The cost is reading /proc/meminfo and /proc/vmstat on every scrape, which is a few hundred microseconds. The scrape interval is not a problem.

The PSI collector reads /proc/pressure/memory on every scrape. The cost is a few hundred microseconds; the kernel updates the value every two seconds. The collector is cheap.

The per-process collector (top -o %MEM or ps) is expensive on hosts with many processes. The node_exporter processes collector walks /proc/<pid>/ and emits aggregate counts, not per-process memory. Per-process memory on a high-process host is collected by a dedicated exporter or by an internal metric on the application.

Production guidance

Alert on MemAvailable and PSI, not on MemFree. The two metrics are not the same.
SLO the panel on memory pressure, not on memory usage. The user feels the pressure, not the usage.
Disable swap on hosts whose memory is sized correctly. Swap is a safety net for sizing errors, not a working-set solution.
Set explicit vm.overcommit_memory per host class. Heuristic is fine for general-purpose hosts; 2 for hosts running untrusted workloads; 1 for database hosts whose memory is sized to the working set.
Forward the kernel ring buffer to the log platform. The node_vmstat_oom_kill counter is enough for alerting; the log line is what the postmortem needs.

Verification

You should now be able to answer:

What is the difference between MemTotal - MemFree and MemAvailable, and why does the dashboard error this?
How does the kernel signal that something is waiting for memory, and how does node_exporter expose that?
What is the canonical source for OOM kills, and how does the metric node_vmstat_oom_kill differ from the journal line?
Why is swap on a database host usually a misconfiguration?
What is the diagnostic order when a host is slow and memory is the suspect?