Memory Metrics

The service is being killed. dmesg shows Out of memory: Killed process 12345 (myapp) total-vm:..., anon-rss:.... The host panel, however, shows 30% memory used. How can a host at 30% memory used have an OOM kill?

The answer is in the difference between allocated memory and available memory, and the difference between reclaimable memory and anonymous memory. A host can show 30% used and still OOM-kill a process, because the 30% figure excluded the page cache that filled up when an application requested more anonymous memory. This lesson is about reading node_exporter memory metrics correctly.

What it is

node_memory_* metrics come from /proc/meminfo, parsed into named gauges. The list is long; the operationally meaningful ones are a smaller set:

# Total memory installed.
node_memory_MemTotal_bytes 6.7e+10

# Truly free — pages the kernel can immediately hand out.
node_memory_MemFree_bytes 8.4e+09

# Memory the kernel estimates it could make available without
# swapping. This is the operational metric.
node_memory_MemAvailable_bytes 3.2e+10

# Page cache (disk-backed pages that can be evicted).
node_memory_Cached_bytes 1.9e+10

# Buffer cache (block-device metadata, similar idea).
node_memory_Buffers_bytes 1.2e+08

# Writeback — pages marked for write to disk but not yet written.
node_memory_Writeback_bytes 0

# Anonymous memory used by applications (heap, stack).
# Approximated as: Active(anon) + Inactive(anon).
node_memory_AnonPages_bytes 1.4e+10

# Slab allocator — kernel object caches. Some is reclaimable.
node_memory_Slab_bytes 9.0e+08

# Reclaimable slab specifically.
node_memory_SReclaimable_bytes 6.0e+08

# Swap total / free / used (in bytes).
node_memory_SwapTotal_bytes 4.0e+09
node_memory_SwapFree_bytes  4.0e+09
node_memory_SwapCached_bytes 0

The cardinalities are bounded: one series per metric name. The node_exporter memory collector adds ~50 series per host. No labels.

MemFree vs MemAvailable

The single most-misread metric in node_memory_* is MemFree. MemFree is “memory the kernel is currently not using for anything.” On a healthy Linux host with a hot page cache, MemFree is often less than 1% of MemTotal. That is not a problem.

MemAvailable is the kernel’s estimate of memory that could be made available to a new request without swapping. It includes MemFree plus a fraction of the page cache that the kernel believes can be evicted plus reclaimable slab. It is the operationally correct “memory pressure” gauge.

The kernel computes MemAvailable as approximately:

MemAvailable = MemFree
             + (Cached - max(0, Cached - (Active_file + Inactive_file)/2))
             + SReclaimable
             - (LowWaterMark * nr_cpus)

The exact formula is in mm/page_alloc.c. The intuition is “what would I have if I reclaimed the cache aggressively?” That is the right question for a new allocation.

Anonymous vs page cache

Two kinds of memory dominate a running process:

• Anonymous memory. The application’s heap and stack, file-mapped MAP_ANONYMOUS regions, malloced regions. Pages that have no on-disk backing. They are evicted to swap (if swap is enabled) or killed (if not). The AnonPages metric is the closest signal.
• Page cache. Pages the kernel has cached for files that are on disk. They are evicted under memory pressure with no application-visible effect (other than disk I/O). The Cached metric covers this.

A host that runs an application whose heap grows has more anonymous pages. A host that runs a database whose buffer pool is mostly in memory has more page cache. The two have different pressure signatures:

• Page-cache pressure. Memory shrinks back as the kernel evicts cache. The application sees no slowdown. The metric is Cached going down. The kernel’s PSI memory pressure may rise briefly.
• Anonymous pressure. Memory shrinks by swapping out anonymous pages (if swap is configured) or by killing processes (if not). The metric is SwapFree going down or node_vmstat_oom_kill increasing. PSI memory pressure is sustained.

A common operational mistake is to treat “memory use” as one number. It is at least two numbers, and the operational question depends on which is growing.

Slab and reclaimable

The kernel uses slab allocators for its own data structures (inode cache, dentry cache, buffer_head cache, and so on). Slab memory appears in node_memory_Slab_bytes. A portion is reclaimable (SReclaimable); the rest is not (SUnreclaim).

A host with a large SUnreclaim is using kernel memory that cannot be reclaimed. This is rare but occurs on hosts with many open files, many mounts, or a kernel leak. The metric is worth a panel; alerts on it are unusual.

Swap

Swap is a single number (SwapTotal_bytes), a free number (SwapFree_bytes), and a cached number (SwapCached_bytes). The operational signals:

• SwapUsed = SwapTotal - SwapFree rising over time. The host is paging out anonymous memory.
• node_vmstat_pswpin and node_vmstat_pswpout rising. Page-ins and page-outs per second. Sustained non-zero values mean the host is actively swapping.
• node_vmstat_oom_kill increasing. The OOM killer fired.

A common operational question is “do we have swap enabled?” On a database host the answer is usually “yes, modestly, for emergency use only.” On an application host with no swap, the answer is usually “no, because we sized memory right.” Both are defensible. The operational rule is the same: monitor swap usage and OOM events regardless of policy.

How to configure it

The memory collector is enabled by default; there is no flag to tune. The configuration is on the Prometheus side.

A canonical recording-rule group:

# /etc/prometheus/rules/memory.yml
groups:
  - name: memory
    interval: 30s
    rules:
      - record: instance:memory_available_ratio
        expr: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes

      - record: instance:memory_anon_ratio
        expr: node_memory_AnonPages_bytes / node_memory_MemTotal_bytes

      - record: instance:memory_swap_used_bytes
        expr: node_memory_SwapTotal_bytes - node_memory_SwapFree_bytes

      - record: instance:memory_oom_kills:increase5m
        expr: increase(node_vmstat_oom_kill[5m])

      - record: instance:memory_pressure_stall_seconds:rate5m
        expr: rate(node_pressure_memory_stall_seconds_total[5m])

The recording rules give stable names for dashboards and alerts. The first rule — memory_available_ratio — is the canonical “how much memory can the host still use” gauge. The fifth rule is the PSI pressure-stall signal.

The PromQL patterns the on-call uses:

# Top 10 hosts by memory pressure.
topk(10, 1 - instance:memory_available_ratio:)

# Hosts that have OOM-killed in the last hour.
instance:memory_oom_kills:increase1h > 0

# Hosts where swap is in use and available is low.
instance:memory_swap_used_bytes > 0
  and instance:memory_available_ratio < 0.1

# Hosts with sustained memory pressure stall.
instance:memory_pressure_stall_seconds:rate5m > 0.2

How to validate it

# READ-ONLY
# 1. Memory metrics are exposed.
curl -sf http://localhost:9100/metrics | grep -E '^node_memory_(MemTotal|MemFree|MemAvailable|Cached|AnonPages|SwapTotal)_bytes'

# 2. VMSTAT (for OOM detection) is exposed.
curl -sf http://localhost:9100/metrics | grep '^node_vmstat_oom_kill'

# 3. PSI memory pressure is exposed.
curl -sf http://localhost:9100/metrics | grep '^node_pressure_memory'

# 4. Cross-check against /proc/meminfo directly.
grep -E '^(MemTotal|MemFree|MemAvailable|Cached|AnonPages|SwapTotal|SwapFree):' /proc/meminfo

Expected for a healthy 64-GiB host:

$ curl -sf http://localhost:9100/metrics | grep -E '^node_memory_MemAvailable|^node_memory_MemTotal'
node_memory_MemAvailable_bytes 3.2e+10
node_memory_MemTotal_bytes 6.7e+10
$ curl -sf http://localhost:9100/metrics | grep '^node_vmstat_oom_kill'
node_vmstat_oom_kill 0
$ curl -sf http://localhost:9100/metrics | grep '^node_pressure_memory'
node_pressure_memory_stall_seconds_total 1.23e+02
node_pressure_memory_some_seconds_total 0.87e+02

If node_pressure_memory_* is missing, the pressure collector is disabled. The default in 1.8.x is enabled; older versions needed --collector.pressure.

How it can fail

1. The “free” number looks low. Symptom: panel shows MemFree_bytes at 800 MiB on a 64-GiB host; on-call thinks the host is full. Cause: confusion with MemAvailable. Fix: panel the ratio MemAvailable / MemTotal; alert on that.
2. OOM happened but the dashboard was green. Symptom: application restarts; dmesg shows the kill; node_exporter showed 30% used. Cause: the alert was on MemAvailable / MemTotal; the OOM fired before the kernel reclaimed enough cache to push MemAvailable below threshold. Fix: alert on node_vmstat_oom_kill changes, not on the level gauge. The OOM event itself is the signal.
3. Swap-in-use false alarm. Symptom: an alert fires because some swap is in use. Cause: alert threshold is too low. Fix: alert on “swap is full AND available is low” — both must be true.
4. PSI pressure metric missing. Symptom: node_pressure_memory_* is absent. Cause: the pressure collector is disabled. Fix: enable it.
5. Memory leak undetectable at the host level. Symptom: application memory grows until OOM, but the host-level MemAvailable looks fine until the last minute. Cause: the host has enough cache to absorb the leak until the leak fills it. Fix: alert on the application’s own memory metric (RSS, heap), not the host gauge.
6. Container memory limit not reflected. Symptom: a container is killed by its cgroup limit, not by the host. Cause: cgroup-level metrics come from cAdvisor/kubelet, not node_exporter. Fix: deploy the container collector alongside node_exporter; alert on the cgroup metric.

Security implications

Memory metrics are low-risk: they reveal total memory and usage shape. They do not reveal process memory contents or user data. The risk surfaces are:

• The /metrics endpoint is unauthenticated. Network ACL or TLS + auth on the listener is the standard mitigation.
• MemAvailable reveals whether a host is under pressure, which is reconnaissance for a noisy-neighbour attack on shared infrastructure.
• node_exporter does not write to memory metrics; the only write path is the textfile collector (opt-in).

The deeper risk is information disclosure on multi-tenant hosts. A multi-tenant VM host that exposes node_memory_* to tenants reveals how busy the host is. In a hostile environment that is reconnaissance.

Performance implications

The memory collector is cheap: a single read of /proc/meminfo and /proc/vmstat. Scrape duration impact is in single milliseconds.

The cost comes from the dashboard side. A panel that shows MemAvailable_bytes on a thousand hosts is one thousand series. With a recording rule the dashboard pulls from the recording-rule series. The cardinality of node_memory_* is itself low; the rate of change is what matters.

The PSI pressure collector is also cheap. The kernel pre-aggregates the pressure counters; the collector just reads them.

Production guidance

• Panel MemAvailable / MemTotal as the canonical “memory used” gauge. Do not panel MemFree.
• Alert on node_vmstat_oom_kill:increase > 0 for 1m. This is the loud signal.
• Alert on MemAvailable_bytes / MemTotal < 0.05 for 5m as the early-warning signal.
• Panel PSI pressure (node_pressure_memory_some_seconds_total) alongside the level gauges. Pressure tells you “the system is being asked for memory it cannot give right now.”
• Document the swap policy in the runbook. Be explicit about whether swap is on or off, and what “swap in use” means in the team’s response procedure.
• For containers, deploy cAdvisor or the OpenTelemetry collector with the container receiver to expose cgroup- level metrics. Host metrics do not reflect cgroup limits.

Verification

You should now be able to answer:

• Why is MemAvailable the right metric and not MemFree?
• What is the difference between anonymous memory and the page cache, and what does that imply for memory pressure?
• How do you detect that an OOM kill has happened on a host, and what does node_vmstat_oom_kill tell you?
• Why is “swap in use” not, by itself, an alertable condition?
• What does PSI memory pressure tell you that MemAvailable does not?

Quiz