Filesystem Metrics

A container orchestrator reports “no space left on device.” The dashboard for the host shows the root filesystem at 47% used. How can a filesystem at 47% be full? The cause is one of three things: the filesystem is full by inode count, the filesystem is full by reserved blocks that the metric excludes, or the disk is full on a sibling filesystem that node_exporter filtered out. Each has a different fix. This lesson is about reading node_filesystem_* correctly so that none of those is a surprise.

What it is

The filesystem collector in node_exporter walks /proc/1/mountinfo, takes the union of mount points visible to PID 1, and calls statvfs(2) on each. For each mount it exposes four metric families:

# HELP node_filesystem_size_bytes Filesystem size in bytes.
# TYPE node_filesystem_size_bytes gauge
node_filesystem_size_bytes{device="/dev/sda1",fstype="ext4",mountpoint="/"} 5.0e+11
node_filesystem_size_bytes{device="/dev/sdb1",fstype="ext4",mountpoint="/var"} 1.0e+12
node_filesystem_size_bytes{device="tmpfs",fstype="tmpfs",mountpoint="/run"} 1.6e+09

# HELP node_filesystem_avail_bytes Filesystem available space in bytes.
# TYPE node_filesystem_avail_bytes gauge
node_filesystem_avail_bytes{device="/dev/sda1",fstype="ext4",mountpoint="/"} 2.7e+11
node_filesystem_avail_bytes{device="/dev/sdb1",fstype="ext4",mountpoint="/var"} 8.2e+11
node_filesystem_avail_bytes{device="tmpfs",fstype="tmpfs",mountpoint="/run"} 1.5e+09

# HELP node_filesystem_files_total Filesystem total inodes.
# TYPE node_filesystem_files_total gauge
node_filesystem_files_total{device="/dev/sda1",fstype="ext4",mountpoint="/"} 3.2e+07
node_filesystem_files_total{device="/dev/sdb1",fstype="ext4",mountpoint="/var"} 6.5e+07

# HELP node_filesystem_files_free Filesystem free inodes.
# TYPE node_filesystem_files_free gauge
node_filesystem_files_free{device="/dev/sda1",fstype="ext4",mountpoint="/"} 3.1e+07
node_filesystem_files_free{device="/dev/sdb1",fstype="ext4",mountpoint="/var"} 6.4e+07

Four labels per series: device, fstype, mountpoint, and the mode of the mount (visible via separate metrics like node_filesystem_readonly). The cardinality is bounded by the number of distinct mounts on the host — typically 10–50, much higher on container hosts.

Size vs available

node_filesystem_size_bytes is the total capacity of the filesystem in bytes. node_filesystem_avail_bytes is the bytes available to a non-root user. The difference between size and avail is used + reserved, where reserved is the blocks the filesystem reserves for the root user (ext4 default: 5%).

A filesystem at 100% avail / size is empty. A filesystem at 0% is full — by the non-root user’s perspective. The root user can still write into the reserved area; nobody else can. This is why “df -h” reports 95% used when the dashboard reports 100% full: the dashboard is the non-root-user view.

# Filesystem utilisation as a fraction of capacity, root view.
1 - (node_filesystem_avail_bytes / node_filesystem_size_bytes)

# Used fraction including the reserved area, root view.
1 - ((node_filesystem_free_bytes) / node_filesystem_size_bytes)

The first query is what the dashboard should show. The second is what df shows. They differ by the reserved fraction.

Inodes

Filesystems track two resources: bytes and inodes. Inodes hold the metadata for each file (permissions, timestamps, block pointers). A filesystem with bytes available but no inodes available is full by metadata, not by data.

The metrics are node_filesystem_files_total (inodes total) and node_filesystem_files_free (inodes free). A ratio of free to total tells you the inode headroom.

# Inode utilisation.
1 - (node_filesystem_files_free / node_filesystem_files_total)

Inode exhaustion happens most often on:

• Mail servers (one file per message; millions of files).
• Cache servers (small files in big numbers).
• Container hosts with thousands of small overlay files.
• Filesystems that were created with a small inode ratio (ext4 mke2fs -N default is one inode per 16 KiB of capacity, which is fine for big files and too small for millions of small ones).

The alert is 1 - (node_filesystem_files_free / node_filesystem_files_total) > 0.9 for 10m. The fix is usually “delete some files” or “recreate the filesystem with more inodes.”

Mountinfo parsing

node_exporter reads /proc/1/mountinfo to discover mounts. This is more accurate than /etc/mtab (which may be out-of-date) and /proc/mounts (which may be filtered by the process namespace). The four fields it cares about:

36 35 98:0 /mnt/backup /mnt/backup rw,relatime shared:23 - ext4 /dev/sdc1 rw,...

The mountinfo line encodes the mount point, the filesystem type, the source device, and the mount options. The collector uses mountinfo to find the set of mounts visible to the host and to handle mount namespaces correctly.

The implication for Prometheus queries is that mountpoint="/" and device="/dev/sda1" are reliable. The collector also strips the [/...] sub-mount entries that overlay filesystems use.

Per-filesystem filter — the lucent fix

The “lucent” failure is the dashboards dominated by pseudo-filesystems. A container host with 50 bind mounts of /proc, /sys, and /dev has 50 node_filesystem_* series per metric per host, all showing small tmpfs/devpts values. The actual root filesystem is buried in the noise.

The fix is in the systemd unit:

--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|run|var/lib/docker/.+)($|/) \
--collector.filesystem.fs-types-exclude=^(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$

The mount-points-exclude regex drops the noisy bind mounts. The fs-types-exclude drops pseudo-filesystems at the type level. After both, a typical container host exposes 5–15 real filesystems.

The regex syntax is Go RE2. ^ matches the start of the string, $ matches the end, ($|/) allows the regex to match /proc and /proc/anything. A common typo is ^(sys|proc|dev)$ which only matches exact sys/proc/ dev strings and misses the slashes.

Read-only filesystem

A read-only mount is a real operational signal. It can mean:

• A package manager update remounted /usr read-only (modern immutable systems).
• The filesystem corrupted and was remounted read-only by the kernel for safety.
• The mount was deliberately made read-only (security or policy).

The metric is node_filesystem_readonly:

node_filesystem_readonly{device="/dev/sda1",fstype="ext4",mountpoint="/"} 0
node_filesystem_readonly{device="/dev/sdc1",fstype="ext4",mountpoint="/var/lib/postgres"} 0

A value of 1 means “this mount is read-only right now.” The signal is worth an alert on critical-path mounts (the database volume, the log volume).

A separate signal is “the filesystem was just remounted read-only” — this lives in dmesg and the kernel ring buffer, not in node_exporter. The collector exposes the current state; history requires log correlation.

Reserved blocks and watermarks

The reserved-blocks concept is filesystem-specific:

• ext4 reserves 5% of the blocks for root by default. Tunable at mkfs.ext4 -m time or tune2fs -m later.
• XFS has no reserved blocks by default; the equivalent is the allocation group free-space reserve.
• ZFS exposes available separately from free.

The metric node_filesystem_avail_bytes reflects the non-root-user view. If a filesystem is at 100% of avail / size, it is full from the application’s view. The root user can still write up to size * (1 - 0.05) on ext4 defaults; the dashboard shows the more pessimistic view.

How to configure it

The collector is enabled by default. The flags to tune are the two excludes. See the systemd unit in lesson 01.

Prometheus-side, the canonical recording rules:

# /etc/prometheus/rules/filesystem.yml
groups:
  - name: filesystem
    interval: 30s
    rules:
      - record: instance:filesystem_used_ratio
        expr: 1 - (node_filesystem_avail_bytes / node_filesystem_size_bytes)

      - record: instance:filesystem_inodes_used_ratio
        expr: 1 - (node_filesystem_files_free / node_filesystem_files_total)

      - record: instance:filesystem_readonly
        expr: node_filesystem_readonly

The PromQL patterns the on-call uses:

# Filesystems above 90% full.
topk(10, instance:filesystem_used_ratio) > 0.9

# Filesystems with inode pressure.
instance:filesystem_inodes_used_ratio > 0.9

# Filesystems that became read-only unexpectedly.
changes(instance:filesystem_readonly[5m]) > 0

The first query is “where is space going to run out.” The second is the inode version. The third catches the “remounted read-only” event.

How to validate it

# READ-ONLY
# 1. The collector is enabled and emitting filesystems.
curl -sf http://localhost:9100/metrics | grep '^node_filesystem_size_bytes' | head

# 2. The expected filesystems are present.
curl -sf http://localhost:9100/metrics \
  | grep '^node_filesystem_size_bytes' \
  | awk -F'mountpoint="' '{print $2}' | awk -F'"' '{print $1}'

# 3. Cross-check against /proc/mounts.
awk '{print $2}' /proc/mounts | sort -u

# 4. Inode metrics are present.
curl -sf http://localhost:9100/metrics | grep '^node_filesystem_files_'

# 5. Readonly flag is exposed.
curl -sf http://localhost:9100/metrics | grep '^node_filesystem_readonly'

Expected for a database host:

$ curl -sf http://localhost:9100/metrics | grep '^node_filesystem_size_bytes'
node_filesystem_size_bytes{device="/dev/sda1",fstype="ext4",mountpoint="/"} 5.0e+11
node_filesystem_size_bytes{device="/dev/sdb1",fstype="ext4",mountpoint="/var/lib/postgres"} 1.0e+12
$ curl -sf http://localhost:9100/metrics | grep '^node_filesystem_readonly'
node_filesystem_readonly{device="/dev/sda1",fstype="ext4",mountpoint="/"} 0
node_filesystem_readonly{device="/dev/sdb1",fstype="ext4",mountpoint="/var/lib/postgres"} 0

If /var/lib/postgres is missing, it was either excluded by the regex or its parent was. Inspect the regex.

How it can fail

1. Pseudo-filesystems dominate the panel. Symptom: the dashboard has 50 series per host, mostly small tmpfs/devpts values; the real filesystem is hard to find. Cause: --collector.filesystem.fs-types-exclude is missing or incomplete. Fix: add the canonical exclude list.
2. Inode exhaustion invisible on the byte panel. Symptom: “no space left on device” errors; dashboard shows 40% used. Cause: the inode ratio is high; the byte ratio is fine. Fix: panel the inode ratio separately; alert on it.
3. Read-only remount not alerted. Symptom: a filesystem is remounted read-only by the kernel after a corruption; the application fails to write; the alert fires only after the application crashes. Fix: alert on changes(instance:filesystem_readonly[5m]) > 0.
4. Reserved-block confusion. Symptom: the alert fires at 95% used, the operator checks df -h and sees the filesystem at 99%. Cause: df includes the reserved area in the “free” view; avail_bytes excludes it. Fix: document the difference; alert on avail_bytes.
5. Mount table changes silently dropped. Symptom: a new mount appears, but the metrics do not include it. Cause: a bind mount was created under a path that matches the exclude regex. Fix: inspect the actual mount table; adjust the regex to allow the new path.
6. Container overlay counts as a filesystem. Symptom: the panel shows overlay mounts dominating the output. Cause: the fs-types-exclude regex missed overlay. Fix: add overlay to the exclude.

Security implications

Filesystem metrics are moderate-risk:

• They reveal mount points and filesystem types. On a shared host, that is reconnaissance.
• They reveal disk usage shape (e.g. a /var/log volume that grows with user activity). That is also reconnaissance.
• The device label can expose underlying block devices, including LUKS-encrypted ones. The metric does not reveal encryption state, but it does reveal device names.

The standard mitigations apply: bind to a private interface, restrict with a firewall, or enable auth on the listener.

node_exporter does not read file contents; it only calls statfs(2). The textfile collector (opt-in) does read file contents. Drop the textfile collector’s directory to the narrowest possible scope.

Performance implications

The filesystem collector walks the mount table and calls statfs on each mount. On a host with hundreds of mounts (Kubernetes nodes, Docker hosts with many volumes) this is the dominant cost in node_exporter scrape duration.

The mitigations:

• mount-points-exclude removes the mounts the operator does not care about. Cuts the calls dramatically.
• fs-types-exclude removes pseudo-filesystems entirely. Cuts it further.
• A higher scrape interval (30s instead of 15s) trades resolution for cost.

On a typical container host with the canonical excludes, the collector takes 5–30 ms. On a host with hundreds of bind mounts and no excludes, it can take 200–800 ms — long enough to start showing on the dashboard load time.

Production guidance

• One canonical mount-points-exclude regex, in version control.
• One canonical fs-types-exclude regex, in version control.
• Recording rules for byte utilisation, inode utilisation, and readonly state.
• Alerts on:
  • 1 - avail/size > 0.85 for 10m — early warning.
  • 1 - avail/size > 0.95 for 5m — page the on-call.
  • 1 - files_free/files_total > 0.9 for 10m — inode pressure.
  • changes(readonly[5m]) > 0 on critical-path mounts — unexpected read-only.
• A panel that shows the top 10 filesystems by used ratio across the fleet, sorted by hostname. The operator opens this panel first when a “disk full” report arrives.

Verification

You should now be able to answer:

• What does node_filesystem_avail_bytes exclude that df -h does not, and why does it matter for alerts?
• How do inode pressure and byte pressure differ, and when is each the actual problem?
• Why do the mount-points-exclude and fs-types-exclude flags matter for dashboards, and what does “the lucent fix” mean?
• How would you detect that a filesystem was remounted read-only by the kernel?

Quiz