Writeback and filesystem tuning - dirty pages, mount options and fstrim · Linux

The previous lesson established that a large share of storage latency is added above the block layer. This one is about the two places it is added most often: the writeback path, and the filesystem’s own decisions about journalling, timestamps and discard.

Every knob here has a defensible default. The point is not to change them - it is to know which one to reach for when a measurement says something is wrong, and to know what each change costs.

Writeback: the dirty page limits

A buffered write() makes a page dirty and returns. Writeback drains dirty pages to the device in the background. Two thresholds govern it:

sysctl vm.dirty_background_ratio vm.dirty_ratio \
       vm.dirty_expire_centisecs vm.dirty_writeback_centisecs

vm.dirty_background_ratio = 10
vm.dirty_ratio = 20
vm.dirty_expire_centisecs = 3000
vm.dirty_writeback_centisecs = 500

dirty_background_ratio - at this fraction of available memory dirty, kernel writeback threads start flushing in the background. The application is not affected.
dirty_ratio - at this fraction, the writing process is blocked until writeback catches up. This is the throttle.
dirty_expire_centisecs - a dirty page older than this (30 s here) becomes eligible for writeback regardless of the thresholds.
dirty_writeback_centisecs - how often the writeback threads wake to check (5 s here).

Watch the live values:

grep -E '^(Dirty|Writeback|MemAvailable):' /proc/meminfo

Dirty:            418204 kB
Writeback:         12288 kB
MemAvailable:   43221108 kB

Dirty climbing toward dirty_ratio of MemAvailable while Writeback stays small means the device cannot keep up and throttling is imminent.

20% of 512 GB is 100 GB of dirty pages waiting to hit one device

The percentage-based defaults were chosen when servers had a few gigabytes of RAM. On a modern large-memory host they permit an absurd amount of unwritten data.

On a 512 GB host, dirty_ratio = 20 allows roughly 100 GB dirty before the writer is throttled. The consequences arrive together:

A writeback storm. When the background threshold is finally crossed, tens of gigabytes are pushed at the device at once. Every other reader on that device sees latency go from sub-millisecond to seconds. A database on the same volume stalls, and nothing in its own metrics explains why.
A very long throttle. A process that hits dirty_ratio is blocked until writeback drains enough. At 100 GB behind on a 500 MB/s device, that is minutes.
A large data loss window. Up to 100 GB of acknowledged buffered writes exist only in RAM. A power failure loses all of it, and nothing in the application knows.

The fix is to set the byte-valued forms instead, which are absolute and do not scale with RAM:

# Setting either *_bytes disables the corresponding *_ratio.
sudo sysctl -w vm.dirty_background_bytes=$((256 * 1024 * 1024))
sudo sysctl -w vm.dirty_bytes=$((1024 * 1024 * 1024))
sysctl vm.dirty_background_ratio vm.dirty_ratio

vm.dirty_background_ratio = 0
vm.dirty_ratio = 0

The ratios reading 0 confirms the byte values took effect - the two forms are mutually exclusive, and writing one zeroes the other. Size them from the device: background at roughly a second of write throughput, hard limit at a few seconds of it, so a throttle lasts seconds rather than minutes.

Measure the effect on both sides. The device side:

iostat -x 5 12

Look for wkB/s becoming steadier rather than arriving in bursts, and r_await on the same device losing its spikes. The application side is whatever your workload reports as write latency - the two must be checked together, because smoothing writeback can slightly reduce peak sequential throughput while substantially improving read latency, and only you know which one your service is judged on.

Mount options

findmnt -no TARGET,FSTYPE,OPTIONS -t ext4,xfs

/ ext4 rw,relatime

atime

Every read of a file updates its access timestamp, which is a write. The three behaviours:

strictatime - update on every access. A read-heavy workload turns into a write workload. Almost never wanted.
relatime - the default. Update only if the existing atime is older than mtime/ctime, or older than a day. Cheap, and keeps atime useful enough for “has this file been read recently” tooling.
noatime - never update. Cheapest.

noatime is the classic first recommendation in tuning guides, and on a modern kernel relatime has already removed most of the cost. Measure before assuming: on a mail spool or a large static file cache with millions of reads it is still worth having; on a database whose data lives in a handful of large files it changes essentially nothing, because those files are read constantly and their atime is already fresh.

# /etc/fstab
UUID=b0434124-e13c-4dff-9006-5abf403266fe / ext4 defaults,noatime 0 1

Some applications do read atime - mutt uses it to detect new mail, and some backup and tiering tools use it to find cold data. Check before setting noatime on a filesystem you do not own.

ext4 journal mode

data=ordered     default. Metadata journalled; data written before
                 its metadata commits. Protects against reading
                 stale blocks after a crash.
data=writeback   metadata journalled, data ordering not enforced.
                 Faster; a crash can expose stale block contents
                 in files that were being extended.
data=journal     data AND metadata journalled. Everything written
                 twice. Slowest, strongest.

data=writeback is the one that appears in tuning guides with the caveat omitted. The risk is not filesystem corruption - the filesystem stays consistent - it is that after a crash a file extended just before the crash can contain whatever was previously in those blocks, which may be another file’s data. On a host handling data belonging to more than one tenant, that is a security consideration and not only a durability one.

commit=nrsec controls how often the journal commits (5 seconds by default). Raising it batches more work per commit and reduces f/s in iostat, at the cost of widening the crash window by the same amount.

Say which failure you are buying, then choose

Every option in this section trades a quantity of durability for a quantity of speed, and the honest way to present the choice is to name the failure:

Change	What you gain	What you can lose on a crash
`noatime`	Fewer metadata writes	Nothing - atime is not durability
`commit=30`	Fewer journal commits	Up to 30 s of metadata changes
`data=writeback`	Faster extends	Stale block contents in recently extended files
`barrier=0` / `nobarrier`	Large gain on cache-backed devices	Filesystem consistency itself
`dirty_bytes` raised	Smoother bursts	More acknowledged data held only in RAM

The first row is free. The last row is a decision with an owner. Write the row you chose into the change record, because a mount option is invisible until the crash that reveals it.

XFS

XFS has fewer knobs by design and its defaults are good. The two worth knowing are logbsize=256k, which enlarges the in-memory log buffer and helps metadata-heavy workloads, and allocsize=, which sets the speculative preallocation size for streaming writes. noatime applies the same way. Do not go looking for an XFS equivalent of data=writeback; there is not one, because XFS never journals file data.

Discard: inline or batched

An SSD needs to be told which blocks are no longer in use, or its garbage collector works against stale data and write amplification climbs. Two ways to tell it.

Inline discard - the discard mount option - issues a TRIM as part of every delete. It sounds efficient and often is not: on many devices a discard is a slow, synchronous command, so file deletion becomes slow and the discards arrive in a stream of tiny ranges the device handles poorly.

Batched discard - fstrim - walks the free space and issues large discard ranges in one pass. This is the default on every mainstream distribution, delivered as a timer:

systemctl cat fstrim.timer

[Unit]
Description=Discard unused filesystem blocks once a week
Documentation=man:fstrim
ConditionVirtualization=!container
ConditionPathExists=!/etc/initrd-release

[Timer]
OnCalendar=weekly
AccuracySec=1h
Persistent=true
RandomizedDelaySec=100min

[Install]
WantedBy=timers.target

Note what that unit does with everything from the scheduling part: Persistent=true so a host that was off catches up, RandomizedDelaySec=100min so a fleet does not trim simultaneously, and AccuracySec=1h because a weekly maintenance job does not need to fire at a precise instant.

Check it is enabled and see what it reclaims:

systemctl list-timers fstrim.timer
sudo fstrim -av

/boot: 743.6 MiB (779616256 bytes) trimmed on /dev/sda1
/: 412.1 GiB (442495008768 bytes) trimmed on /dev/sda2

fstrim on a thin-provisioned or shared volume is not a free operation

fstrim -av on a large filesystem issues discards across all its free space. On a plain local SSD that is fine and takes seconds. Elsewhere it is not:

Thin-provisioned SAN volumes interpret the discard as “release this space”, which is usually what you want, but the array does real work for it and a fleet trimming at once has taken storage arrays down.
Some SSD firmware handles a very large discard range badly and blocks other I/O for the duration. This produced a well-known class of “the host freezes for 30 seconds every Monday” incidents.
Snapshots. Discarding blocks referenced by a snapshot can force copy-on-write allocation rather than freeing anything, so usage goes up.

Test with --dry-run first, which reports what would be trimmed without issuing anything:

sudo fstrim -av --dry-run

And check whether the device even supports it before assuming the timer is doing anything:

lsblk --discard -o NAME,DISC-GRAN,DISC-MAX

Zero in both columns means discard is unsupported on that path, and the weekly timer has been a no-op the whole time.

Read-ahead, checked against the workload

cat /sys/block/sda/queue/read_ahead_kb

Read-ahead is how far past a detected sequential pattern the kernel reads. A large value helps sequential scans - backups, analytics table scans - and hurts random-access workloads, where it fills page cache with data that will never be read and consumes device bandwidth doing it.

Confirm which kind of workload you have before changing it:

iostat -x 5 3

Compare rareq-sz - the average read request size in kB - against the filesystem block size. Requests much larger than the block size mean the kernel is merging sequential reads and read-ahead is working. Requests at or near the block size mean random access, and raising read-ahead will make things worse.

Persist any change with a udev rule rather than an rc.local echo, so devices added later inherit it - the pattern is in the queue depth lesson.