State in an immutable fleet - what survives replacement and what does not · Linux

Immutable infrastructure has exactly one hard requirement, and every difficulty with it descends from that requirement: replacing an instance discards everything written to its root disk since launch.

For a stateless web server that is a feature. For a real production system it is a design constraint that has to be applied to every path the machine writes to - and most teams discover the paths they missed weeks later, when the loss finally surfaces.

Audit the machine first

Before making a class of host disposable, find out what it is actually writing. Two commands go a long way.

# Files on the root filesystem modified since the machine booted,
# excluding the paths you expect to churn
find / -xdev -type f -newermt "$(uptime --since)" \
  -not -path '/proc/*' -not -path '/sys/*' -not -path '/run/*' \
  -not -path '/var/log/*' -not -path '/var/lib/systemd/*' 2>/dev/null

Read-only / Safe

$ find /var -xdev -type f -mmin -60 2>/dev/null | head

/var/log/kern.log
/var/log/wtmp
/var/lib/myapp/outbox/00042.msg
/var/lib/myapp/outbox/00043.msg
/var/spool/postfix/deferred/A/AB12C4

Illustrative output

The first two lines are expected. The last three are the ones that matter: an application spool and a mail queue, both on the root disk, both containing work that has been accepted from somebody and not yet delivered.

The second command is who has files open:

sudo lsof -w /var/lib /srv /opt 2>/dev/null | awk '$4 ~ /[0-9]+[uw]/ {print $1, $NF}' | sort -u

lsof restricted to writable descriptors tells you which processes are writing where. Between the two you get an inventory of state, which is the thing you need before you can decide anything.

Where each class of state belongs

State	Belongs	If left on the root disk
Database files, uploads	Persistent volume or managed service	Lost at replacement
Logs	Shipped to a collector as written	The logs explaining the incident die with the instance
Caches	Local - they are rebuildable	Cold start after every replacement, which is a capacity issue
Sessions	External store	Users logged out on every deploy
Queues and spools	A real queue service, or a volume	Accepted work silently dropped
Metrics buffers	Short flush interval, accept small loss	Minutes of gaps at every replacement
Machine identity	Generated at first boot	See below

Logs. journald on a disposable instance is a buffer, not storage. Being honest about that is better than pretending:

# /etc/systemd/journald.conf on an instance that ships its logs
# Storage=volatile
# RuntimeMaxUse=200M

Storage=volatile keeps the journal in /run only. It makes the instance faster and the loss explicit, and it forces the question “is the shipper actually working” to be answered before you need the logs rather than after. The cost is that if the shipper fails, the logs are gone entirely - so pair it with an alert on the shipper, not merely on the application.

Caches. Genuinely rebuildable state can stay local, but “rebuildable” is not “free”. A replaced instance with a cold cache serves slower and pushes more load to the tier behind it. During a rolling replacement of a whole fleet, that is a sustained capacity event, and it is the usual explanation for a deploy that looked fine in staging and caused a latency excursion in production. Replace in smaller batches, and let each batch warm before the next - which is what the checkpoint settings on a rolling refresh are for.

Machine identity. A replaced instance has new SSH host keys, so every operator gets a host key warning and eventually learns to type yes without reading it. That is a real security regression caused by the deployment model. SSH certificates fix it properly - clients trust a CA rather than individual host keys - and that is covered in linux-ssh-certificates-and-bastions.

The spool directory nobody classified

An application accepts a webhook, writes it to /var/lib/myapp/outbox/, and a worker delivers it within a few seconds. Under normal conditions the directory is empty, so it never appeared in anyone’s list of state.

Then a downstream service is slow, the outbox grows to a few thousand messages, and an autoscaling refresh replaces the instance. The messages are gone. Nothing logs an error - the application had already returned 200 to the sender, and the instance that held the backlog no longer exists.

This is the characteristic immutable-infrastructure data loss: it needs two independent things to happen at once, so it does not appear in testing, and it is silent because the component that would have complained was deleted.

Find these before they find you. Any directory that accumulates files under load and drains under normal conditions is a queue, whatever it is called, and a queue on a disposable root disk is a data-loss defect. Either move it to a volume with its own lifecycle or use a queue service that acknowledges after delivery rather than after acceptance.

Stateful services on disposable compute

The pattern is not “immutable does not apply to databases”. It is replace the compute, keep the data: the instance is disposable, the volume attached to it is not.

The mechanics:

Data lives on a block volume whose lifecycle is independent of the instance - explicitly not delete-on-termination.
Replacing the instance means detach, terminate, launch from the new image, attach, mount by UUID, start the service.
The image contains the database software and its configuration. The volume contains the data. Neither contains the other.

The mount must be by UUID and must tolerate a missing volume at boot, or the replacement instance will fail to boot rather than fail to mount - a distinction that matters at three in the morning:

UUID=$(sudo blkid -s UUID -o value /dev/disk/by-id/DEVICE_ID)
echo "UUID=$UUID /var/lib/pgsql xfs defaults,nofail,x-systemd.device-timeout=30 0 0" \
  | sudo tee -a /etc/fstab

nofail keeps the boot going; x-systemd.device-timeout stops systemd waiting the full default timeout for a device that is not coming.

Severity: data-loss risk - one volume, one instance

The detach-and-reattach dance has a failure mode that destroys data rather than losing it. A block volume attached to two instances that both mount a non-cluster filesystem produces immediate, unrecoverable corruption: each kernel caches metadata it believes it owns exclusively, and each writes over the other’s structures.

This happens in immutable pipelines specifically because the new instance is launched before the old one is confirmed gone. The old instance is unresponsive, the automation assumes it is dead, it attaches the volume elsewhere, and the old instance comes back.

The requirement is the same one clustering has always had: confirm the old instance cannot write before the new one attaches. Terminate it through the API and wait for the terminal state, or fence it. An unresponsive instance is not a stopped instance. linux-two-nodes-mounting-the-same-block-risk covers why this is unrecoverable rather than merely bad.

For a replicated database, the replacement is a failover, not a deploy:

Launch a new instance from the new image as a replica.
Let it replicate until it is caught up. This is dominated by the data volume, not by the image, and can take hours.
Promote it, redirect clients, retire the old primary.

Everything an image pipeline gives you still applies - the software is baked and tested, the replacement is a launch rather than an in-place upgrade, and rollback is the previous image. What changes is the timing: you cannot roll a database fleet the way you roll a web tier, and treating them the same way is how a routine deploy becomes an outage.

In-flight work

Replacement also discards work that is in progress, which is state with a very short lifetime:

Requests being served. Deregister from the load balancer, wait for the connection-draining period, then stop. A replacement that skips the drain returns errors to real users for every deploy, at a rate low enough that nobody attributes it to the deploy.
Long-running jobs. A worker that has claimed a job and is eight minutes into it needs to either finish or return the job to the queue. TimeoutStopSec must be long enough for the first, and the job protocol must support the second.
Buffered telemetry. A metrics agent with a 60-second flush interval loses up to 60 seconds of data per replacement. Usually acceptable, occasionally not - and worth knowing rather than discovering during an incident review.

Audit the machine first

Where each class of state belongs

Stateful services on disposable compute

In-flight work

Knowledge check