Instance lifecycle - reboot, stop/start, terminate, replace

On a physical server there are two states that matter: running and not running. In a cloud the list is longer, the transitions are triggered by an API rather than by a person in the room, and each transition destroys a different subset of the machine.

The names differ per provider - AWS says stop, terminate; Azure says deallocate, delete; GCP says stop, delete - but the underlying model is the same everywhere, and it is worth learning as a model rather than as three vocabularies.

The four transitions and what they cost

Three consequences follow, and each one has cost somebody a night.

A reboot is not a stop/start. The guest calls reboot, the virtual machine restarts on the same hypervisor host, and the local disk is still there. This is why ephemeral storage looks reliable in testing: the only transition anyone exercises in testing is a reboot. A stop/start moves the instance to another host, and the local disk it had was on the old one.

A stop/start keeps the instance id. The machine came up from scratch, cloud-init ran again, and it still skipped every per-instance module - because the identity it keys on did not change.

A replacement is not a restart. Nothing that was on the root disk exists any more. Not the logs, not the config somebody edited by hand, not the packet capture you were halfway through.

Reading the transition from inside the guest

The guest can tell you which of these just happened.

# When did this kernel start, and how many boots does the journal have?
uptime --since
journalctl --list-boots | tail -3

# Did the machine change identity, or just restart?
cat /var/lib/cloud/data/instance-id
cat /var/lib/cloud/data/previous-instance-id

$ journalctl --list-boots | tail -3

 -2 4605076f366744e0bb4e345a1878bd80 Mon 2026-08-03 14:38:09 UTC Mon 2026-08-03 21:26:28 UTC
-1 3cf0bb394749407c89159992bdbf349c Mon 2026-08-03 21:27:47 UTC Fri 2026-08-07 10:01:01 UTC
0 1a4832f913bf413db46f7727d0c9174e Fri 2026-08-07 10:01:11 UTC Tue 2026-08-11 15:32:02 UTC

More than one boot in the journal means the root disk survived a restart - a fresh instance has exactly one. That single fact separates “this machine rebooted” from “this is a new machine with the same name in your inventory”, and it is the first thing to check when a host you have been debugging suddenly looks healthy and empty.

The cloud-init side is just as direct:

$ cat /var/lib/cloud/data/instance-id /var/lib/cloud/data/previous-instance-id

iid-datasource-none
NO_PREVIOUS_INSTANCE_ID

NO_PREVIOUS_INSTANCE_ID means this is the first boot cloud-init has recorded. On a real instance the two files hold the platform id, and a difference between them is exactly what makes cloud-init treat the boot as a first boot and re-run the per-instance modules.

You can see the record it keeps:

ls /var/lib/cloud/instances/
ls /var/lib/cloud/instances/*/sem/ | head
ls /var/lib/cloud/sem/

Each file under instances/<id>/sem/ is one completed per-instance module. /var/lib/cloud/sem/ holds the per-once semaphores, which are deliberately outside the per-instance directory so they survive an instance id change.

Shutdown is a deadline, not a request

When the platform stops an instance it does not pull the plug immediately. It signals an orderly shutdown - usually an ACPI power-button event, which systemd-logind turns into poweroff.target - waits, and then hard-kills the virtual machine when the timer expires. The wait is a small number of minutes and you do not control it.

Linux then has to finish shutting down inside that window, and systemd has its own timers underneath:

$ systemctl show -p DefaultTimeoutStopUSec

DefaultTimeoutStopUSec=1min 30s

A service that ignores SIGTERM gets 90 seconds and then SIGKILL. Stack two or three of those in series, add an inhibitor, and the OS is still shutting down when the platform stops caring:

$ systemd-inhibit --list

UPower                       0   root 1675527 upowerd         sleep    Pause device polling      delay
Unattended Upgrades Shutdown 0   root    1531 unattended-upgr shutdown Stop ongoing upgrades     delay

3 inhibitors listed.

Replacement, and what it deletes

An instance is replaced when an autoscaling group refreshes it, when a health check fails long enough, when a deploy rolls, or when the platform reclaims capacity. From inside the guest, replacement is indistinguishable from an unannounced power-off.

What goes with it:

• /var/log in its entirety - including the journal that explains why the instance was unhealthy.
• Any hand-edit somebody made while debugging. This is a feature: an undocumented fix cannot outlive its instance. It is also the trap, because the fix stops working the moment the fleet scales out and nobody connects the two events.
• Core dumps, perf output, packet captures, coredumpctl state - all the evidence, all on the root disk.
• Anything an application wrote to a local path: uploads, a SQLite file, a cache somebody assumed was durable.

The design answers are unglamorous and they all amount to do not keep it on the root disk: ship logs to a central collector as they are written, put application state on a volume with its own lifecycle, and treat local disk as scratch.

Termination notices

Most platforms publish an advance warning on the instance metadata service before they reclaim an instance - spot interruption notices, scheduled maintenance events, or a lifecycle hook that holds the instance in a terminating state until you release it. The endpoint differs per provider; the pattern does not.

A small poller is enough, and it belongs in the image:

#!/bin/bash
# Poll the platform's termination-notice endpoint. The path below is a
# placeholder - substitute the one your provider documents.
set -euo pipefail

NOTICE_URL="http://169.254.169.254/latest/meta-data/spot/instance-action"

while true; do
  if curl -sf --max-time 2 "$NOTICE_URL" >/dev/null; then
    logger -t drain "termination notice received, draining"
    systemctl stop myapp.service      # stop taking new work
    curl -sf -X POST https://lb.example.com/deregister/"$(hostname)" || true
    break
  fi
  sleep 5
done

Two minutes of warning is enough to deregister from a load balancer, finish in-flight requests and checkpoint. It is not enough to do any of that if the first anyone hears of it is the instance disappearing.

Knowledge check