high riskservice affecting~30 min

Runbook: Reboot a production server

1 · Prerequisites

Confirm every item is in place before any state change.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Confirm out-of-band console (BMC/iDRAC/iLO/hypervisor) access works NOW, before the reboot, not after
· Confirm the change window and that the change is approved
· Confirm every service on the host is enabled to start at boot - systemctl is-enabled for each
· Confirm /etc/fstab is valid: mount -a on the running system must be a no-op with no errors
· Confirm the intended kernel is the default boot entry, and that the previous kernel is still installed
· Confirm a current backup or snapshot exists and its age is acceptable
· Confirm whether the host is a cluster node or holds a VIP - if so, drain it first

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Announce the window and silence the monitoring for this host only
2Prove the host will come back: fstab, boot entry, enabled units, no pending config errors
3Drain traffic: cluster standby, load-balancer removal, or VIP failover
4Stop stateful services cleanly rather than letting the shutdown do it
5sync, then reboot with the console open
6Watch the boot on the console; do not wait blindly for SSH
7Validate: mounts, services, kernel, listeners, application health
8Return traffic and remove the drain
9Un-silence monitoring and record the reboot

4 · Verification

Confirm the procedure actually fixed the problem.

✓The host booted the intended kernel (uname -r matches the plan)
✓Every filesystem in fstab is mounted (findmnt --verify and a mount count comparison)
✓systemctl --failed is empty
✓Every service that was listening before is listening again (ss -lntup comparison against the pre-reboot capture)
✓The application answers its own health check from outside the host
✓Cluster node is Online and out of standby; VIP is where it should be
✓Monitoring is un-silenced and green

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶If the host does not boot, use the console and select the previous kernel from the GRUB menu
↶If a filesystem fails to mount, boot to emergency mode from the console and correct /etc/fstab - never leave a bad fstab in place
↶If a service does not start after the reboot, treat it as a separate incident and use the failed-service runbook
↶If the change cannot be completed, restore traffic to the other nodes and leave this host drained rather than half-returned

6 · Escalation

When the runbook isn't enough, contact:

· Host does not reach the bootloader: escalate to hardware/vendor with the BMC event log
· Root filesystem was demoted read-only by the kernel before or after the reboot: escalate; do not remount read-write
· The host is the only instance of a service with no redundancy: escalate for a change of plan, not a faster reboot
· Console access is unavailable: stop. Do not reboot a production host you cannot reach without SSH.

Rebooting is easy. Coming back is the hard part, and the faults that stop a host coming back — a bad fstab line, a service that was never enabled, a kernel that does not boot — are all discoverable before you type reboot. That is what this runbook is for.

Step 1: Prove the host will come back

Every check in this step is read-only and takes seconds. Skipping them is where reboot incidents come from.

Read-only / Safefstab validation— Validates mount configuration without mounting anything new.

# Syntax and resolvability of every fstab entry
findmnt --verify --verbose

# Everything in fstab is already mounted: this must print nothing
sudo mount -a -n -v 2>&1 | grep -v 'already mounted'

# Every device referenced still exists
lsblk -o NAME,UUID,LABEL,MOUNTPOINT
grep -vE '^\s*#|^\s*$' /etc/fstab

A stale UUID in /etc/fstab is the classic reboot killer. The filesystem is mounted right now because it was mounted before the disk was replaced or the volume renamed. At boot, systemd cannot find it, the mount unit fails, and the host drops to emergency mode waiting for a root password on a console you may not have.

Read-only / Safeboot readiness— Reads unit enablement, kernel entries and pending config errors.

# Will the services actually start at boot? "active" is not "enabled".
systemctl list-unit-files --state=enabled --no-pager | grep -E 'nginx|postgres|myapp'
systemctl is-enabled nginx postgresql myapp

# Anything already broken
systemctl --failed --no-pager
sudo systemd-analyze verify /etc/systemd/system/*.service 2>&1 | head

# Which kernel will boot, and is the previous one still there
uname -r
ls /boot/vmlinuz-*
sudo grubby --default-kernel 2>/dev/null || grep -E '^GRUB_DEFAULT' /etc/default/grub

systemctl is-enabled is the check people skip. A service started by hand months ago is running now and will not be running after the reboot.

Step 2: Capture the “before” picture

You cannot verify recovery without a baseline.

Read-only / Safebaseline capture— Records pre-reboot state to compare against afterwards.

B=/var/tmp/prereboot-$(date -u +%Y%m%dT%H%M%SZ)
sudo mkdir -p "$B"
sudo sh -c "
ss -lntup            > $B/listeners.txt
systemctl list-units --type=service --state=running --no-pager > $B/services.txt
findmnt -lo TARGET,SOURCE,FSTYPE,OPTIONS > $B/mounts.txt
uname -r             > $B/kernel.txt
ip -br addr          > $B/addrs.txt
"
echo "$B"

Step 3: Drain

Cluster-wide riskdrain— Moves work off this host. Affects placement across the cluster or load balancer.

# Pacemaker node: move resources off and wait for it to finish
sudo pcs node standby $(hostname -s) --wait=300
pcs status --full

# Load-balancer backend: drain rather than remove, so sessions finish
echo "disable server web/web1" | sudo socat stdio /run/haproxy/admin.sock

# Keepalived VIP holder: let the peer take the VIP
sudo systemctl stop keepalived
ip -br addr | grep -F '<vip>' || echo 'VIP released'

Wait for the drain to complete before continuing. A --wait that times out means resources did not move, and rebooting anyway is an outage.

Step 4: Stop stateful services cleanly

The shutdown sequence will stop services, but on its own schedule and with a timeout. Databases and queues deserve a deliberate stop while you are watching.

Service impact possibleclean stop— Stops services in a controlled order before the reboot.

sudo systemctl stop myapp.service
sudo systemctl stop postgresql
systemctl is-active postgresql || echo 'stopped'

# Flush pending writes
sync

Step 5: Reboot, with the console open

Service impact possiblereboot— Restarts the host. All local services stop. Requires console access to recover from a failed boot.

# Open the console FIRST, in another window
# sudo ipmitool -I lanplus -H <bmc-ip> -U admin -f /etc/bmc-pw sol activate

sudo systemctl reboot

Prefer systemctl reboot over reboot: it runs the full shutdown transaction rather than short-circuiting it.

Watch the console. The two failures you are watching for both happen before SSH is available:

The boot stops at a GRUB prompt or a firmware menu.
The boot reaches “Give root password for maintenance”, which means a mount unit failed.

Step 6: Validate against the baseline

Read-only / Safepost-reboot validation— Compares the recovered host against the captured baseline.

B=/var/tmp/prereboot-<timestamp>

uname -r; cat $B/kernel.txt
systemctl --failed --no-pager

# Mounts: every fstab entry present
findmnt --verify
diff <(findmnt -lo TARGET,SOURCE,FSTYPE,OPTIONS) $B/mounts.txt

# Listeners: anything missing is a service that did not come back
diff <(ss -lntup) $B/listeners.txt

# Addresses
diff <(ip -br addr) $B/addrs.txt

diff against the baseline finds the service nobody remembers. A listener present before and absent after is the whole check.

Read-only / Safeapplication health— Exercises the service the way a client does, from outside the host.

curl -sS -o /dev/null -w '%{http_code}\n' https://app.example.com/healthz
sudo -u postgres psql -c 'SELECT 1' >/dev/null && echo 'db OK'
journalctl -b -p err --no-pager | tail -30

Step 7: Return traffic

Cluster-wide riskundrain— Returns the host to service. Affects placement across the cluster or load balancer.

sudo pcs node unstandby $(hostname -s) --wait=300
pcs status --full

echo "enable server web/web1" | sudo socat stdio /run/haproxy/admin.sock
sudo systemctl start keepalived

# Leftover constraints from any move during the window
sudo pcs constraint config --full | grep -i 'cli-ban\|cli-prefer'

Then un-silence monitoring — and let it confirm the host is healthy rather than confirming it yourself.

Common patterns

Symptom	Likely cause	Resolution
Boot stops at “Give root password for maintenance”	A mount unit failed; usually a stale UUID in fstab	Console, fix fstab, consider `nofail` for non-essential mounts
Host up, a service missing	It was running but never `enabled`	`systemctl enable`; that is why Step 1 checks it
Booted the wrong kernel	`GRUB_DEFAULT` not `saved`, or a one-shot entry consumed	Set the default deliberately; verify with `uname -r`
SSH never returns, console shows a login prompt	Network config did not apply, or the interface renamed	Console; check `networkctl status`, `ip -br addr`
Resources did not come back to the node	Still in standby, or a leftover `cli-ban`	`pcs node unstandby`; `pcs constraint config --full`
Boot hangs waiting on a network mount	No `nofail`/`_netdev` and the server is unreachable	Add `_netdev,nofail,x-systemd.device-timeout=`
Application starts before its data mount	Mount ordering not expressed	`RequiresMountsFor=` on the unit