Bare-metal recovery - restoring to a fresh host

Bare-metal recovery is restoring a host from nothing - new hardware, blank disks, or a new cloud instance. This lesson covers the procedure.

When you need bare-metal recovery

• Hardware destroyed (fire, flood, theft).
• Disk corruption beyond repair.
• New region / new data centre.
• Migrating to a new platform.

The procedure is the most thorough restore: nothing remains from the previous host.

The procedure

1. Provision new hardware (or cloud instance).
2. Install the operating system (autoinstall or manual).
3. Configure the network (IP, hostname, DNS).
4. Apply configuration management (Ansible, Puppet).
5. Restore data from backup.
6. Restore secrets from vault.
7. Start services.
8. Verify (smoke tests, monitoring).
9. Update DNS and load balancer to point to new host.
10. Decommission old host.

Time depends on automation:

• Fully automated: 30-60 minutes.
• Partially automated: 2-4 hours.
• Manual: 1-2 days.

Automate the install

For Debian/Ubuntu, use cloud-init or autoinstall:

# cloud-init user-data
#cloud-config
users:
  - name: deploy
    lock_passwd: true                  # key auth only, no password to guess
    ssh_authorized_keys:
      - ssh-ed25519 AAAA...

write_files:
  # Scope the grant to the commands the deploy account actually runs.
  - path: /etc/sudoers.d/20-deploy
    permissions: '0440'
    owner: 'root:root'
    content: |
      Cmnd_Alias DEPLOY_SVC = /usr/bin/systemctl restart myapp.service, \
                              /usr/bin/systemctl reload nginx.service, \
                              /usr/bin/systemctl status myapp.service
      deploy ALL=(root) NOPASSWD: DEPLOY_SVC

packages:
  - openssh-server
  - postgresql-15
  - nginx

runcmd:
  # An invalid drop-in is skipped silently, taking the grant with it.
  # Fail the build here rather than discovering it during an incident.
  - visudo -c -f /etc/sudoers.d/20-deploy
  # The OpenSSH unit is ssh.service on Debian/Ubuntu and sshd.service on
  # RHEL/Rocky/Alma. Detect it rather than hardcoding one: a rebuild
  # template that enables the wrong name comes up with no SSH at all,
  # which on a bare-metal recovery means another trip to the console.
  # Each runcmd string is its own shell, so this has to be one line.
  - for u in ssh.service sshd.service; do systemctl cat "$u" >/dev/null 2>&1 && { systemctl enable --now "$u"; break; }; done

For RHEL family, use kickstart:

# kickstart (RHEL 9)
url --url="http://mirror.example.com/rhel/9/"
rootpw --iscrypted $6$...          # SHA-512; generate with: openssl passwd -6
network --bootproto=dhcp
ignoredisk --only-use=sda          # nothing outside this list is touched
clearpart --all --initlabel --drives=sda
part / --fstype="xfs" --grow --ondisk=sda

Three details in that file are the difference between a rebuild and a data-loss incident.

• clearpart --all with no --drives=. Bare clearpart --all wipes the partition table of every block device the installer can see. On a database or storage node that is the attached SAN LUN, the iSCSI target, and the second data disk — the volumes you were about to restore onto. Always constrain with --drives=, and pair it with ignoredisk --only-use= so the restriction is stated twice. Better still, detach the data LUNs before the rebuild.
• rootpw --iscrypted $1$.... $1$ is MD5-crypt: fast, unsalted by modern standards, trivially crackable, and not acceptable on RHEL 8 or 9. Use SHA-512 ($6$) or yescrypt ($y$), generated with openssl passwd -6. Kickstart accepts the MD5 hash without complaint, so nothing warns you.
• The install directive. Removed from kickstart in RHEL 8. A file that still carries it is a file nobody has run against a supported release.

Configuration management

Apply the host’s role with configuration management:

ansible-playbook -i new-host site.yml

The Ansible playbook applies:

• User accounts.
• Package installation.
• Service configuration.
• Firewall rules.
• Monitoring agents.

Restore data

# Mount backup
mount backup-host:/backups /mnt/backup

# Restore data
rsync -a /mnt/backup/new-host/ /var/lib/myapp/

# Or BorgBackup
borg extract /mnt/backup/repo::latest

Restore secrets

# Vault
vault kv get -format=json secret/myapp > /etc/myapp/secrets.json
chmod 600 /etc/myapp/secrets.json

# Or AWS Secrets Manager
aws secretsmanager get-secret-value --secret-id myapp/prod

Validate

# Smoke tests
curl -I http://localhost/health

# Data verification
psql -c "SELECT count(*) FROM users;"

# Performance baseline
# Run your standard performance tests

# Compare with peer hosts
# Same workload, same latency, same throughput

Update DNS and load balancer

# Remove old host from DNS
nsupdate <<EOF
server dns.example.com
zone example.com
update delete myapp.example.com A 10.0.0.10
send
EOF

# Add new host: the grammar is
#   update add <name> <ttl> [class] <type> <data>
nsupdate <<EOF
server dns.example.com
zone example.com
update add myapp.example.com 300 A 10.0.0.20
send
EOF

The TTL comes before the record type. update add myapp.example.com A 86400 10.0.0.20 transposes the two, and nsupdate rejects the whole update — quietly enough that an operator watching a scrolling terminal misses it. The cutover then simply does not happen: traffic keeps going nowhere while everyone stares at the new host wondering why it is idle. Always read nsupdate’s output, and confirm with dig +short myapp.example.com @dns.example.com.

Use a short TTL at cutover, and lower it in advance. Publishing a 86400s TTL during a recovery pins the new answer in every resolver cache for a day, so the next correction — including rolling back — takes a day to propagate. Drop the record to 300s at least one old-TTL period before a planned cutover, and raise it again once the dust settles.

Or update the load balancer pool.

Decommission the old host

# Stop services
sudo systemctl stop myapp
sudo systemctl disable myapp

# Detach from cluster
# (Cluster-specific commands)

# Remove from monitoring
# (Monitoring-specific)

Then, and only then, the wipe. Identify before you destroy:

# Substitute your own values before running:
SERIAL=S3Z9NX0M123456

# 1. Prove which host this session is on.
hostname; cat /etc/machine-id

# 2. Prove which device you mean, by serial - not by /dev/sda.
lsblk -o NAME,SIZE,SERIAL,MOUNTPOINT
ls -l /dev/disk/by-id/

# 3. Wipe the serial-verified path.
sudo blkdiscard /dev/disk/by-id/ata-VENDOR_MODEL_"$SERIAL"
# Or, for media without discard: sudo shred -v -n 1 /dev/disk/by-id/...

# 4. Power off.
sudo shutdown -h now

Knowledge check