Restore services and configuration - the full-system recovery · Linux

Restoring a single file is common; restoring an entire service is rarer but higher-stakes. This lesson covers the full procedure.

The full restore procedure

For a complete service restore:

Identify the dependencies: packages, services, configuration, data.
Restore configuration: /etc, /usr/local/etc.
Restore data: databases, document stores, user data.
Restore secrets: certificates, keys.
Start the service.
Verify: smoke tests, monitoring, real traffic.
Document: every step and any deviations.

Identify dependencies

Before restoring, list what the service needs:

# Package list
dpkg --get-selections > /backup/package-list.txt    # Debian
rpm -qa > /backup/package-list.txt                  # RHEL

# Service dependencies
systemctl list-dependencies my-service

# Network dependencies
ss -tlnp    # listening ports

The backup should include:

Configuration in /etc and /usr/local/etc.
Service-specific data in /var/lib, /srv, or application location.
Secrets in a vault (not in the backup).
Package list (for reinstalling the OS if needed).

Restore configuration

Restoring /etc over a running system is the single most irreversible thing in this lesson: it can replace the config that lets you log in, with the session you would need to fix it. Four steps, in this order.

# 1. Dry run FIRST. Read the file list before anything is written.
rsync -aAXH --numeric-ids --dry-run --itemize-changes \
      backup@backup-host:/backup/etc/ /etc/

# 2. Snapshot the current /etc so the restore is reversible
sudo tar --acls --xattrs --selinux --numeric-owner \
     -czf /root/etc-pre-restore-$(date +%F-%H%M).tar.gz -C / etc

# 3. Restore with metadata, then relabel
sudo rsync -aAXH --numeric-ids backup@backup-host:/backup/etc/ /etc/
sudo restorecon -R /etc          # RHEL family: without this, sshd will not start

# 4. Validate syntax BEFORE you lose the session that can fix it
sudo sshd -t && sudo visudo -c && sudo systemctl daemon-reload

Keep the console open until a NEW login succeeds

Steps 1, 2 and 4 exist because of the same failure: a /etc restore that lands wrong takes away the way back in.

--dry-run --itemize-changes shows you what would be replaced. Read it. A restore from a backup taken before a network reconfiguration will happily put back the old sshd_config, nftables.conf and sssd.conf.

rsync -a alone drops ACLs, extended attributes and SELinux labels — the contexts are xattrs, and -a does not include -X. Ownership and modes look right afterwards, so nothing appears wrong until sshd refuses to start with AVC denials. -aAXH --numeric-ids plus restorecon is the form that survives an enforcing host.

--numeric-ids matters when restoring onto a rebuilt host: without it, rsync maps by name, and a UID that was allocated differently on the new system silently reassigns ownership.

After systemctl reload sshd, keep the console session open and open a new SSH connection from a second terminal. The session you already have is an established connection and proves nothing about whether a fresh login still works.

Verify against the source, not against an undefined third path:

rsync -aAXH --numeric-ids --dry-run --itemize-changes \
      backup@backup-host:/backup/etc/ /etc/    # a clean run prints nothing

For application-specific configuration, restore from the same backup.

Restore data

For databases, use the application’s restore tool. The order of operations matters more than the tool.

Never drop the live database before the dump has been proven good

The procedure that turns a recoverable incident into permanent data loss looks like this:

# WRONG - do not do this
psql -c "DROP DATABASE mydb;"
createdb mydb
pg_restore -d mydb /backup/mydb-2026-08-09.dump

DROP DATABASE succeeds instantly and cannot be undone. Only then do you find out whether the artefact is any good. If the dump is truncated, half-written by a job that failed at 03:00, encrypted with a key nobody has, taken from the wrong host, or in a format pg_restore will not read, you have just destroyed the last surviving copy of the data you were trying to recover. Under incident pressure, at 03:00, the operator runs the documented procedure verbatim - so the procedure must not contain this shape.

The safe pattern has three phases: prove the artefact, restore beside the live database, then swap.

# 1. PROVE the artefact before destroying anything
pg_restore -l /backup/mydb-2026-08-09.dump | head   # readable? right objects?
ls -lh /backup/mydb-2026-08-09.dump                 # plausible size?

# 2. Restore into a NEW database - the live one is untouched
createdb mydb_restore
pg_restore -d mydb_restore --exit-on-error /backup/mydb-2026-08-09.dump
psql -d mydb_restore -c 'SELECT count(*) FROM users;'   # sanity check

# 3. Only after verification, swap the names
psql -c 'ALTER DATABASE mydb RENAME TO mydb_old;'
psql -c 'ALTER DATABASE mydb_restore RENAME TO mydb;'
# Keep mydb_old until the service has run under real traffic.

# MySQL equivalent
gunzip -t /backup/mydb-2026-08-09.sql.gz    # archive intact?
mysql -e 'CREATE DATABASE mydb_restore;'
gunzip -c /backup/mydb-2026-08-09.sql.gz | mysql mydb_restore
mysql -e 'SELECT count(*) FROM mydb_restore.users;'
# verify, then RENAME and swap - do not DROP first

# Files - -aAXH --numeric-ids, not -a: application data carries ACLs and
# SELinux labels too, and -a drops both
rsync -aAXH --numeric-ids backup@backup-host:/backup/data/ /var/lib/myapp/

Rehearse the failure path, not just the happy path. A restore drill that only ever uses a known-good dump never teaches the operator what a bad artefact looks like, which is precisely the situation in which they will reach for this page.

Application data restores are app-specific; check the app’s documentation.

Restore secrets

Secrets are typically in a vault, not in the backup. Restore from the vault:

# HashiCorp Vault
vault kv get -format=json secret/myapp > /etc/myapp/secrets.json
chmod 600 /etc/myapp/secrets.json

# AWS Secrets Manager
aws secretsmanager get-secret-value --secret-id myapp/prod > /tmp/secret.json
jq -r '.SecretString' /tmp/secret.json > /etc/myapp/secrets.json
chmod 600 /etc/myapp/secrets.json

Never store secrets in backups without additional encryption.

Start the service

sudo systemctl start my-service
sudo systemctl status my-service
sudo journalctl -u my-service -n 50

If the service fails to start:

Check logs: journalctl -u my-service -n 100.
Check dependencies: missing package, missing config.
Check permissions: wrong owner on a file.
Check network: not listening, can’t connect to database.

Verify

After start, verify the service works:

# Smoke test
curl -I http://localhost/health

# Synthetic monitoring (already in place)
# Should show the service as up

# Real traffic
# Wait for the load balancer to send real traffic

# Compare with baseline
# Logs, metrics, etc. should look like the baseline

Common pitfalls

Wrong package version: backup is from an older version.
Missing config files: not everything was in the backup.
Wrong network config: ifconfig vs netplan vs NetworkManager.
Hostname mismatch: restored config has old hostname.
User/group mismatch: LDAP or local users different.