Build and recovery procedures - the operational reference

Build and recovery procedures are the operational reference for bringing a service up and bringing it back. This lesson covers how to write and maintain them.

Build procedure

The build procedure is the steps to bring a service up from nothing:

1. Provision infrastructure (VMs, network, storage).
2. Install dependencies (packages, libraries).
3. Configure the application (config, secrets).
4. Deploy the application (artifact, container).
5. Verify the service is up (smoke test, health check).
6. Document the deployment.

BUILD PROCEDURE: <service>
================================

1. Provision:
   - VM: <spec>
   - Network: <vlan, IP>
   - Storage: <volume>

2. Install:
   - apt install <packages>

3. Configure:
   - /etc/<service>/<config>: <template>
   - secrets: <vault path>

4. Deploy:
   - systemctl enable <service>
   - systemctl start <service>

5. Verify:
   - curl http://<service>/health
   - check monitoring

Recovery procedure

The recovery procedure is the steps to bring a service back from a failure:

1. Assess the failure (is it the host, the service, or the data?).
2. Take the appropriate action (restart, failover, restore).
3. Verify recovery.
4. Document the incident.

RECOVERY PROCEDURE: <service>
==================================

1. Assess:
   - ping <host>
   - systemctl status <service>
   - journalctl -u <service> -n 50

2. Action:
   - If service down: systemctl restart <service>
   - If host down: failover to <backup>
   - If data lost: restore from backup

3. Verify:
   - curl http://<service>/health
   - check monitoring
   - smoke test

4. Document:
   - log incident
   - post-mortem

Use during incidents

The build and recovery procedures are the reference during incidents. The on-call follows the procedure to bring the service back.

The discipline:

• Procedures are stored in a known location (wiki, runbook repo).
• They are linked from alerts.
• They are version-controlled.
• They are tested during DR drills.

Keep current

The procedures must be current:

• After any infrastructure change, update the build procedure.
• After any incident, update the recovery procedure.
• Quarterly: review all procedures.

A stale procedure is worse than no procedure. The discipline: keep the procedures current.

Knowledge check