Runbook: Keepalived failover - test and recover
1 · Prerequisites
Confirm every item is in place before any state change.
2 · Pre-checks
Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.
- · Verify keepalived is running on both hosts
- · Verify the VIP is on the expected host
- · Verify health checks work
- · Verify the keepalived configuration matches production
3 · Procedure
Execute each step in order. Verify the expected output of a step before moving to the next.
- 1Identify the current state (master / backup, VIP location)
- 2If the master is down: verify the backup took the VIP
- 3If the backup did not take the VIP: check keepalived logs
- 4If the service is broken on the new master: investigate
- 5Bring the master back when ready
- 6Verify the VIP returns to the master (preemption)
- 7Document the test or incident
4 · Verification
Confirm the procedure actually fixed the problem.
- ✓VIP is on the expected host
- ✓Service works on the new host
- ✓Preemption works
- ✓No data corruption
5 · Rollback
If verification fails, undo the procedure in reverse order.
- ↶If the failover was incorrect, force the VIP back to the original master
- ↶Investigate the cause (network blip, false positive health check)
- ↶Re-test the failover
6 · Escalation
When the runbook isn't enough, contact:
- · If keepalived does not work on the new master, escalate to the network team
- · If the service is broken on the new master, escalate to the application team
- · If the failover is happening repeatedly, escalate to the cluster team
This runbook tests and recovers from a keepalived failover. The test verifies the configuration; the recovery is the actual failover.
When to use this runbook
Use this runbook when:
- A VIP needs to be moved (planned failover).
- A test of the failover is required (quarterly).
- Recovery from a keepalived failure.
Inputs
Gather before starting:
- VIP address.
- Master and backup host names.
- Maintenance window (for testing).
- Communication channels (war room, on-call).
Procedure
Step 1: Capture the current state
# On the master
ip addr show eth0 | grep <vip>
sudo journalctl -u keepalived -n 20Document: which host has the VIP, when the last failover was, any recent changes.
Step 2: Choose the failover method
- Graceful: stop keepalived on the master; backup takes over via preemption. Used for planned failover.
- Health check: kill the service on the master; backup takes over via health check. Used for service-failure testing.
- Forced: directly demote the master in keepalived config. Used for emergencies.
Step 3: Test graceful failover
# Stop keepalived on the master
sudo systemctl stop keepalived
# Verify the backup took the VIP
ssh backup "ip addr show eth0 | grep <vip>"The VIP should now be on the backup.
Step 4: Test health check failover
# Stop the service on the master (not keepalived)
sudo systemctl stop myservice
# Verify the backup took the VIP after the check fails
ssh backup "ip addr show eth0 | grep <vip>"The VIP should migrate to the backup after the health check
fails (typically after fall failures, e.g. 3).
Step 5: Verify the service
# The service should be reachable on the VIP
curl -I http://<vip>/healthIf the service does not work, investigate:
- Is the service running on the new master?
- Is the network reachable?
- Is the firewall allowing traffic?
Step 6: Bring the original master back
# Restart the service on the original master
sudo systemctl start myservice
# Restart keepalived
sudo systemctl start keepalivedBy default, the original master (higher priority) takes back
the VIP. With nopreempt, the current master keeps it.
Step 7: Document
In the run log:
- Time of test.
- Method (graceful, health check, forced).
- Time to recover.
- Findings.
- Improvements.
Common patterns
| Symptom | Likely cause | Resolution |
|---|---|---|
| VIP did not migrate | Health check did not fail | Check check script |
| VIP on wrong host | Priority wrong | Verify priorities |
| Preemption did not happen | nopreempt is set | Remove or wait |
| Backup not taking over | STONITH / network issue | Verify network and keepalived logs |
Knowledge check
Knowledge check · 3 questions
Q1. What is the first step in testing keepalived failover?
Q2. A keepalived failover is only needed in production if there are real failures.
Q3. Which of the following are valid keepalived test methods? Select all that apply.
Passing score: 75%. Answers are checked in this browser.