Redundancy and fault tolerance - building for failure

Redundancy is the practice of having more than one of every component. Fault tolerance is what happens when one fails. This lesson covers the patterns and trade-offs.

Redundancy at each layer

The rule: every single point of failure (SPOF) is a candidate for redundancy.

Active-passive vs active-active

• Active-passive: one node is active, the other is standby. On failure, standby takes over. Examples: VRRP, Pacemaker, DRBD.
• Active-active: both nodes serve traffic. On failure, the survivor handles the load. Examples: HAProxy pair, database replication with read replicas.

Active-active has higher utilisation but is more complex (shared state, split-brain risk). Active-passive is simpler but wastes resources.

Fault tolerance patterns

Failover

The classic pattern. Active node fails, standby takes over.

• Pros: simple.
• Cons: brief downtime during failover.

Load balancing with health checks

Multiple instances behind a load balancer. Failed instances are removed from rotation.

• Pros: automatic, no manual intervention.
• Cons: instances must be stateless or share state.

Active-active with quorum

Multiple nodes that vote on which is the leader. On failure, the remaining nodes form a new quorum.

• Pros: no downtime.
• Cons: requires an even number of nodes (typically 3 or 5).

Geographic replication

Multiple regions, each with its own infrastructure. On regional failure, traffic shifts to another region.

• Pros: DR-grade availability.
• Cons: expensive, complex.

Apply to Linux

For a Linux service:

Service (e.g. nginx):
- 2+ instances behind a load balancer
- Health checks: TCP connect, HTTP /health
- Session: sticky sessions or shared session store

Database:
- Primary + replica
- Replica promoted on primary failure
- Application reconnects to new primary

Storage:
- RAID 1/10 for local redundancy
- Replication to a second host for off-host redundancy

Each layer is redundant; failure of one component is absorbed by the others.

Over-redundancy

More redundancy is not always better:

• 3-way replication costs 3x the storage of 1-way.
• Multi-region costs 2-3x the cost of single-region.
• Each redundant component is a potential failure point.

Match redundancy to the availability target. 99% does not need multi-region; 99.99% might.

Knowledge check