Failure domains - what fails together

A failure domain is a set of components that fail together. Designing across failure domains is the key to high availability. This lesson covers the common failure domains and how to design for them.

What is a failure domain

A failure domain is a unit of failure. Components within a failure domain share a risk; components across failure domains have independent risks.

Examples:

• All servers in one rack share the same power and network switch.
• All servers in one data centre share the same grid and cooling.
• All servers in one region share the same metro network.
• All instances of one application share the same code and config.
• All services behind one load balancer share the same load balancer.

Common failure domains

Design across failure domains

Redundancy must be across failure domains:

• Two servers in the same rack: not redundant (same power and network).
• Two servers in different racks: redundant for rack-level failure.
• Two servers in different data centres: redundant for site-level failure.
• Two services with the same code: not redundant (same bug).
• Two services with different code (e.g. primary and DR script): redundant for code-level failure.

The standard: separate the redundant components by at least one failure domain.

Test failure domain isolation

For each failure domain:

• Identify what could fail.
• Verify the redundant component takes over.
• Document the test.

Examples:

• Power failure: kill the primary’s PDU. Verify the secondary takes over.
• Rack failure: simulate a rack loss. Verify the service continues.
• Application failure: deploy a known-broken version. Verify the load balancer removes it.

Without testing, the redundancy is theoretical.

Knowledge check