Lab: HA design tradeoffs - choosing the right pattern

This lab designs an HA topology for a sample workload and documents the tradeoffs.

Tasks

Task 1: Define the workload

Sample workload:

• Web application: nginx + Node.js.
• Database: PostgreSQL.
• Steady-state load: 1000 RPS.
• Target availability: 99.95% (4 hours downtime per year).
• Region: single cloud region, multi-AZ.

Task 2: Identify failure domains

For this workload:

• AZ failure: AZ goes offline. Probability medium.
• Region failure: rare but possible.
• Application bug: deploy issue. Probability medium.
• Database corruption: data issue. Probability low.
• Network partition: AZ-to-AZ latency spike. Probability low.

Task 3: Design the topology

For each service:

• nginx: 2 instances in different AZs to start with. Load balancer with health checks. Task 4 will tell you whether 2 is actually enough.
• Node.js: 2 instances, stateless, behind nginx. Same caveat.
• PostgreSQL: primary in AZ1, replica in AZ2. Automatic failover with Patroni or similar.
• Object storage: S3 or equivalent. Multi-AZ.
• Load balancer: managed service (ALB, etc.).

Task 4: Capacity plan

N+1 sizing is judged on the survivors, never on the fleet as a whole. State the rule before you use it:

N+1 utilisation = peak demand / (total capacity - largest single unit)

The all-up figure — peak demand divided by total capacity — describes the state you are already in and are not worried about. It is not a capacity plan.

For 1000 RPS:

• nginx: 2 x 800 = 1600 total, 800 surviving. 1000 / 800 = 125% → UNDER-PROVISIONED. Use 3 x 800: 1600 surviving, 1000 / 1600 = 63%. Good.
• Node.js: 2 x 600 = 1200 total, 600 surviving. 1000 / 600 = 167% → UNDER-PROVISIONED. 3 x 600 gives 1200 surviving = 83%, which still misses the 80% target below. Use 4 x 600: 1800 surviving, 1000 / 1800 = 56%.
• PostgreSQL: primary 800 RPS, replica promoted to 800 RPS. 1000 / 800 = 125% → UNDER-PROVISIONED. Split the read traffic onto replicas so that the write-only load on a promoted primary stays under target, or size the pair for 1250 RPS each.

Required check on your own design: for every component, write out peak / (total - largest unit) and confirm it is under 80%. Any component that is not under 80% must be resized before you move on.

Task 5: Document the design

The capacity section must quote the N+1 figure, not the all-up figure. Every component line needs the surviving-capacity number and a verdict.

HA DESIGN
========
Workload: Web application
Peak demand: 1000 RPS

Components (sized after Task 4):
- nginx: 3 instances (AZ1, AZ2, AZ3), 800 RPS each
- Node.js: 3 instances (AZ1, AZ2, AZ3), 600 RPS each
- PostgreSQL: primary (AZ1), replica (AZ2), 1250 RPS each
- Object storage: S3 multi-AZ

Failure domains:
- AZ failure: tolerates (one instance in each AZ)
- Application bug: tolerates (rollback)
- Database corruption: tolerates (restore from backup)
- Region failure: does not tolerate (need DR site)

Capacity (N+1 = peak / (total - largest single unit)):
- nginx:      1000 / (2400 - 800)  =  63%  PASS (< 80%)
- Node.js:    1000 / (1800 - 600)  =  83%  FAIL (> 80%) -> 4 instances
- PostgreSQL: 1000 / (2500 - 1250) =  80%  BORDERLINE -> offload reads
- Steady state (all up): 42% per component - informational only
- Growth: 30% headroom for unplanned growth, applied to the N+1 figure

Tradeoffs:
- 3 instances per service: N+1 costs one extra unit per component
- 2 AZs in 1 region: tolerates AZ but not region
- Active-active database: more complex than active-passive
- Object storage: managed service for multi-AZ

Cost:
- 3 nginx: small
- 3-4 Node.js: small
- 2 PostgreSQL (primary + replica), sized for solo operation: larger
- Load balancer: managed, scales
- Object storage: scales with usage

Note the Node.js line. It reads FAIL at 83%, which is the point: the document is a gate, not a summary. A component that fails its N+1 check goes back to Task 4 and gets resized before the design is signed off.