Capstone project: design a Proxmox infrastructure

Why this matters in production

A learner who can produce a defensible architecture for a realistic business has demonstrated the curriculum’s competencies.

The scenario

You are the infrastructure architect for “Greenfield SaaS Inc”, a B2B SaaS company with:

• 150 employees, headquartered in Datacenter East.
• ~40 production VMs (Linux and Windows).
• Critical PostgreSQL database (peak 12k IOPS, 80 GB working set).
• Active Directory for identity.
• 24×7 operations required.
• RPO 15 min for critical systems; RTO 1 h for critical systems.
• Less critical services: longer RPO/RTO acceptable.
• A single host failure must not cause downtime.
• Backups must survive loss of the primary cluster.
• Security team requires MFA and least privilege.
• Rolling maintenance required (no full-cluster downtime windows).
• Expected growth: +40 % over three years.
• Budget envelope: ~$250k hardware, no annual support.

Your deliverable

Produce a design document covering:

1. Compute: node count, CPU/RAM per node, model choices.
2. Cluster: cluster size, quorum strategy (with or without QDevice).
3. Storage: backend choice (local ZFS, Ceph, SAN/NFS) with sizing.
4. Network: VLANs, traffic separation, redundant switches, OOB management.
5. HA: rules, capacity buffer for one node failure.
6. Backup: PBS topology, retention, off-site sync, encryption.
7. DR: documented recovery strategy.
8. Security: auth realms, RBAC roles, MFA, firewall zones.
9. Monitoring: what to monitor, SLOs, alerting thresholds.
10. Maintenance plan: rolling upgrades, capacity reviews, patching cadence.
11. Operational artefacts: daily/weekly/monthly checklists, runbooks, escalation tree.

For each decision, explain:

• What you chose.
• Why you chose it.
• What alternatives you considered.
• What trade-offs you accepted.

Model solution

Below is one defensible architecture. Multiple other architectures are valid; the goal is for you to justify your choices.

Architecture A: 3-node hyperconverged + Ceph

Compute:

• 3 identical nodes, 1-socket Xeon or EPYC, 16 cores, 128 GB RAM.
• Local storage: 2 × 1.92 TB NVMe (OS), 4 × 3.84 TB NVMe (Ceph), 1 × 960 GB NVMe (write log/SLOG-equivalent for Ceph).
• 10 GbE NICs: 1 management, 1× LACP bond for VM/migration, 1× Ceph public, 1× Ceph cluster.
• OOB: iDRAC on each node.

Cluster: 3 nodes; expected votes=3, quorum=2. No QDevice needed.

Storage: Ceph RBD on the NVMe OSDs. Default replicated pool size=3, min_size=2. ~24 TB raw, ~8 TB usable. Plenty for 40 VMs.

Network: VLAN-aware bridge on the management+VM bond; SDN VNet for the database tier; Ceph cluster network on the dedicated Ceph NICs.

HA: HA enabled on critical VMs with N+1 capacity buffer. Total committed memory plus N+1 headroom = 60% of physical.

Backup: PBS at a separate physical site, syncing hourly. Encrypted client-side. Retention: keep-daily=7, keep-weekly=4, keep-monthly=6.

DR: PBS off-site sync. Runbook for restoring from PBS. RTO ~2 hours; RPO ~1 hour.

Security: AD realm for human admins; MFA mandatory; tokens for automation; firewall allow-listed from management subnet.

Monitoring: Prometheus + Grafana via pve-exporter. Alerts on Ceph HEALTH_WARN, quorum loss, capacity thresholds, backup failures.

Maintenance: rolling upgrades one node at a time; quarterly capacity reviews; monthly DR tabletop; restore certification quarterly.

Architecture B: 5-node shared-storage SAN

Compute:

• 5 compute nodes (4 used + 1 spare for HA failover).
• 16 cores, 128 GB RAM each.
• Shared SAN (e.g. NetApp or similar) with iSCSI.

Storage: iSCSI LUNs from a dedicated SAN. Multipath for redundancy.

Trade-offs vs Architecture A:

• Simpler storage model.
• Clear separation between compute and storage teams.
• Higher hardware cost (SAN + storage network).
• Single SAN failure defeats HA (mitigated by dual controllers).

Architecture C: 5-node with mixed local + Ceph

Compute: 5 nodes. Local ZFS for OS and Tier 1-2 VMs; Ceph for Tier 0 VMs that need shared storage.

Trade-offs: Best of both worlds but operationally complex. Two storage stacks to manage.

Evaluation criteria

Whichever architecture you choose, evaluate it against:

• RPO compliance: Can it meet the 15-min RPO for critical systems?
• RTO compliance: Can it meet the 1-hour RTO?
• Capacity headroom: Does N+1 fit?
• Security: MFA + RBAC + firewall?
• Cost: Within the $250k budget?
• Operational complexity: Can the team actually run it?

What you submit

For the capstone, produce:

1. An architecture diagram.
2. A bill of materials with prices.
3. The 11 sections above, each with rationale.
4. A trade-off discussion: what you gave up and why.
5. A risk register: what could go wrong and how you’d respond.

Model solution discussion

The model solution presents three architectures (A, B, C above). They are all defensible. The decision among them is driven by:

• Team competence (Ceph operations are non-trivial).
• Budget (SAN adds cost but reduces operational complexity for the storage team).
• Compliance (PCI/SOC2 may require separation of duties; favour Architecture B).

There is no single right answer. The grading criteria are:

• Justified choices.
• Explicit trade-offs.
• Operational realism.
• Recognition of failure modes.

Production considerations

Common mistakes

• Choosing a technology because it sounds impressive, not because it fits.
• Ignoring the operational complexity.
• Not budgeting for growth.

Key takeaways

• The architecture is a trade-off, not a winner.
• Justify every choice.
• Validate with stakeholders.

Knowledge check