Cost vs availability vs complexity · Proxmox VE

Why this matters in production

There are no universal “best” infrastructure decisions. There are choices with explicit trade-offs. This lesson teaches the framework to reason about them so that you can defend your choices and recognise when someone else’s “best practice” is wrong for your context.

The three axes

Every infrastructure decision lives on three axes:

flowchart LR
  C[Cost] --- A[Availability]
  A --- X[Complexity]
  X --- C

Cost — hardware, software, licenses, power, cooling, support contracts, operator time.
Availability — how often the platform is up, what failure modes it tolerates, RPO/RTO for the workloads.
Operational complexity — how many people, what skills, what on-call burden, what documentation and tooling is needed to keep the platform running.

Improving one usually worsens another. The job is to find the point on the triangle that matches the business need.

Examples

“Just add another node”

Adding a 4th node to a 3-node cluster improves availability (one more failure tolerated during rolling maintenance) and increases cost. It does not necessarily increase complexity if the team already knows how to operate a 3-node cluster. The trade-off is clean.

“Move from local storage to Ceph”

A move to Ceph adds complexity significantly (Ceph has its own operational model, networking requirements, hardware requirements, monitoring, recovery procedures). It improves availability for shared-storage workloads (no single host is a SPOF for a disk). It increases cost (extra SSDs, faster NICs, RAM for OSDs). The trade-off pays off when the workload needs what Ceph provides (live migration with replicated disks, distributed snapshots). It does not pay off for a 10-VM workload.

“Add a DR site”

A DR site adds cost (second set of hardware, second PBS, possibly second colocation) and complexity (two clusters to patch, replication jobs to manage, DR testing to perform). It improves availability (regional failure tolerated). The trade-off pays off when the business risk of a site loss justifies the cost.

The complexity tax

Every feature has an operational cost:

Feature	Cost	Complexity cost
HA	Spare capacity, fencing hardware	Restart logic, anti-affinity rules
Replication	2× storage IO during replication	Replication schedule monitoring, lag alerts
Ceph	Disks, RAM, network	MON/MGR/OSD operations, recovery, PG tuning
SDN	Control plane, BGP peers	Routing troubleshooting, FRR knowledge
PBS off-site sync	Network bandwidth, second PBS	Sync schedule monitoring, retention alignment
Multi-site	Two of everything	Two clusters to patch, two runbooks

Complexity does not scale linearly. Two clusters require roughly twice the operational attention; a five-region deployment may require ten times the operational attention of a single region.

Quantifying one decision properly

“Quantify” is easy to say. Here is what it looks like for a real decision — Ceph versus local ZFS with scheduled replication — for a 30-guest estate on three nodes.

	Local ZFS + `pvesr` replication	Hyper-converged Ceph
Extra hardware	None beyond the nodes	2 × 25 GbE per node, more OSDs, +64 GiB RAM per node
Usable capacity from 24 TB raw	~12 TB (mirrored)	~8 TB (`size=3`)
RPO on node loss	The replication interval — 15 min at best	Zero
RTO on node loss	Minutes, plus manual promotion if replication lagged	Seconds, HA restarts the guest
Live migration	Requires a storage migration, minutes per guest	Immediate
New concepts to operate	ZFS snapshots, replication job monitoring	MON/MGR/OSD, CRUSH, PGs, mClock, recovery
Failure modes to have runbooks for	Replication lag, a job that silently stopped	OSD loss, a full pool, split-brain, recovery storms
On-call skill required	ZFS, which the team already has	Ceph, which is a genuine specialism

The decision falls out of one question that has nothing to do with the technology: is a 15-minute RPO acceptable for these workloads?

If yes, replication wins on every axis — cheaper, more usable capacity, and it introduces no new operational discipline. If no, Ceph is the answer and its complexity is not gratuitous, it is the price of the RPO.

What makes this a real trade rather than a preference is that both answers are correct for some estates, and the failure mode is choosing before asking the question.

The complexity tax is paid later, by someone else, at 3 a.m.

This is the structural reason complexity gets chosen too often, and it is not a knowledge problem.

Cost is paid up front, visibly, by a budget holder who has to approve it. Complexity is paid continuously, invisibly, by whoever is on call — frequently not the person who chose it, and frequently not for six months. The two costs are decided by the same person and felt by different ones, and only one of them shows up in the business case.

The specific pattern: a Ceph cluster is deployed for a 30-guest estate because it is the best-practice answer. It works. Eighteen months later the person who built it has moved on, the estate has never had an OSD failure so nobody has practised recovery, and the first real incident is handled by someone reading the Ceph documentation for the first time, during the outage.

Why the obvious defence fails. “We will document it” is sincere and insufficient — the documentation describes the happy path, because that is the path that existed when it was written. “We will train the team” competes with everything else the team is doing. Neither survives the person who cared about it leaving.

What actually works is smaller than either. Ask, for each piece of complexity: who is the second person who can operate this, and when did they last do it? If the answer is “nobody” or “never”, the complexity is not yet owned by the organisation — it is owned by one person, and it is a single point of failure with a notice period.

That question also gives you a defensible reason to say no to complexity that would otherwise be waved through, and a defensible reason to accept it when the answer is good.

The fourth axis: reversibility

Cost, availability and complexity are the classic three, and they leave out the property that most determines how much a decision deserves to be argued about.

How expensive is this to undo?

Decision	Reversible?
VM memory, `cpuunits`, disk cache mode	Immediately, live
Storage backend for one guest	Hours; a storage migration
Adding or removing a node	A day; drain, remove, rebuild
Ceph versus local storage	Weeks; every guest migrates
Storage controller in RAID mode rather than HBA	Rebuild the node. The data does not survive
UEFI versus legacy boot	Reinstall
The VMID scheme, the network addressing plan	Effectively never; everything references them

The practical rule that falls out: spend your analysis budget in proportion to irreversibility. An hour arguing about cpuunits values is an hour wasted, because you can change it in a second and measure. An hour arguing about controller mode before the nodes are racked is the cheapest hour in the project.

This also reframes the “best practice” conversation usefully. For a reversible decision, adopting a default and moving on is correct — the cost of being wrong is one command later. For an irreversible one, “it is best practice” is not a reason, because you are going to live with the consequence either way and you need to know why it is best practice for your case.

Write the decision down, with its expiry

A short decision record beats a long design document, because it gets read. Four lines:

The decision. “Local ZFS with 15-minute replication, not Ceph.”
The constraint that drove it. “RPO of 15 minutes is acceptable for every workload in the current inventory; confirmed with the application owners on 2026-03-04.”
What would change it. “A workload arriving with an RPO requirement under 15 minutes, or the estate passing roughly 60 guests, at which point the replication job count becomes its own operational burden.”
When to revisit. “Annually, or when either trigger fires.”

The third line is the one that matters and the one usually missing. A decision without its reversal condition becomes permanent by default, and gets defended by people who do not know why it was made.

Pushing back on best practice

“Best practice” arguments often ignore context. Counter-questions:

What failure mode does this protect against, and what is the cost of that failure?
What is the operational cost of implementing this protection?
Is there a simpler way to address the same failure mode?

Example: “You should always use ECC memory.”

Counter: “The workload is a dev/test cluster on hardware with no ECC option. The cost of bit-rot is acceptable; the cost of buying server-grade hardware is not.” That is a defensible position for that workload. The “always” in the original argument is what is wrong.

What happens when the person who chose the architecture leaves?

The architecture stops being a set of decisions and becomes a set of facts, and that transition is where most of the damage happens.

Decisions become constraints. While the original architect is present, “we use local ZFS with replication” is a choice with a reason, revisitable when the reason changes. After they leave, it is simply how things are. The next person inherits a configuration with no record of what it was trading off, so they cannot tell a deliberate choice from an accident — and both look identical in pvesm status.

The consequence is usually paralysis rather than error. A team that does not know why something was done is reluctant to change it, which is rational. So the estate accumulates decisions nobody will touch, and the ones that were genuinely temporary — a workaround, a throttle, a pinned kernel — become permanent because removing them feels riskier than leaving them.

The failure that actually costs money is different again: the first incident in an area of complexity nobody has practised. Ceph recovery, a corosync partition, a ZFS pool that will not import. The runbook, if it exists, covers the happy path. Documentation written by the person who understood it assumes the understanding it was meant to transfer.

What survives a departure, in rough order of value:

Decision records with reversal conditions. Four lines per decision, as above. They are the only artefact that carries the reasoning rather than the outcome, and reasoning is what the next person needs.

Break-fix rehearsal. Not documentation — practice. Pull an OSD in a maintenance window and let someone who did not build the cluster recover it. An hour spent doing that is worth more than any amount of writing, and it converts “we have a runbook” into “two people have done this”.

A named second operator per subsystem. If nobody can be named, the subsystem is owned by an individual rather than by the organisation, and that is a finding worth raising while the individual is still there.

Configuration in version control. Not because it prevents this, but because it makes the history readable. A commit message is a decision record that nobody had to remember to write separately.

The design implication. This is a genuine argument for simplicity that is independent of cost — every piece of complexity you take on is a bet that the organisation, not any individual, will still be able to operate it in three years. Sometimes that bet is clearly worth making. The point is to make it knowingly rather than to discover it at handover.

Common mistakes

Adding complexity before the complexity is needed. A 5-node hyperconverged Ceph cluster for 10 VMs is over-engineered.
Avoiding complexity the business actually needs. Three servers in one rack with no backups is under-engineered.
Letting cost drive alone — a $5k cluster that loses a day of data every quarter is more expensive than the $50k cluster that never does.
Comparing options without quantifying them. Without numbers the discussion is decided by who argues most confidently.
Ignoring who pays the complexity. Cost is paid up front by a budget holder; complexity is paid continuously by whoever is on call, often a different person.
Spending equal analysis on reversible and irreversible decisions. An hour on cpuunits is wasted; an hour on controller mode before racking is the cheapest in the project.
Treating “best practice” as a reason for an irreversible decision. You need to know why it is best practice for your case, because you will live with it either way.
Recording the decision without its reversal condition. It then becomes permanent by default and gets defended by people who do not know why it was made.
Assuming documentation transfers understanding. It documents the happy path. Rehearsal transfers understanding.

Key takeaways

Every decision lives on cost, availability and complexity axes — plus a fourth, reversibility, which should decide how much analysis it gets.
Quantify trade-offs in concrete numbers; without them the decision goes to whoever is most persuasive.
Complexity is paid later and by someone else. Ask who the second person is who can operate each subsystem, and when they last did.
Spend analysis in proportion to how expensive a decision is to undo. Controller mode and boot mode deserve hours; disk cache mode deserves a measurement.
Write a four-line decision record: the decision, the constraint that drove it, what would change it, and when to revisit. The third line is the one usually missing.
Push back on “always” and “never.” The right answer depends on context — but for an irreversible decision, so does the reason.