Cold, warm and hot: what each actually costs

Disaster recovery and high availability answer different questions, and once that separation is settled the next question becomes a physical one: where does the recovery capability live, and how much of it is already running before anything goes wrong. The industry answers with three temperatures. Cold, warm and hot are useful words and dangerous ones, because each tends to arrive already attached to a recovery time — days, hours, seconds — measured in somebody else’s estate, if it was measured anywhere. This lesson detaches the number from the temperature and puts it back on a list of stages that your estate still has to perform.

A model is a claim about what already exists

Strip the vocabulary away and a recovery site is a partially built copy of a production estate. Building that copy takes a fixed sequence of work: obtain capacity, provision hosts and addresses, install the platform, apply configuration, place identity and trust material, load the data, start processes in dependency order, verify the result, and admit traffic. Every recovery model is a decision about how far up that sequence you have paid before the incident begins, and therefore how much of it remains to be done while the business is down.

Cold means capacity and nothing above it. There may be a contract, a rack, a cloud account, or a neighbouring cluster with room to spare; there is no provisioned host, no applied configuration and no data in place. Everything above capacity happens after the declaration, with the clock running.

Warm means the stack is built and the data is arriving, but the service is not serving. Hosts exist, the platform is installed, configuration has been applied, and data is copied or replicated on some cadence. What remains is promoting that data to a usable state, closing whatever gap the cadence left, starting or scaling up the processes, verifying and cutting traffic over.

Hot means the service is running and current, and the remaining act is to admit traffic to it — together with the decision and the verification that precede admitting traffic, neither of which is free or fast.

None of those three definitions contains a unit of time. They are statements about state, and a duration follows from them only once you know what the remaining stages cost in your estate. The boundaries between them are also conventions rather than standards. NIST SP 800-34 Rev. 1 describes alternate site types along this same axis and adds mirrored and mobile sites to the list. The AWS disaster recovery guidance divides the same continuum into four strategies rather than three, separating a pilot light that keeps a core running at minimal size from a warm standby that keeps a scaled-down but functional copy. Both are describing one underlying variable — how much has already been materialised — and both cut it in different places. That is the first reason a temperature cannot carry a number across an organisational boundary: the word does not denote the same amount of pre-built state in two documents, so two teams comparing warm sites are comparing different things.

Deriving the number stage by stage

Take a concrete estate and state its assumptions aloud, because a recovery time that does not name its assumptions is not a derivation, it is a wish with a decimal point.

An order-entry service runs on six virtual machines. Its PostgreSQL database holds 400 GiB and its file storage another 500 GiB. The service authenticates against a directory, resolves names through internal DNS, obtains TLS certificates from an internal issuer, pulls container images from a registry, and reads credentials from a secret store. Assume the recovery site’s network path and target storage together sustain 220 MiB/s during a restore — a figure that must be measured on that path rather than taken from a datasheet, because it is the number in this exercise that varies by an order of magnitude between estates and it multiplies everything downstream of it.

Sited cold, the stages run like this. Declaration and mobilisation costs whatever your escalation actually takes at three in the morning, which is a property of the rota and not of the architecture; call it 20 minutes for a group that has been drilled. Obtaining capacity costs minutes if there is a standing account with quota already raised, and days if a purchase order stands between you and a server — this single stage is where the word cold ranges over three orders of magnitude, and it is the reason two cold sites are not comparable. Provisioning from infrastructure code costs perhaps 35 minutes for six machines and their network, on the assumption that the code has been applied recently against a comparable target and does not fail on a provider version it has never seen. Identity, DNS and trust costs 40 minutes to restore the directory, move zone authority and obtain an issuer able to sign, and it sits squarely on the critical path because nothing above it starts without it. Data is 900 GiB at 220 MiB/s, which is about 70 minutes of pure transfer, before decompression, before replaying the write-ahead log to the chosen point, and before any index or integrity validation. Starting and verifying in dependency order costs 25 minutes. Admitting traffic costs the DNS time-to-live or the load-balancer change, plus the human decision to make it — say 10 minutes.

Add them up with a capacity stage of 10 minutes and the derived figure is about three and a half hours. What matters is not the total but its structure: seven addends, each with a named assumption that can be attacked individually. Halving the restore throughput adds 70 minutes. Infrastructure code unapplied for four months turns the 35-minute provisioning stage into an unbounded debugging session. Neither risk is visible in “we have a cold site”.

Now site the same estate warm. Provisioning disappears because the hosts exist. Most of the data stage disappears because only the gap since the last replication interval remains. The identity and trust stage disappears only if the directory and the certificate issuer were themselves replicated — and that is the trap. If the internal issuer runs solely at the primary, the 40-minute stage stays exactly where it was and the warm site recovers no faster along that path than the cold one did. The label changed; the critical path did not.

The costs that never reach the infrastructure invoice

The infrastructure cost of each model is the easy part, and it is the part budgets are built on. The costs that decide whether a recovery works are elsewhere, and each model accrues a different one.

A warm site accrues configuration drift. It is a second estate that must be kept converged with the first, and it diverges in proportion to how fast production changes and how rarely the site is reconciled from the source production is built from. The cruelty of this cost is that it is silent: a drifted warm site reports healthy, its hosts are up, its replication is current, because none of those signals measures sameness. Only a reconciliation run that reports the differences, or a rehearsal that fails on one, exposes it. A plan-and-compare pass against the declarative source is the cheapest exposure available, and its output is the honest inventory of what the site has become.

A hot site doubles the change surface. Every configuration change, every deployment, every schema migration and every certificate rotation now has two places it can be wrong, plus a synchronisation path between them that can be wrong on its own. The propagation that makes the site hot is also a propagation of faults: a bad migration or an accidental mass delete arrives at the standby with the same promptness as legitimate traffic. Running hot therefore does not reduce the need for point-in-time recovery; it raises it, because the standby cannot be the thing you go back to.

A cold site accrues unexercised risk, and its cost is the easiest to under-report because none of it lands in a monthly bill. Every stage in that cold ledger is an assumption nobody has tested: that the capacity is obtainable, that the infrastructure code still applies, that the restore path delivers the throughput it delivered last year, that the runbook’s third step survived the identity migration. The cost is not zero. It is deferred into the incident and paid there, at the worst exchange rate available, by people who are tired and under observation.

All three models also carry the cost of the exercise itself, which competes with feature work and is the first thing cut when the quarter gets busy — the usual route to three sites at three temperatures and no idea what any of them would do.

Mixing models is the normal answer

Nothing requires one model for a whole estate, and choosing one for everything is usually a symptom rather than a strategy. Hot for everything is the most expensive architecture available and, on a system nobody would miss for a day, buys nothing but change surface. Cold for everything is cheap until the business discovers that payments cannot be down for eleven hours. The normal answer is a mixture, assigned from service tiering rather than enthusiasm.

The mixture has one hard rule. A service recovers no faster than the slowest thing it cannot start without, so the model assigned to a service is only as good as the models assigned to its dependency closure — identity, DNS, certificate issuance, secret storage, image supply, and the monitoring you intend to judge the restore with. A hot payments service standing on a cold directory recovers cold. Inversions of this kind are the most common finding in an honest recovery review, because tiers get assigned to the systems people care about while the plumbing underneath keeps whatever nobody argued about.

The mixture also has a practical limit. Every distinct model is another runbook to maintain, another rehearsal to schedule and another set of assumptions to re-measure. Two or three models across an estate is normal and manageable; six bespoke arrangements are a burden that quietly reduces rehearsal frequency, and rehearsal frequency is the variable that decides whether any of this works.

The label must never supply the number

Everything above converges on one discipline. A recovery model tells you which stages have been pre-paid. It does not tell you what the remaining stages cost, because that depends on your data volumes, your restore path, your dependency graph, your automation and your people. The number therefore has to be derived each time from the estate’s own graph: sum the outstanding stages along the longest path from declaration to working service, include the human stages, and carry the assumptions with the sum.

Two consequences follow. Any recovery time that cannot be traced to a named stage list, an assumption per stage and a date per assumption is a category name wearing a number, and belongs in the unverified column until the derivation exists. And the derivation expires — a data-volume increase, a new dependency, a change in where trust material is issued, or the slow decay of automation nobody has run will each invalidate it silently. A figure derived eighteen months ago describes an estate that no longer exists.

What converts a derivation into a capability is an exercise that walks the stages and measures them. Until then you hold an argument — a good one, if the assumptions are honest — about what would happen.

Production discipline

1. Publish the ledger, not the temperature. For every tier-bearing service, record the stages between declaration and working service, which are pre-paid under the chosen model and which remain. A diagram labelled “warm DR” with no stage list behind it communicates a feeling, not a capability.
2. Derive every recovery time from the dependency graph. Sum the outstanding stages along the longest path, include the human stages, name the assumption behind each duration, and date it. Refuse to publish a figure that cannot be reconstructed from that arithmetic.
3. Price the invisible cost of the model you chose. Assign an owner and a cadence to warm-site drift, to the doubled change surface of a hot pair, and to the unexercised assumptions of a cold site. An unowned invisible cost is the one that shows up during the incident.
4. Assign models to dependency closures, not to services. Check that identity, DNS, certificate issuance, secret storage and image supply are sited no colder than the services standing on them, and treat every inversion as either a dependency to re-fund or a commitment to withdraw.
5. Re-derive after every architecture change and record the date. Moving an issuer, growing a dataset, adding a dependency or changing a replication cadence all move the sum while leaving the label untouched. The date on the derivation is what tells a reader whether to believe it.

Cross-course references

• Proxmox VE for Production Operators — Part XIV (Disaster Recovery) covers the hypervisor-level mechanisms that decide which stages a virtualised recovery site can pre-pay, including replicated guest storage and the restore-to-alternate-cluster path; read it as the concrete answer to what “the hosts already exist” costs to maintain for the warm model described here.
• Terraform for Production Sysadmins — Part XVII (Drift Detection and Reconciliation) is the measurement discipline this lesson says a warm site cannot survive without, because a standby diverges from production silently and a plan-and-compare pass against the declarative source is the cheapest way to convert that divergence into a list you can act on.
• Terraform for Production Sysadmins — Part XXVIII (Disaster Recovery and Resilience) covers rebuilding an environment from code, which is exactly the provisioning stage a cold site pays during the incident; its material on what code can and cannot reconstruct is what tells you whether the 35-minute assumption in this lesson’s derivation is defensible in your estate.

Quiz