Ceph on cloud infrastructure: when it is a real choice

Advanced⏱ ~18 mincephcephadm

What you'll learn

Distinguish provider-durable volumes from ephemeral instance storage
Model cross-availability-zone transfer as a replication cost
Map availability zones onto CRUSH failure domains
Treat instance replacement as routine OSD replacement

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Running Ceph on a provider that already replicates your storage means paying for durability twice, and the reasons to do it anyway are real but specific.

What the provider already gives you

Backing store	Provider durability	Ceph adds
Managed block volume	replicated inside the zone, priced accordingly	zone-spanning placement and control
Local instance NVMe	none; lost on stop or retirement	all of the durability
Managed object service	full, with its own SLA	nothing, unless the S3 semantics differ
Managed file service	full	nothing, unless POSIX behaviour differs

On managed volumes the honest justification is control and portability.
On instance storage it is durability, because there is none otherwise.

The cost shape

Line item	On-prem	Cloud
Capacity	capital, depreciated	per GB per month, times the replication factor
Replication traffic	free on your own fabric	charged per GB across zones, continuously
Recovery traffic	free	charged, and a host loss generates a lot of it
Power, space, cooling	yours	included
Hardware refresh	every five years	never
Exit cost	resale value	egress charges on everything

A size 3 pool with one copy per zone sends two zone-crossing copies for
every byte a client writes, forever. That is a recurring line item, not a
migration cost.

# what the cluster is actually moving
ceph osd pool stats
ceph -s | grep -E 'client|recovery'

Failure domains you actually get

ceph osd crush add-bucket az-a datacenter
ceph osd crush add-bucket az-b datacenter
ceph osd crush add-bucket az-c datacenter
ceph osd crush move az-a root=default
ceph osd crush move ceph-01 datacenter=az-a
ceph osd crush rule create-replicated by-az default datacenter
ceph osd crush tree

Domain	Available in a cloud
Availability zone	yes, and it is the only one that is guaranteed
Rack	not exposed; do not model it
Host	yes, but two instances may share a physical host
Power feed	not exposed

Instance lifecycle is OSD lifecycle

ceph orch device ls
ceph orch osd rm 14 --replace --zap
ceph orch osd rm status

ceph orch apply osd --all-available-devices
ceph orch ps --daemon-type osd --refresh

On-prem, replacing an OSD is an incident. On instance storage it is a
routine consequence of instance replacement, and it has to be automated
or it will not keep up.

Why replicating on top of managed volumes pays for durability twice

A managed block volume is already stored redundantly by the provider, which is why its per-GB price is several times the price of the raw media underneath it. Putting a size 3 Ceph pool on those volumes provisions three of them per logical byte, each internally redundant, so the bill carries the provider redundancy three times over while the durability gain is modest — the underlying volumes already survive device failure, and what Ceph adds is placement across zones and control of the data path. On local instance storage the arithmetic inverts completely: the provider offers no durability at all, so the replication factor is buying the only durability that exists and every copy is earning its cost. Which backing store you are on decides whether the replication factor is a purchase or a duplicate charge.

Quiz

Knowledge check · 4 questions

Q1. What makes local instance NVMe a better backing store for Ceph than managed block volumes, from a cost standpoint?

It delivers lower latency and higher IOPS, and the per-GB price of a managed volume is set by that performance rather than by the redundancy behind it It survives an instance stop, so an OSD returns with its data intact and the cluster is spared a backfill that a managed volume would have avoided anyway It supports larger OSDs than a managed volume permits, so the same usable capacity needs fewer OSDs and consumes less of the placement group budget The provider supplies no durability for it, so the replication factor buys the only durability there is rather than duplicating one already in the price
Q2. Cross-zone replication traffic is a migration cost that ends once the cluster is populated.

True False
Q3. Design failure domains for a cloud Ceph cluster.

Nine instances across three availability zones, each with local NVMe. The deployment currently uses the default host failure domain and a size 3 pool.
Q4. Why does instance replacement have to be automated on a cloud Ceph cluster?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Decide first whether the backing store is provider-durable or ephemeral; that single fact determines whether your replication factor is buying durability or paying for it a second time. Model availability zones as the CRUSH failure domain, since it is the only boundary a provider actually guarantees, and budget the cross-zone transfer that follows.

Cross-course references

Kubernetes: node autoscaling that terminates instances is a storage event when storage is local
Linux: redundancy layered on redundancy multiplies cost and adds little durability