Ceph · Self-assessment
Knowledge checks
Every knowledge check in this course, in curriculum order. Each link opens the page at its quiz. The questions are auto-graded in the browser and nothing is recorded — a wrong answer costs you only the explanation, which is the part worth reading.
- Knowledge checks
- 744
- Parts covered
- 124
- Of all lessons
- 100%
Part I
Storage Fundamentals
6 checks
- Storage primitives — why the primitive is the contractBlock, file, and object are the three storage primitives. Choosing the primitive is the first and least reversible decision in a storage design, and Ceph is the rare system that serves all three from one cluster.→
- Block storage - the primitive every database depends onWhat block storage is, how it differs from file and object storage, where Ceph RBD fits in the primitive taxonomy, and why block storage is the right answer for databases, VM disks, and random-IO workloads.→
- File storage - the shared namespace primitiveWhat file storage is as a primitive, where it differs from block and object storage, why databases prefer block and where POSIX shared filesystems are the right answer, and how CephFS implements the file primitive on RADOS.→
- Object storage — the flat namespace primitiveObject storage exposes opaque blobs in a flat namespace over HTTP. What the S3 contract actually guarantees, why there are no directories, why objects are immutable, and where RGW fits.→
- DAS, SAN, NAS — topology is not the same as primitiveDAS, SAN, and NAS describe how a consumer reaches storage; block, file, and object describe what the storage does. Conflating the two axes is why people expect Ceph to behave like a SAN.→
- Durability and availability — two different numbersDurability is the probability that data survives; availability is the probability that it can be reached right now. Ceph settings move them independently, and confusing them is how clusters get configured into data loss.→
Part II
Storage Performance Fundamentals
6 checks
- IOPS, throughput, latency — and why two of them fightThree numbers describe every storage workload. IOPS and throughput are derived from the same hardware budget, latency is what the application actually feels, and optimising one routinely damages another.→
- Queue depth and block size — the two knobs that set the costQueue depth is how many requests are in flight; block size is how much data each one carries. Together they determine whether a workload is latency-bound or bandwidth-bound, and they explain most benchmark disagreements.→
- Random and sequential, read and write — the four profilesEvery storage workload is a mixture of four access profiles. Each behaves differently on each device class, and Ceph adds its own asymmetry because writes must reach every replica.→
- Tail latency — why p99 is the number that mattersAverages hide the operations that hurt. p99, p99.9, and p99.99 describe what applications and users actually experience, and in a distributed system the tail is amplified rather than averaged away.→
- High throughput is not low latencyRecovery, backfill, and scrub are throughput workloads that share hardware with latency-sensitive clients. Making them faster makes clients slower, and the throttles exist so that the operator chooses the balance deliberately.→
- Fast disks are not fast distributed storageLocal NVMe latency is a floor, not a budget. A distributed write adds network round trips, replication to the slowest peer, and software path — and the result is several times the device number, by design.→
Part III
Storage Hardware
6 checks
- HDD — where spinning disks still belong in a Ceph clusterHow a hard drive actually performs, why the seek is the number that matters, and which Ceph pools an HDD tier serves well. HDD is not obsolete; it is specialised.→
- SATA SSD — the general-purpose Ceph OSDSATA SSDs remove the seek penalty and cap out on interface bandwidth and queue depth. They are the pragmatic default for mixed Ceph workloads, with two failure modes worth knowing.→
- Enterprise SSD — power-loss protection and why it is not optionalWhat separates a datacentre SSD from a consumer one: capacitors that make an acknowledged write durable, steady-state write performance, and endurance ratings you can plan against.→
- NVMe — parallelism, not just speedNVMe removes the SATA interface and queue ceilings and replaces a single command queue with many. What that changes for a Ceph OSD, and why one NVMe device often warrants several OSDs.→
- Endurance and DWPD — sizing flash for years of writesDWPD and TBW describe how much a drive can be written before it wears out. Ceph multiplies client writes by replication and amplification, so the budget must be computed rather than assumed.→
- Device health — SMART, NVMe logs, and what actually predicts failureWhich SMART and NVMe attributes correlate with imminent failure, which are noise, and how Ceph device health monitoring turns them into an operational signal rather than a spreadsheet.→
Part IV
Failure Domains
6 checks
- What a failure domain is, and why the answer is never "the disk"A failure domain is the set of things that die together. Getting the boundary wrong is how a 3-replica cluster loses data to a single event.→
- Physical topology — mapping racks, rows, and chassis into CRUSHHow to translate a real datacentre layout into a CRUSH hierarchy that Ceph can act on, and the topology mistakes that only surface during an outage.→
- Power domains — the failure boundary CRUSH cannot seePDUs, UPS units, and utility feeds define correlated failure boundaries that no Ceph command reveals. How to find them and encode them in the topology.→
- Network domains — switches, uplinks, and partition behaviourNetwork topology creates failure domains as real as power ones, and network failures behave differently: hosts stay alive but stop agreeing. What that does to a Ceph cluster.→
- Correlated failure — why independent probability estimates misleadDurability maths assumes failures are independent. Real clusters have shared firmware, shared manufacturing batches, and shared operators, all of which break that assumption.→
- Placement decisions — turning failure domains into pool configurationThe final step: choosing size, min_size, and CRUSH rules that deliver the tolerance the design promised, and understanding what each choice costs.→
Part V
Distributed Systems Foundations
6 checks
- Distributed state — where the truth livesEvery distributed system must answer one question: which node holds the authoritative answer? Ceph answers it with the cluster map, and everything else follows from that choice.→
- Node failure — "did not answer in time" is the only signal you getA distributed system cannot distinguish a dead node from a slow one. Everything about failure detection follows from that limitation, including why Ceph waits before acting.→
- Network partition — when both halves are alive and neither is sureA partition splits a cluster into groups that cannot see each other but are all running. Ceph resolves it by majority, and the consequences of that choice are worth understanding before you meet one.→
- CAP in operational terms — what Ceph gives up and whenCAP is often quoted and rarely applied. What it actually constrains, which side Ceph chooses, and where that choice becomes visible in day-to-day operation.→
- Quorum — why majority is the only safe ruleQuorum prevents two halves of a split system from both believing they are in charge. Why a strict majority is required, why odd counts matter, and what quorum costs.→
- Replication — the cost of surviving a node failureReplication answers node failure by keeping whole copies. What that costs in space, write bandwidth, and latency, and how those costs shape cluster design.→
Part VI
Ceph Architecture
6 checks
- Ceph architecture overview — the path a write takesFollow one write from an application through librbd, librados, CRUSH, the primary OSD, and BlueStore to disk. Every later troubleshooting lesson refers back to this path.→
- The storage layer cake — RBD, CephFS, and RGW on one RADOSThree service layers present three different storage models, all on the same object store. What each layer adds, what it cannot change, and why the distinction matters when things go wrong.→
- The daemons — mon, mgr, osd, mds, rgw and what each one ownsFive daemon types, five distinct responsibilities. Which are essential, which are per-service, how many of each to run, and what happens when each one dies.→
- Cluster map and cluster identity — FSID, monmap, osdmapThe cluster map is the consensus state every daemon reads. What each component contains, how often it changes, and why the FSID is the one identifier that must never be duplicated.→
- The client protocol and msgr2 — how clients talk to the clustermsgr2 is the wire protocol between clients and daemons. What it provides, how ports and addresses are presented, and the connectivity problems that present as Ceph problems.→
- Deployment models — cephadm and what it replacedcephadm is the supported way to deploy and manage Ceph. What it does, how it differs from what came before, and why the older tools should not be used on new clusters.→
Part VII
RADOS
6 checks
- Objects — the unit of storage everything else is built fromRADOS stores objects: a name, data, extended attributes, and an omap. Every RBD image, every file, and every S3 object is made of them.→
- Pools — the unit of policyA pool groups objects under one durability, placement, and access policy. Almost every design decision in Ceph is expressed as a pool setting.→
- CRUSH — placement without a lookup tableCRUSH computes where data lives from the cluster map. Understanding it as a function rather than a directory explains most of Ceph behaviour during change.→
- Replication in RADOS — primary, acting set, and the commitHow RADOS actually performs a replicated write: who the primary is, what the acting set means, and where the guarantees come from.→
- Failure recovery in RADOS — what happens after an OSD goes awayThe sequence from a failed heartbeat to a clean cluster: detection, marking out, peering, recovery or backfill, and the throttles that govern the pace.→
- librados — the API every Ceph service is built onlibrados is the bottom of the stack: objects, atomic operations, watch/notify, and object classes. When direct librados use is right, and when it is not.→
Part VIII
Monitors
6 checks
- What the monitors actually doMonitors maintain the cluster map, agree on changes through consensus, and serve the map to everyone else. They are not in the data path, and that distinction matters.→
- Cluster maps in detail — what each one holds and who reads itFive maps with different contents, change rates, and consumers. Reading them directly is how many diagnoses actually get made.→
- Paxos at the depth a Ceph admin needsMonitors agree through a Paxos variant. What that guarantees, what it costs, and which observable behaviours it explains.→
- The monitor store — RocksDB, growth, and backupsThe monitor store holds the cluster identity and every map. What it contains, why it grows during incidents, and how to back it up before you need to.→
- Monitor election and the leader leaseHow monitors choose a leader, what the lease does, and how to read election behaviour as a diagnostic signal rather than as noise.→
- Why three monitors — and why not two, four, or sevenMonitor count is one of the few numbers in Ceph with a correct answer determined by arithmetic. Working through it removes a whole class of design mistakes.→
Part IX
Monitor Quorum
6 checks
- The quorum model in operationQuorum is not an abstraction during an incident: it decides whether the cluster serves I/O at all. How to read it, monitor it, and reason about available tolerance.→
- Losing one monitor — a stable state, not an emergencyA three-monitor cluster running on two is fully functional with zero remaining tolerance. What to do, in what order, and what not to do.→
- Losing two monitors — the cluster stopsWith two of three monitors gone there is no quorum, no map commits, and no I/O. What is actually happening, what is safe to do, and how to get back.→
- Asymmetric failures — when reachability is not mutualThe hardest monitor faults are the ones where A can see B but B cannot see A. Why they happen, why they confuse every summary view, and how to find them.→
- Monitor recovery order — what to try, in what sequenceA decision tree for restoring monitors, from the cheapest and safest action to the most drastic, with the criteria for moving between steps.→
- Quorum failure exercises — two scenarios worth rehearsingTwo realistic quorum incidents worked end to end: a host carrying two monitors, and a network split isolating one. Both are avoidable by design.→
Part X
Manager Daemons
6 checks
- The manager daemon — what it does and what it does notMGR provides metrics, orchestration, and the dashboard. It is not in the data path, and understanding that boundary changes how you respond when it fails.→
- Manager modules — which to enable and which to leave offModules add capability and cost. A tour of the ones that matter, what each provides, and the discipline of enabling only what is used.→
- The dashboard — useful, and an exposed web serviceWhat the built-in dashboard does well, where the CLI remains better, and the security posture a web UI on a storage cluster requires.→
- The orchestrator — how ceph orch drives cephadmThe orchestrator module is the interface between Ceph commands and daemon deployment. What it can do, how it reconciles, and how to work with it rather than against it.→
- Manager failover — what happens and what to verifyActive/standby failover takes seconds and affects nothing in the data path. What restarts, what is lost, and how to test it before you need it.→
- The balancer — evening out PG distribution with upmapCRUSH distributes PGs pseudo-randomly, which is not the same as evenly. The balancer corrects the residual imbalance, and upmap mode is how it does it precisely.→
Part XI
OSD Architecture
6 checks
- The OSD process — one daemon, many responsibilitiesWhat a ceph-osd process actually does: serving clients, replicating, peering, recovering, scrubbing, and reporting. Understanding the workload explains the resource profile.→
- BlueStore is the only backend — what that meansBlueStore writes directly to raw block devices with no filesystem in between. Why that replaced FileStore, and what changed for operators.→
- OSDs and devices — one, or several, per driveThe default is one OSD per device, and on fast NVMe several OSDs per device is a supported and often necessary configuration. How to decide, and what it costs.→
- OSD memory and threads — where the resources goosd_memory_target, shards, and thread pools. What each controls, how they interact, and how to size an OSD host that survives a peering storm.→
- Heartbeats — how OSDs detect each other and report failureThe heartbeat mechanism decides when an OSD is considered down. What is exchanged, over which network, and why heartbeats can succeed while the OSD is useless.→
- PG ownership — primaries, replicas, and why it mattersEvery PG has one primary that serves all client I/O for it. How primaries are chosen, what the role costs, and how uneven primary distribution creates hotspots.→
Part XII
BlueStore
6 checks
- BlueFS, DB, and WAL — the three pieces inside BlueStoreBlueStore holds object data on the block device and metadata in RocksDB, which runs on a minimal filesystem called BlueFS. Understanding the split explains every device-placement decision.→
- The write-ahead log — durability ordering for metadataThe WAL makes metadata changes durable before they are applied. Why that matters, what it costs, and when a separate WAL device is worth having.→
- The DB device — RocksDB, omap, and compactionRocksDB holds object metadata, allocation state, and omap. How it grows, why it compacts, and what compaction does to latency.→
- Allocation — how BlueStore lays data on the deviceBlueStore manages free space itself. How the allocator works, what fragmentation costs, and why min_alloc_size matters for small-object workloads.→
- Checksums — the promise that data comes back as it went inBlueStore checksums every write and verifies on every read. What that catches, what it costs, and how it interacts with scrubbing.→
- Collections — how BlueStore organises objects by placement groupEach PG is a collection in BlueStore. Why that grouping matters for recovery, scrub, and the operations that must enumerate objects.→
Part XIII
CRUSH Fundamentals
6 checks
- The CRUSH algorithm — computing placement from the mapCRUSH turns an object name and a cluster map into an ordered list of OSDs, deterministically and without any lookup. How the computation works and what it guarantees.→
- The CRUSH map — buckets, items, and weightsThe map is a weighted tree. What each element does, how weights propagate, and how to read a map well enough to predict its behaviour.→
- The select operation — choosing items at each levelchoose and chooseleaf are where a rule expresses its failure-domain requirement. What each does, how firstn and indep differ, and when to use which.→
- The take operation — where the descent beginstake chooses the entry point into the CRUSH tree. Usually the root, sometimes a specific subtree, and with device classes it is how tiering is expressed.→
- The emit operation and reading a complete ruleemit returns the selection. Putting take, select, and emit together, and reading real rules well enough to predict exactly what they do.→
- CRUSH versus centralised lookup — what the trade actually isA lookup table gives exact control and needs a service to hold it. CRUSH gives computation with no service and accepts pseudo-random distribution. What each buys.→
Part XIV
CRUSH Failure Domains
6 checks
- Host as the default failure domainCeph defaults to separating replicas across hosts. Why that default is right for most clusters, and the exact conditions under which it is not.→
- Rack failure domain — aligning replicas with the buildingA rack rule guarantees no two copies share a rack. What it requires, what it costs, and the arithmetic that decides whether your cluster can support it.→
- Chassis, room, and row — extending the hierarchyBeyond host and rack, Ceph offers bucket types for dense hardware and large facilities. When each earns its place, and when depth is just complexity.→
- Weights across failure domainsWeights propagate upward, so a failure domain inherits its children total. What that means for uneven domains, and why it decides usable capacity.→
- What changes when weights changeA weight change recomputes CRUSH, remaps PGs, and backfills. How much moves, how long it takes, and how to control the impact.→
- When CRUSH cannot satisfy the ruleUndersized PGs on a healthy cluster mean the rule asks for more than the topology provides. How to diagnose it quickly and what the options are.→
Part XV
CRUSH Maps and Rules
6 checks
- The replicated rule — the form you will write most oftenThree steps that cover the vast majority of production pools. What each argument means, how to create rules with the CLI, and how to verify the result.→
- The erasure-coded rule — indep, chunk positions, and profilesEC rules differ from replicated ones in ways that matter for correctness. Why profiles generate them, and what to check when you inherit one.→
- Multiple rules on one clusterPools bind to rules, so one cluster can serve very different durability and placement policies. How to design a rule set that stays comprehensible.→
- Device-class rules — one cluster, several performance tiersDevice classes let a single cluster serve NVMe, SSD, and HDD pools with no manual tree maintenance. How to use them and what to verify.→
- Editing rules — which changes are safe and which are notSome rule edits are free, some move a little data, and some move everything. Knowing which is which before you apply is the whole skill.→
- Rule and weight changes that move data — planning the whole eventBringing together every operation that relocates data: what triggers it, how much moves, how to pace it, and how to know when it is done.→
Part XVI
Device Classes
6 checks
- How an OSD gets its device classceph-volume assigns hdd, ssd, or nvme at OSD creation from what the kernel reports. When that is wrong, and how to correct it.→
- CRUSH rules that select by classClass-aware rules pin a pool to one media type. The syntax, the shadow-tree mechanism, and the requirement that catches people.→
- Mixed-storage clusters — safe when separated, dangerous when notA cluster with HDD, SSD, and NVMe is normal and effective. Without class separation it is a cluster whose fast pools run at HDD speed.→
- The lifecycle of a device class assignmentClasses are set at creation, persist across restarts, and change only deliberately. What happens at each stage of an OSD life, and where drift creeps in.→
- Wrong class placement — the most common pool mistakeA rule naming a class that too few hosts provide leaves PGs undersized. The diagnosis takes two commands and the fix is a design decision.→
- The performance impact of device class — the biggest lever you haveChoosing which class a pool sits on changes latency by orders of magnitude. How to measure the difference, and how to spend a limited flash budget.→
Part XVII
Pools
6 checks
- Pool purpose — the unit where policy is decidedEvery durability, placement, and access decision in Ceph is a pool property. Designing a cluster is largely designing its pools.→
- Replicated pools in productionThe default and the right answer for most workloads. Sizing, min_size, the capacity cost, and the operational behaviours that follow.→
- Erasure-coded pools in productionEC trades write cost and recovery complexity for capacity efficiency. When that trade is right, how to configure it, and the constraints that surprise people.→
- Application tagging — telling Ceph who owns a poolThe application tag drives service tooling and documents intent. Setting it takes one command and omitting it produces a warning people mute.→
- Pool quotas — bounding capacity per poolQuotas cap a pool bytes or object count and are enforced, though with a reporting lag. What they protect against and what they do not.→
- Pool autoscaling — letting the manager size pg_numThe pg_autoscaler computes an appropriate pg_num from pool data share and cluster size. How to adopt it safely on an existing cluster.→
Part XVIII
Placement Groups
6 checks
- Why placement groups existPGs are the indirection layer between objects and OSDs. Without them, tracking placement would scale with object count instead of with cluster size.→
- Object to PG — the hashing stepThe first half of placement is a hash of the object name masked by pg_num. Simple, deterministic, and with consequences for what happens when pg_num changes.→
- PG to OSD — the CRUSH stepThe second half of placement takes the PG id and returns an ordered OSD list. What determines the result and what changes it.→
- PG count tuning — the numbers and what they costChoosing pg_num per pool within a cluster-wide budget. The arithmetic, the constraints, and how to change it without disruption.→
- PG states — the vocabulary you read during every incidentA tour of the states a PG can report, what each means operationally, and which combinations demand action.→
- PGs per pool — managing the cluster-wide totalEach pool carries its own pg_num and they all draw on the same per-OSD budget. How to allocate, monitor, and rebalance the total.→
Part XIX
PG States
6 checks
- active+clean — what the steady state actually guaranteesThe state every PG should be in. What Ceph has verified to report it, what it has not, and why a fully clean cluster is still worth monitoring.→
- active+degraded — serving with fewer copies than promisedDegraded means the pool is honouring reads and writes while holding less redundancy than configured. What causes it, how urgent it is, and when it is not self-healing.→
- active+undersized — the acting set is shortUndersized means CRUSH could not fill the acting set. Sometimes a failure, often a placement constraint, and the two need different responses.→
- Peering — agreeing on what the PG containsBefore a PG can serve, its OSDs must agree on authoritative history. What peering does, why it blocks I/O, and what makes it slow.→
- Inactive PGs — the most urgent state in CephAn inactive PG serves nothing. The causes, the diagnosis, and the decisions — including the irreversible one that should almost never be made.→
- Inconsistent PGs — scrub found replicas that disagreeDeep scrub compares replica contents and reports disagreement. What it means, how to determine which copy is right, and when repair is safe.→
Part XX
PG Peering
6 checks
- Up set, acting set, and primaryThree related concepts that appear in every PG diagnostic. What each means, when they differ, and what the difference tells you.→
- Peering history — finding the authoritative copyPeering compares what each OSD knows about the PG past to decide whose copy is current. The intervals, the logs, and why history matters more than data.→
- Peering after an OSD restartThe cheapest peering case: an OSD returns quickly, its log covers the gap, and recovery copies only what it missed.→
- Peering after an OSD failureWhen an OSD does not come back, the cluster marks it out, CRUSH recomputes, and backfill rebuilds its share elsewhere. The sequence and its costs.→
- Peering after a topology changeCRUSH changes make every affected PG re-peer and remap. Why this differs from a failure, and how to make it predictable.→
- Stuck peering — diagnosis and resolutionPeering that does not complete blocks I/O. The causes, the diagnostic sequence, and the resolutions ordered from safest to most drastic.→
Part XXI
PG Autoscale
6 checks
- pg_autoscale_mode: on, warn, and offThe autoscaler has three modes and they mean very different things operationally. What each does, and which one belongs on which pool.→
- Reading autoscaler suggestionsThe autoscale-status table has a column for every input the module used. Reading it correctly tells you why the recommendation is what it is.→
- target_size_bytes and target_size_ratioTwo ways to tell the autoscaler what a pool will become rather than what it is. When each fits, and how they interact.→
- How the autoscaler responds to workload changeData growth and OSD count both move the recommendation, but through different paths and on different timescales.→
- Applying autoscaler recommendations safelyA recommendation you accept becomes hours of backfill. The procedure for applying one without surprising your users.→
- Bounding what the autoscaler may dobias, bulk, pg_num_min, pg_num_max, and the global budget knob — the controls that keep autoscaling inside limits you choose.→
Part XXII
PG Investigation
6 checks
- Reading a single PG with pg queryceph pg query returns everything the primary knows about a placement group. Which fields answer which question.→
- Cluster-wide PG state: pg stat, pg dump, health detailThree commands at three levels of detail. Which to reach for depends on whether you are counting, listing, or triaging.→
- When to intervene, and when to waitMost PG problems resolve themselves. Knowing which ones will not is the skill that separates useful intervention from harmful.→
- PGs stuck scrubbing or deep-scrubbingA scrub that never finishes is almost always one slow device. How to confirm that and what to do about it.→
- PGs stuck in recoveryRecovery that has stopped is different from recovery that is slow. Separating the two, and the specific causes of each.→
- Deep investigation: per-OSD latency and logsWhen the PG-level view has told you all it can, the next layer down is the OSD itself: perf counters, admin socket, and the log.→
Part XXIII
Replication
6 checks
- size and min_size: what each one controlsTwo numbers that decide how many copies you keep and how many you need to keep serving. Confusing them is the most consequential mistake in Ceph pool configuration.→
- How a replicated write actually completesThe primary-driven write path, step by step, and what the client is waiting for at each stage.→
- Writes while a replica is missingA degraded pool keeps accepting writes, and the mechanism that lets missing replicas catch up afterwards is the PG log.→
- Reads on a degraded poolReads are served by the primary. What happens when the primary is the OSD that failed, and what read balancing changes.→
- Choosing size: what each value actually buyssize 2, 3, and 4 quantified — failure tolerance, capacity cost, and the correlated-failure reasoning that decides between them.→
- Why min_size 1 loses dataThe specific mechanisms by which min_size 1 destroys data, and the safe alternatives when availability pressure is real.→
Part XXIV
Replica Failure Scenarios
6 checks
- One OSD fails on a size-3 poolThe ordinary case, minute by minute: what the cluster does, what clients see, and what you should and should not do.→
- Two OSDs fail on a size-3 poolThe outcome depends entirely on whether the two failures overlap on the same PGs — and usually only a small fraction do.→
- Cascading and correlated failureIndependent failure is a modelling assumption, and real hardware violates it. How cascades happen and how CRUSH contains them.→
- Reasoning about data safety from the cluster mapHow to determine, from ceph osd tree and the pool's rule, exactly what your cluster can survive — before it has to.→
- The acting set is the safety boundaryEvery availability question about a PG reduces to how many current copies its acting set holds. The precise rules for reads and writes.→
- Recovery restores redundancy; it does not change the boundaryThe distinction between being safe and being fully redundant, and why the gap between them is where risk accumulates.→
Part XXV
Erasure Coding Fundamentals
6 checks
- k and m: what an erasure code profile meansData chunks, coding chunks, and the loss tolerance they buy — plus the min_size detail that surprises most operators.→
- Capacity efficiency and the real cost of a profileThe overhead arithmetic, why it is not the whole story, and how min_alloc_size can quietly erase the savings.→
- How EC chunks are placedChunk placement uses indep rather than firstn, and the reason why is one of the more elegant details in CRUSH.→
- EC recovery and what it costsReconstructing one chunk requires reading k chunks from k different OSDs. The consequences run through every part of EC operations.→
- Choosing among the common profiles4+2, 6+3, 8+3, and 10+4 compared on the four axes that decide the choice, with the constraints that rule each in or out.→
- EC for RBD and CephFS: what it takes and when it fitsEC-backed block and file storage is supported and requires specific configuration. The requirements, the caveats, and the honest performance picture.→
Part XXVI
Erasure Coding Trade-offs
6 checks
- CPU cost of erasure codingEvery EC write encodes and every degraded read decodes. What that costs, where it lands, and when it becomes the binding constraint.→
- Write amplification on EC poolsOne client write becomes k+m OSD operations. The arithmetic, and how alignment decides whether it is 1.5× or far worse.→
- Why EC handles small writes badlyThe concrete latency arithmetic for a small write on EC versus replication, and the workloads that make it unacceptable.→
- The recovery cost of erasure codingRecovery volume scales with k, and on wide profiles it becomes the constraint that decides cluster design.→
- EC durability compared with replicationA wide EC profile can be more durable than size 3, and can also be less. What determines which, and how to compare them honestly.→
- Capacity planning for erasure-coded poolsWorking a full EC capacity plan: raw to usable, fill headroom, failure reserve, and the growth model.→
Part XXVII
Replication vs Erasure Coding
6 checks
- Why RBD generally wants replicationBlock devices produce exactly the access pattern EC handles worst. The reasoning, and the RBD workloads that are exceptions.→
- Object archives are where EC paysRGW workloads have the write pattern, object size, and access profile that make erasure coding straightforwardly correct.→
- Large immutable data: the clearest EC caseBackups, log archives, media, and scientific data share a profile that makes the EC decision straightforward.→
- Databases: primary data replicated, everything around it ECDatabase platforms produce several distinct storage workloads, and only one of them requires replication.→
- A decision framework for pool typeFive questions, asked in order, that determine whether a workload belongs on replication or erasure coding.→
- Running replicated and EC pools on one clusterThe production default is both pool types side by side. What that requires operationally, and the interactions to watch.→
Part XXVIII
Ceph Networking
6 checks
- The public network: what crosses itEvery client operation and every monitor conversation rides the public network. What that means for sizing and for failure behaviour.→
- The cluster network: when to have oneA separate OSD-to-OSD network isolates replication and recovery from client traffic. What it buys, what it costs, and the failure mode it introduces.→
- Client traffic characteristics by interfaceRBD, CephFS, and RGW produce distinctly different network profiles. Knowing which you have shapes every sizing decision.→
- Replication traffic and what it multipliesEvery client write becomes several OSD-to-OSD writes. The multiplier by pool type, and its implications for network sizing.→
- Recovery traffic and the throttles that shape itRecovery is the largest network consumer a Ceph cluster produces, and it is entirely governed by settings you control.→
- MTU, bonding, and VLANs: the link layer under CephThe three link-layer decisions that most often cause Ceph problems, and how to get each one right.→
Part XXIX
Network Design
6 checks
- Choosing link speed for a Ceph clusterWhat each generation actually delivers, and how to derive the requirement from the cluster rather than from a rule of thumb.→
- Bonding for redundancy and capacityLACP and active-backup solve different problems. Which to use, and what each actually protects against.→
- Switches as failure domainsThe switch is a failure domain whether or not your CRUSH map says so. Aligning the two is what makes redundancy real.→
- LACP in detail: modes, rate, and hashingThe three LACP settings that determine whether a bond behaves the way you assume, and how to verify them from both ends.→
- Routed network designs for CephLayer 3 to the host removes large broadcast domains and scales cleanly. What it requires of Ceph and what it changes.→
- Jumbo frames: benefit, risk, and verificationWhat jumbo frames actually buy on a Ceph cluster, and the discipline required to deploy them without creating a hard-to-diagnose fault.→
Part XXX
Network Failure Behaviour
6 checks
- Packet loss and why it destabilises OSDsLoss rates far too small to notice on a ping test are enough to make a Ceph cluster flap. The mechanism, and how to detect it.→
- How network latency propagates into storage latencyNetwork RTT is paid on every operation and multiplied by the replication path. The arithmetic, and what it means for placement.→
- Asymmetric paths and one-way problemsTraffic can be fine in one direction and broken in the other. Ceph assumes symmetry, and the diagnosis requires directional tools.→
- Switch failure: what Ceph seesThe same switch failure is invisible, disruptive, or catastrophic depending entirely on the design underneath it.→
- Bond member failure versus bond failureOne is invisible and halves your capacity; the other takes the host offline. Telling them apart, and detecting the silent one.→
- MTU mismatch: the most baffling slow-ops causeThe failure that presents as storage slowness, passes every routine check, and is confirmed in one command.→
Part XXXI
Ceph Authentication (cephx)
6 checks
- The cephx authentication protocolShared-secret mutual authentication with time-limited tickets. What it protects against, and what it does not.→
- Entities: users and daemons in the cephx namespaceEvery actor in a Ceph cluster is an entity with a name, a key, and capabilities. The naming scheme and what it implies.→
- Keyrings: format, locations, and handlingThe file that holds a secret. Where Ceph looks for it, how it should be protected, and how to work with it safely.→
- Capabilities: what an entity is permitted to doCapabilities are granted per subsystem and evaluated by the daemon serving the request. The structure and the vocabulary.→
- Capability semantics: r, w, x, and scopingWhat each permission letter actually permits, and how pool, namespace, and object-prefix scoping narrow a grant.→
- Rotating cephx keys without an outageKey rotation requires an overlap period because Ceph entities hold one key at a time. The procedure that makes it safe.→
Part XXXII
Least Privilege Capabilities
6 checks
- Minimal capabilities for RBD clientsWhat an RBD client genuinely needs, why the RBD profiles are the right tool, and what a too-broad grant exposes.→
- Minimal capabilities for CephFS clientsCephFS clients need MDS, OSD, and MON capabilities, and path restriction is what makes multi-tenant CephFS viable.→
- RGW identity: two separate systemsS3 and Swift users are RGW-level identities with no cephx presence. Understanding the split prevents a class of confusion.→
- Capabilities for monitoring and metrics clientsMonitoring needs to read cluster state and nothing else. The narrowest grant that works, and what a broad one exposes.→
- client.admin and the discipline around itThe bootstrap credential has unlimited authority. How to use it sparingly, and what to use instead for routine work.→
- Running a capability auditA repeatable procedure for reviewing every credential in a cluster and reducing each to what it needs.→
Part XXXIII
Encryption
6 checks
- msgr2 and encryption on the wireThe v2 messenger protocol, what it adds over v1, and the distinction between integrity and confidentiality.→
- Encryption at rest with dm-crypt and self-encrypting drivesProtecting data on the physical device, what it actually defends against, and the key management that decides whether it works.→
- Deploying secure mode across a clusterThe procedure for turning on wire encryption without disconnecting clients, and the cost to expect.→
- crc mode: what it is for and its limitsCRC mode exists for compatibility and for hosts that cannot afford encryption. What it does provide, and what it does not.→
- RGW server-side encryption: SSE-S3, SSE-KMS, and SSE-CThree ways object data is encrypted at the gateway, differing entirely in who holds the key.→
- Protecting monitor state at restThe monitor store holds the keys to the cluster. What is in it, how to protect it, and why its backups are as sensitive as the data.→
Part XXXIV
Multi-Tenancy Concepts
6 checks
- Tenant isolation for RBD: pools and namespacesTwo mechanisms with different costs. When a pool per tenant is right, and when a namespace is better.→
- CephFS subvolumes and multi-tenant file storageSubvolumes give each tenant a managed directory with a quota, a path-restricted credential, and a lifecycle.→
- RGW tenants and bucket namespace separationThe tenant mechanism gives each customer their own bucket namespace, so two tenants can both own a bucket named "backups".→
- RGW quotas: user, bucket, and defaultsQuotas at three levels with different enforcement characteristics, and the statistics lag that makes overshoot inevitable.→
- Matching the isolation mechanism to the tenantCeph offers isolation at several levels with very different costs. A framework for choosing per tenant rather than per platform.→
- Testing tenant isolation before launchAn isolation boundary that has not been attacked is an assumption. The tests to run and the results to record.→
Part XXXV
RBD Architecture
6 checks
- The RBD I/O path from guest to OSDHow a 4 KiB write inside a VM becomes a RADOS operation, and which layer contributes what to latency.→
- Image layout: objects, striping, and sparsenessHow an image maps onto objects by default, and what the striping parameters change for large sequential workloads.→
- RBD image features and what each costsThe feature bits, what they enable, which clients support them, and why the default set is what it is.→
- RBD snapshots and copy-on-writeA snapshot is a point in time, implemented by preserving old object versions on write. What that costs and when.→
- Clones: copy-on-write images from snapshotsA clone starts as a reference to a protected snapshot and diverges as it is written. The mechanism, the dependency, and flattening.→
- Three ways to attach an RBD imageKernel RBD, librbd, and rbd-nbd differ in performance, feature support, and failure behaviour. Choosing between them.→
Part XXXVI
RBD Images
6 checks
- Creating RBD images with the right parametersThe decisions made at creation time — size, object size, features, data pool — and which of them cannot be changed later.→
- Choosing a feature set per image classDifferent image classes have different feature needs. Establishing per-class defaults rather than deciding per image.→
- Flattening clones and managing the dependency graphWhen to break the parent link, what it costs, and how to keep a clone-heavy platform manageable.→
- Resizing RBD images, up and downGrowing an image is routine and requires filesystem work above it. Shrinking is a data-loss operation with a specific safe procedure.→
- Image metadata and the objects behind itWhat rbd info reports, where that information is stored, and the metadata objects an image depends on.→
- Snapshots, clones, and images: what each one isThree related concepts that are routinely conflated. The distinctions, and the operational consequences of each.→
Part XXXVII
RBD Snapshots
6 checks
- Taking snapshots that are actually usableThe command is trivial; making the resulting snapshot restorable requires coordination with the guest.→
- Rolling back to a snapshotRollback rewrites the image in place and discards everything since. The procedure, the alternatives, and when each fits.→
- What snapshots protect against, and what they do notSnapshots share every failure domain with the image. Placing them correctly in a data-protection strategy.→
- Snapshot protection and its role in clone safetyProtection makes a snapshot immutable so clones can depend on it. Managing protection through a template lifecycle.→
- The snapshot and clone lifecycle end to endPutting the pieces together into a working template and provisioning workflow, with the operational practices that keep it maintainable.→
- The performance cost of snapshots and clone depthSnapshots cost write latency on first modification; clone chains cost read latency. Quantifying both and keeping them bounded.→
Part XXXVIII
RBD Performance
6 checks
- Queue depth: the parameter that decides RBD throughputCeph latency is fixed by physics; throughput comes from concurrency. Why queue depth matters more than any tuning setting.→
- librbd caching and what it is safe to enableClient-side cache modes, the durability implications of each, and where caching actually helps.→
- Block size and the layers that have oneFour layers each have a notion of block size, and mismatches between them cause read-modify-write amplification.→
- What replication adds to every RBD writeThe second and third writes are not free, and they land on the same devices and network as the first.→
- Why one slow OSD dominates RBD latencyEvery write waits for the slowest replica, and every OSD participates in hundreds of PGs. The mathematics of the outlier.→
- The network component of RBD latencyTwo round trips per write, paid on every operation. Measuring the network term and knowing when it dominates.→
Part XXXIX
RBD Troubleshooting
6 checks
- When an image will not openFour distinct causes with four distinct error messages. Reading the message correctly is most of the diagnosis.→
- Diagnosing RBD authentication failuresAuthentication failures present through several unhelpful error messages. Working from symptom to cause systematically.→
- When rbd map failsThe kernel client has its own failure modes, distinct from librbd. Diagnosing each from the message and dmesg.→
- Working an RBD latency complaint to its causeA structured descent through the layers, with the command that settles each one.→
- Blocked requests and full-cluster conditionsSlow ops and blocked requests name the OSD and the stage. Reading them, and handling the capacity thresholds that cause them.→
- Kernel client and librbd: differences that matter in an incidentThe two clients fail differently, log differently, and recover differently. Knowing which you are dealing with shapes the response.→
Part XL
CephFS Architecture
6 checks
- The CephFS split path: metadata through the MDS, data directClients talk to the MDS for namespace operations and to the OSDs for file contents. Why that split is the whole design.→
- The metadata pool and the data poolsCephFS requires at least two pools with entirely different characteristics. Sizing and placing each correctly.→
- File layouts: how a CephFS file maps to objectsLayout attributes control striping and pool placement per file or directory, set through extended attributes.→
- MDS ranks and how the namespace is dividedA rank is a share of the namespace. Multiple ranks scale metadata throughput; they also add complexity.→
- The MDS journal and why recovery worksMetadata operations are journalled to RADOS before being applied. That journal is what makes MDS failover safe.→
- CephFS capabilities: the client caching contractPer-inode capabilities let clients cache metadata and data safely. How they are issued, revoked, and where they cause trouble.→
Part XLI
Metadata Servers (MDS)
6 checks
- Active and standby MDS daemonsThe standby types, what each costs, and how many you actually need.→
- How ranks divide one namespaceMultiple ranks present a single filesystem while partitioning its metadata. The partitioning is dynamic, and that has consequences.→
- The MDS cache and its memory limitThe MDS holds metadata in memory, and how much determines both performance and stability.→
- MDS failover: the sequence and its timingsWhat happens between an MDS dying and the filesystem serving again, stage by stage.→
- What happens during MDS recoveryThe states an MDS passes through on takeover, what each is doing, and how to tell a slow recovery from a stuck one.→
- Why CephFS does not have MDS split brainRank assignment is arbitrated by the monitors and enforced by blocklisting. The mechanism, and what it guarantees.→
Part XLII
CephFS Operations
6 checks
- Creating a CephFS filesystem correctly the first timeThe pools, the settings, and the decisions that are awkward to change once the filesystem holds data.→
- Mounting CephFS: kernel and FUSETwo clients with different performance, feature support, and failure behaviour. Mounting each correctly and choosing between them.→
- CephFS quotas and their enforcement modelQuotas are extended attributes enforced cooperatively by clients, with a propagation delay that makes overshoot inevitable.→
- CephFS snapshots and the .snap directoryDirectory-level snapshots exposed through a hidden directory. How they work, what they cost, and where to be careful.→
- Subvolumes as the operational unitThe managed abstraction that bundles a directory, a quota, a data placement, and credential generation.→
- CephFS client behaviour and consistencyHow clients cache, when they revalidate, and the consistency guarantees that follow from the capability model.→
Part XLIII
CephFS Failure Scenarios
6 checks
- Responding to an MDS daemon crashWhat the cluster does automatically, what clients experience, and the narrow set of cases needing operator action.→
- Diagnosing a slow MDSFour distinct causes with four different fixes. Distinguishing them before changing anything.→
- When a CephFS pool fillsThe metadata pool and the data pool fail differently when full, and the metadata case is the more serious.→
- When a client sees the wrong thingMissing files, permission errors, and stale views. Working from symptom to cause without assuming a consistency bug.→
- MDS journal damage and recoveryThe rarest and most serious CephFS failure. The supported recovery path, in order, with the destructive steps clearly marked.→
- Session timeouts, eviction, and stale file handlesWhy clients get evicted, what it costs them, and how to configure the timeouts to match your environment.→
Part XLIV
Object Storage Foundations
6 checks
- What object storage actually promisesThe contract is narrower than a filesystem and that narrowness is the point. What you get, and what you give up.→
- Buckets: the unit of policy and the unit of scaleA bucket is a namespace, a policy boundary, and — importantly for Ceph — an index with scaling characteristics of its own.→
- The S3 API surface RGW implementsWhat is supported, what is not, and how to verify compatibility before committing an application.→
- Flat namespaces and the prefix conventionObject keys are flat strings; the hierarchy is a convention implemented in listing. What that means in practice.→
- S3 versioning and its capacity consequencesVersioning protects against overwrite and deletion, and every version consumes capacity until something removes it.→
- Lifecycle rules: automated expiration and transitionThe mechanism for retention policy at scale, and the operational details that determine whether it works.→
Part XLV
RADOS Gateway (RGW)
6 checks
- RGW: translating S3 into RADOSWhat the gateway does to a request, the pools it touches, and why it is stateless.→
- Placing and sizing RGW daemonsWhere gateways run, how many to run, and the resource characteristics that determine the answer.→
- Zones: the unit of data placementA zone owns a set of pools and the objects in them. The concept underlying both single-site and multisite deployments.→
- Zonegroups and the shape of multisiteA zonegroup is the boundary within which zones replicate. The topologies it enables and what each provides.→
- The RGW pools and what each requiresA deployment creates several pools with very different characteristics. Placing and sizing each correctly.→
- High availability for RGWGateways are stateless, so availability is a load-balancing problem. The components, the health checks, and the failure behaviour.→
Part XLVI
RGW Users and Credentials
6 checks
- Creating and managing RGW usersThe user record, what it holds, and the lifecycle operations that matter operationally.→
- Access keys, rotation, and credential handlingS3 credentials are a key pair on a user record. Multiple pairs per user is what makes rotation straightforward.→
- Subusers and scoped credentialsA subuser is a credential with restricted access under a parent user. Where it fits and where bucket policies fit better.→
- Applying RGW quotas in practiceSetting quotas at the right level, monitoring consumption, and handling the aggregate over-commitment RGW does not track.→
- Bucket policies: the general access control mechanismJSON policies expressing per-bucket, per-action, per-principal rules, including conditions. The evaluation model and the common patterns.→
- RGW admin capabilitiesCapabilities on an RGW user grant access to the admin REST API. Scoping them for automation and self-service.→
Part XLVII
RGW High Availability
6 checks
- Running multiple gateways wellAdding gateways is easy; getting the configuration consistent and the capacity right takes a little more.→
- Load balancing RGW: layer 4 versus layer 7Both work. The choice turns on TLS termination, health checking, and what visibility you need.→
- Setting up RGW multisiteThe configuration sequence for a second zone, and what to verify at each step.→
- Sync policies: controlling what replicatesGranular control over which buckets replicate where, replacing the all-or-nothing zonegroup default.→
- Eventual consistency across zonesReplication is asynchronous, so zones diverge under load. What applications must tolerate and how conflicts resolve.→
- RGW failure modes and their responsesA catalogue of what breaks, what each looks like, and the first action for each.→
Part XLVIII
RGW Troubleshooting
6 checks
- S3 authentication failures and what each meansFour error codes with four distinct causes. Reading the code correctly ends most of the investigation.→
- Working an AccessDenied to its sourceFour independent mechanisms can deny a request. Checking them in order rather than guessing.→
- When the pools behind RGW are the problemGateway errors that originate below the gateway, and how to trace them to the responsible pool.→
- Tracing RGW latency to its layerA request crosses the client network, the balancer, the gateway, and the pools. Attributing the time.→
- Load balancer failures in front of RGWThe balancer is a single point of failure unless it is not, and a weak health check makes it a source of failure rather than protection.→
- Diagnosing multisite sync problemsSync falls behind, stalls, or errors. Reading the status output and working each case.→
Part XLIX
cephadm
6 checks
- Bootstrapping a cluster with cephadmThe single command that creates a cluster, what it actually does, and the options worth setting at bootstrap.→
- Adding and managing hostsWhat the orchestrator needs from a host, how to add one, and how to remove one without losing data.→
- Declaring services with the orchestratorServices are declared, not deployed — the orchestrator reconciles reality to the declaration. The consequences of that model.→
- Reading cluster state through the orchestratorFour commands that answer what is declared, what is running, where, and why not.→
- Device inventory and OSD creationWhat makes a device available, how the orchestrator decides, and how drive groups automate OSD deployment.→
- Labels as the placement mechanismLabels decouple service declarations from host names, which is what makes a cluster expandable without editing specifications.→
Part L
Cluster Deployment
6 checks
- How many hosts a Ceph cluster needsThe minimum is set by the failure domain and the replication factor, not by capacity. Deriving the number from the requirements.→
- Placing monitorsMonitors are the cluster's consensus layer. Where they run and how many determines what the cluster survives.→
- Placing managersOne manager is active and the rest stand by. What runs there, and what a manager outage actually costs.→
- OSD layout: devices, hosts, and what to co-locateOne OSD per device is the default and the exception is worth understanding. Plus the DB and WAL placement decision.→
- Network design for a production clusterWhich networks a Ceph cluster needs, what each carries, and how to size them.→
- Time and name resolution as deployment prerequisitesTwo infrastructure dependencies that cause a disproportionate share of Ceph problems when they are not right.→
Part LI
Host Preparation
6 checks
- The Linux baseline for a Ceph hostDistribution, kernel, filesystem, and tuning choices that matter, and the ones that do not.→
- Host networking configuration for CephInterface naming, bonding, VLANs, and addressing, and the failure each gets wrong.→
- Configuring time synchronisation for CephThe specific configuration Ceph needs, and how to verify it is actually working rather than merely installed.→
- Hostnames and resolution on Ceph hostsHow the cluster identifies hosts, why mismatches are confusing, and what to verify.→
- Preparing disks for OSD useWhat makes a device usable, how to clear one safely, and the hardware-level settings that matter.→
- The container runtime under cephadmWhat cephadm needs from podman, how daemons are actually run, and how to diagnose runtime problems.→
Part LII
Time Synchronisation
6 checks
- What breaks in Ceph when time is wrongFive distinct mechanisms depend on synchronised clocks, and each fails differently.→
- Monitor elections and clock skewHow consensus depends on time, what skew does to it, and how to recover a cluster destabilised by it.→
- Certificates, msgr2, and timeWhat clock skew does to TLS and to msgr2 secure mode, and how to distinguish it from a genuine certificate problem.→
- Log correlation and why it needs synchronised timeReconstructing an incident across hosts depends on their clocks agreeing. What that is worth and how to preserve it.→
- Choosing and configuring a time sourcechrony, ntpd, and systemd-timesyncd compared, and how to structure time sources for a cluster.→
- A time skew incident, worked throughA complete incident from first symptom to prevention, showing how the diagnosis is missed and how it is found.→
Part LIII
Cluster Health
6 checks
- The three health states and what they actually meanHEALTH_OK, WARN, and ERR describe the presence of checks, not the severity of your situation. Reading them correctly.→
- What HEALTH_OK does not tell youThe absence of firing checks is not a statement about durability, performance, or capacity trajectory.→
- Reading HEALTH_WARN correctlyWarnings span from cosmetic to urgent. Triaging them by what they mean rather than by their label.→
- HEALTH_ERR and the checks that produce itError-severity checks mean part of the cluster cannot do its job. The short list, and the first action for each.→
- Getting the most from ceph health detailThe command that turns a state into a diagnosis, including its structured output and the fields that matter.→
- A reference for the health checks you will actually seeThe checks that appear in practice, what each means, and the first command for each.→
Part LIV
ceph status and health detail
6 checks
- Reading ceph -s line by lineThe one command everyone runs, and the fields within it that most people skim.→
- ceph health, detail, and the mute mechanismThree forms of the same command at three levels of detail, plus the controls for suppressing known conditions.→
- Inspecting monitor quorumThe command that shows who is in quorum, who is not, and what each monitor believes.→
- Monitor status at a glanceThe compact monitor summary, the monmap, and what changes to each mean.→
- Manager status and modulesWhat is active, what is standing by, which modules are running, and what to check when a module misbehaves.→
- PG statistics as a progress meterThe one-line PG summary, and how to use it to distinguish progress from a stall.→
Part LV
OSD States
6 checks
- up and down: the liveness axisWhether an OSD is answering. How the cluster decides, and what the state does and does not imply.→
- in and out: the placement axisWhether CRUSH places data on an OSD. This is the axis that causes data to move.→
- The four OSD states and what each meansup/in, up/out, down/in, and down/out are four distinct situations with four distinct responses.→
- Why liveness and placement are separate concernsThe separation is deliberate and enables the down-out interval, maintenance mode, and gradual draining.→
- The lifecycle of an OSD through its statesDeployment to retirement, and the transitions in between, with the command and the cost for each.→
- The safe order for OSD operationsThe sequence exists because skipping a step causes avoidable degradation or an incomplete removal. What each step protects.→
Part LVI
OSD Failure
6 checks
- The first minutes of an OSD failureWhat happens between an OSD failing and the cluster settling, and what each observation tells you.→
- What degraded PGs mean after an OSD failureReading the PG state to determine exposure, and distinguishing degraded from the states that are worse.→
- Deciding what to do about a failed OSDFour options with different costs. Choosing between them from the evidence rather than by default.→
- Marking an OSD out and watching the rebalanceThe command, what it triggers, and how to monitor and pace what follows.→
- Stopping the OSD daemon at the right momentWhy the daemon stops after the drain rather than before, and how to stop it cleanly.→
- Purging an OSD from the clusterThe final step, what it removes, and how to verify nothing is left behind.→
Part LVII
Replacing Failed OSDs
6 checks
- Identifying the physical disk behind a failed OSDGetting from an OSD number to a bay in a chassis, without guessing.→
- Draining before replacingThe wait between marking out and pulling the disk, and how to know when it is over.→
- Stopping and purging before the swapClearing the cluster's records so the replacement is a clean deployment rather than a conflict.→
- The physical replacementThe hardware step, the verification around it, and the failure modes specific to it.→
- Creating the replacement OSDDeploying an OSD on the new device, with the DB placement and the drive group considerations that apply.→
- Watching the new OSD fillBackfill into a new OSD, how to monitor it, and when the replacement is genuinely finished.→
Part LVIII
Recovery
6 checks
- What recovery does and how it knows what to copyRecovery restores missing object copies using the PG log to determine exactly which objects need them.→
- The settings that bound recovery speedWhat each throttle controls, how they interact, and which one to reach for.→
- How recovery affects client I/ORecovery and clients share the same devices and network. Quantifying the interference and deciding how much to accept.→
- Per-pool recovery priorityNot all pools matter equally during a recovery. The mechanism for expressing that, and its limits.→
- Backfill priority and how it differs from recovery priorityBackfill is lower priority than recovery by design, and the per-pool control operates within that.→
- Knowing when recovery is finishedSeveral signals, what each confirms, and why the obvious one is not sufficient on its own.→
Part LIX
Backfill
6 checks
- Backfill and recovery: two mechanisms, two triggersThey look similar in the status output and differ in what causes them, what they copy, and how urgent they are.→
- Every change that triggers backfillA catalogue of the operations that move data, with the volume each produces.→
- Controlling backfill speedThe settings that bound backfill, how they differ from the recovery ones, and which apply under mClock.→
- The client impact of backfill and how to bound itBackfill is bulk sequential movement running for extended periods. What that does to clients and how to measure it.→
- Pausing and resuming backfillThe flags that stop data movement, what each covers, and the ones to avoid.→
- Estimating how long a backfill will takeThe arithmetic, the sources of error, and how to give an answer that holds.→
Part LX
Recovery Tuning
6 checks
- The client versus recovery trade-offBoth use the same devices and the same network. Every tuning decision is a choice between them.→
- The recovery settings and what each doesA precise account of the recovery throttles, their units, and their interaction with the scheduler.→
- The backfill settings and their interactionHow the backfill throttles combine, why the scan settings matter, and what to change first.→
- Per-pool recovery priorityNot all pools matter equally during a recovery. How to say so, and what the setting actually does.→
- The balancer and upmapAutomatic rebalancing using upmap exceptions: how it works, how to verify it, and how to control it.→
- Running a recovery tuning cycleBaseline, change, measure, repeat. The discipline that turns tuning into a result rather than an opinion.→
Part LXI
Scrubbing
6 checks
- Scrub and deep scrub: what each actually checksOne compares metadata, the other reads every byte. The difference determines what each can detect.→
- How Ceph schedules scrubsThe intervals, the randomisation, the load check, and why the schedule usually knows better than you do.→
- Scrub impact and how to shape itScrubs read data that clients are also reading. What that costs and how to keep it acceptable.→
- Tuning scrub deliberatelyThe settings that shape scrub, their interactions, and a defensible configuration for common cluster types.→
- When a scrub finds a problemThe health checks a failed scrub raises, what each means, and how to read the detail.→
- Pausing scrubThe flags that stop scrubbing, when pausing is justified, and the discipline for resuming.→
Part LXII
Inconsistent PGs
6 checks
- Detecting an inconsistent PGHow an inconsistency surfaces, what the health output tells you, and what it does not.→
- Diagnosing the inconsistencyWhich object, which shard, which error, and what the device evidence says.→
- Deciding how to repairRepair from the majority, restore from backup, or fix the device first — with the criteria for each.→
- Running the repairThe command, what it does step by step, and how to watch it.→
- Verifying a repairWhat to check after a repair, and how to confirm the data is actually right rather than merely consistent.→
- Preventing inconsistenciesDevice health monitoring, proactive replacement, and the configuration choices that reduce exposure.→
Part LXIII
Capacity Management
6 checks
- Raw capacity and usable capacityThe number on the invoice and the number you can actually store are separated by three multipliers.→
- The cost of replicationWhat size=3 actually costs, what size=2 saves, and why the saving is not worth taking.→
- The cost of erasure codingHow EC overhead is computed, how profiles compare, and the costs that do not appear in the ratio.→
- How much free space to keepHeadroom is not waste — it is the capacity that recovery, backfill, and imbalance consume.→
- Capacity for recovery and backfillRecovery needs somewhere to put the data. Verifying that before it is needed is the whole lesson.→
- Monitoring capacityThe commands, the metrics, and the alerts that give warning while there is still time to act.→
Part LXIV
Nearfull, Backfillfull and Full
6 checks
- The nearfull thresholdThe first warning, what it actually means, and why it is the one with time to act on.→
- The backfillfull thresholdThe point where the cluster stops moving data onto an OSD, and why that is protective rather than punitive.→
- The full thresholdWhere writes stop, what clients see, and why this is the threshold you never want to reach.→
- Why a full cluster cannot heal itselfRecovery, rebalancing, and expansion all need free space. Full removes all three at once.→
- Getting out of nearfullFour remedies, their costs and lead times, and the order to attempt them.→
- Alerting on capacity thresholdsWhich conditions to alert on, at what severity, and what each alert should tell the responder to do.→
Part LXV
Why Full Clusters Are Dangerous
6 checks
- What blocked writes look like from the applicationTracing an application error back to a full Ceph cluster, across each client type.→
- Recovery under capacity constraintWhat recovery needs in order to run, and how it behaves when the space is not there.→
- How a full cluster narrows your optionsThe operations that stop being available, in the order they disappear.→
- Emergency space reclamationWhat can be freed quickly when the cluster is at the wall, in order of speed and reversibility.→
- The full-cluster trap and how it formsThe specific circular dependency that makes a full cluster hard to escape, and where the circle can be broken.→
- Recovering a cluster after a full eventThe sequence back to healthy, what to verify at each stage, and what to fix so it does not recur.→
Part LXVI
Capacity Forecasting
6 checks
- Measuring growth and computing lead timeThe arithmetic that turns a utilisation number into an order date.→
- Retention and churnGrowth is arrivals minus departures. Modelling both is what makes a forecast hold.→
- Planning for new workloadsA new tenant changes the forecast discontinuously. How to fold that into planning.→
- Headroom for a host failureComputing what a single host failure requires, and verifying the cluster has it.→
- Headroom for maintenanceRolling upgrades and hardware work remove capacity temporarily. Planning for that overlap.→
- Keeping a forecast accurateForecasts drift. Recalibration, error tracking, and knowing when to trust the data over the model.→
Part LXVII
Performance Methodology
6 checks
- The layered model for performance workFive layers, each with its own latency contribution. Naming them is what makes an investigation converge.→
- Establishing a performance baselineA baseline is what makes "slow" measurable. How to take one that stays useful.→
- Isolating the slow layerA test per layer that removes the layers above it, so the result is attributable.→
- Correlating evidence across layersThe same event appears differently at each layer. Aligning timestamps is what identifies the cause.→
- Working a slow-ops reportApplying the methodology to the most common performance alert Ceph produces.→
- Working a blocked-ops reportOperations that are not slow but stopped. A different set of causes and a different investigation.→
Part LXVIII
OSD Latency
6 checks
- Finding the slow OSDThe commands that rank OSDs by latency, what each measures, and how to confirm an outlier is real.→
- Connecting OSD latency to what users seeHow a per-OSD number becomes a client-visible percentile, and how to demonstrate the link.→
- Tuning OSD latencyThe settings that shape OSD queueing and threading, and the ones that no longer exist.→
- The device latency floorEvery device class has a floor no amount of tuning goes below. Knowing yours sets realistic expectations.→
- Contention between workloads on shared OSDsPools share devices. What that means for isolation, and what actually provides it.→
- Bounding recovery's cost to client latencyRecovery and clients share the OSD. Setting and enforcing a bound on what recovery may take.→
Part LXIX
Disk Performance
6 checks
- Choosing a device class for a workloadMatching device characteristics to what a pool actually does, and the cost of getting it wrong.→
- Queue depth and where latency accumulatesThe queues an I/O passes through, how depth affects latency and throughput, and what to set.→
- Reading device health dataThe SMART and NVMe attributes that predict failure, and the ones that do not.→
- Flash wear and write amplificationHow Ceph writes multiply before reaching the NAND, and what that means for drive lifetime.→
- Automating device health collectionGetting health data collected, stored, and alerted on without anyone running a command.→
- Deciding when to replace a deviceThe triggers, the thresholds, and the case for replacing before failure.→
Part LXX
Network Performance
6 checks
- When storage latency is really network latencyCeph latency includes several network round trips. Recognising the network's contribution.→
- MTU end to endJumbo frames help when every hop agrees and cause obscure failures when one does not.→
- Bonded links and what happens when one failsA bond is redundancy or capacity, and treating it as both produces a failure you did not plan for.→
- Switch saturation and bufferingWhere frames are dropped, why Ceph is sensitive to it, and how to see it from the switch counters.→
- Asymmetric and unequal pathsECMP and multi-path fabrics can send traffic one way and return it another, with different characteristics.→
- Ceph's network traffic patternsWhat Ceph actually sends, how connection counts scale, and what the fabric must therefore support.→
Part LXXI
Client Performance
6 checks
- librbd client configurationThe client-side settings that shape RBD performance, where to set them, and which actually matter.→
- Kernel RBD and librbd comparedTwo clients with different capabilities, different performance profiles, and different failure behaviour.→
- rbd-nbd and when it is the right choiceA user-space client presenting a block device. What it solves and what it costs.→
- Concurrency and where throughput comes fromAggregate throughput is the product of concurrency and per-operation latency. Both ends matter.→
- Choosing a cache modeWriteback, writethrough, and none — what each guarantees and which workloads suit each.→
- Client-side queueing and threadingWhere client-side work queues, what limits it, and how to tune it for high-concurrency workloads.→
Part LXXII
Benchmarking
6 checks
- rados benchThe RADOS-layer benchmark: what it measures, how to run it correctly, and how to read the output.→
- rbd benchThe built-in RBD benchmark: quick, convenient, and limited in specific ways worth knowing.→
- fio against RBDThe tool for decision-grade measurements: job configuration, percentile reporting, and reproducibility.→
- The per-OSD benchmark`ceph tell osd bench` measures a single OSD directly, which is what mClock uses and what identifies an outlier.→
- Benchmarking without disturbing productionBenchmarks consume the same resources as production. Doing them safely and getting valid numbers.→
- What to capture from a benchmarkThe numbers and context that make a result useful later, and the ones that are commonly missing.→
Part LXXIII
Benchmark Interpretation
6 checks
- Reading a latency distributionWhat each percentile tells you, what the shape of the distribution reveals, and why the mean is nearly useless.→
- IOPS and block sizeIOPS is meaningless without a block size, and the relationship between them is not what it appears.→
- Throughput and saturationWhat saturation looks like, where it occurs, and how to tell which resource is saturated.→
- Benchmarks that misleadThe common ways a benchmark produces a number that does not predict production behaviour.→
- Benchmarking the workload you actually haveSynthetic profiles rarely match production. Characterising a real workload and reproducing it.→
- Tracing tail latency to its sourceThe tail has a cause and the cause is usually identifiable. The method for finding it.→
Part LXXIV
Observability
6 checks
- Getting Ceph metrics into PrometheusThe mgr module, what it exposes, and how it fits an existing monitoring stack.→
- The metrics worth watchingA working set of Ceph metrics that covers health, capacity, performance, and durability.→
- Health status as a metricTurning Ceph's health checks into monitoring signals that are actionable rather than noisy.→
- Monitoring the monitorsQuorum is the cluster's single point of total failure. Watching it properly.→
- OSD state metricsThe up/in matrix as metrics, what each combination means, and which warrant a page.→
- PG state metricsPG states as monitoring signals: which indicate availability, which durability, and which are transient.→
Part LXXV
Prometheus Metrics
6 checks
- Operating the metrics exporterConfiguring the exporter for scale, handling its cost, and dealing with cardinality.→
- Building useful queries from raw metricsThe transformations that turn Ceph's raw series into numbers that answer questions.→
- Labels and cardinalityWhich Ceph metrics carry which labels, how cardinality grows, and how to keep it manageable.→
- Recording rules for CephPrecomputing expensive queries, naming them consistently, and knowing when a rule is worth it.→
- Writing Ceph alerting rulesRules that fire when they should, carry what a responder needs, and route to the right place.→
- Scrape configuration and reliabilityGetting the scrape right, and monitoring the monitoring path itself.→
Part LXXVI
Grafana Dashboards
6 checks
- The cluster overview dashboardThe single screen opened first during an incident, and what belongs on it.→
- The per-pool dashboardDrilling from the cluster to a specific pool: what differs by pool and what the panels must show.→
- The per-OSD dashboardWhere an outlier is confirmed and characterised, and what distinguishes a device problem from a host one.→
- The recovery dashboardUsed only during a recovery, and designed to answer whether it is progressing and when it will finish.→
- The capacity dashboardThe view used for planning and during a capacity incident, and the panels each needs.→
- The host and network dashboardThe layer below Ceph, and the panels that connect a Ceph symptom to a host or fabric cause.→
Part LXXVII
Alerting
6 checks
- The monitor quorum alert and its responseWhat happens when this page fires, what the responder does in the first five minutes, and how to prepare for it.→
- The OSD down alert and its responseWhat to establish in the first minutes, and how the response differs by how many OSDs are involved.→
- Degraded PG alerts and their triageDistinguishing the degraded conditions that need action from those the cluster is handling.→
- The inconsistent PG alert and its handlingA data integrity signal that is rarely urgent and should never be left, and how to handle it correctly.→
- The nearfull alert and the capacity conversationWhat the alert should trigger, and how to convert it into a decision rather than an acknowledgement.→
- The slow ops alert and its escalationAn alert whose severity depends entirely on context, and how to encode that in the response.→
Part LXXVIII
Monitoring the Monitoring Path
6 checks
- Every part of the telemetry path can fail silentlyThe failure modes of a monitoring pipeline, and why each is invisible from inside it.→
- Redundant Prometheus for Ceph monitoringRunning two Prometheus instances, what that protects against, and what it does not.→
- Redundant AlertmanagerThe clustering model, what the gossip protocol synchronises, and the failure modes that remain.→
- Grafana availability and what it actually needsGrafana is a query and rendering layer; making it redundant is simple once its state is understood.→
- Scaling the monitoring stack with the clusterMonitoring capacity grows with the cluster, and running out of it is a monitoring outage.→
- Operating the monitoring stackThe monitoring path needs the same discipline as the cluster: alerts, runbooks, change control, and review.→
Part LXXIX
Slow Ops
6 checks
- What Ceph counts as a slow operationThe threshold, the accounting, and what the report is actually measuring.→
- Slow ops originating at an OSDWhen the OSD itself is the delay: the evidence, the causes, and the remedies.→
- Slow ops originating at the clientWhen the delay is above the cluster: what the cluster sees, and how to establish it.→
- Slow ops originating in the networkThe network signatures in slow ops, and the tests that confirm them.→
- Slow ops caused by recoveryRecovery-induced slow ops: confirming the attribution, deciding whether to act, and what to change.→
- Correlating slow ops with cluster stateA decision procedure that turns a slow ops report into a specific cause in a few minutes.→
Part LXXX
Blocked Operations
6 checks
- Blocked and slow: the distinction that changes everythingTwo conditions that look alike in the health output and need entirely different responses.→
- Blocked by capacity thresholdsWhich thresholds block which operations, and how to restore progress at each.→
- Pool-level blocking conditionsQuotas, application tags, and pool state — the conditions that block one pool while the cluster is healthy.→
- Blocked by insufficient available copiesPGs below min_size: what it means, how it happens, and the options for restoring I/O.→
- Blocked by forgotten flagsWhat each OSD map flag actually prevents, and the harm a forgotten one causes.→
- The blocked operations sweepA single procedure that identifies any blocking condition in under two minutes.→
Part LXXXI
Proxmox Integration
6 checks
- RBD as Proxmox VE storageConfiguring RBD storage, what Proxmox manages for you, and the settings that matter.→
- CephFS as Proxmox storageWhere a shared filesystem fits alongside RBD, and how to configure it.→
- Hyper-converged Proxmox and CephRunning both on the same hosts: what it saves, what it costs, and how to configure it well.→
- A dedicated Ceph cluster for ProxmoxSeparating storage from compute: what it costs, what it buys, and how the deployment differs.→
- Network design for Proxmox with CephThe networks involved, what each carries, and the failure modes of getting the separation wrong.→
- Failure domains for Proxmox workloadsAligning Ceph placement with the physical topology so a rack or power failure does not take out a pool.→
Part LXXXII
Hyper-Converged Ceph
6 checks
- The economics of hyper-convergenceWhat it actually saves, where the saving disappears, and how to compare honestly.→
- Managing compute and storage contentionThe specific resources that contend, how to measure the contention, and the limits that bound it.→
- Correlated failure in a hyper-converged clusterWhen a node fails it takes compute and storage together, and the two recoveries compete.→
- Maintenance on a hyper-converged nodeTaking a node out when it runs both layers: the sequence, the flags, and the verification.→
- Scaling a hyper-converged clusterWhat adding a node does to both layers, and where the approach stops scaling well.→
- Migrating from hyper-converged to dedicatedThe signals that the architecture no longer fits, and how to make the transition without an outage.→
Part LXXXIII
Dedicated Ceph Cluster
6 checks
- Designing a dedicated storage clusterHost roles, daemon placement, and the decisions that are hard to change later.→
- Network design for a dedicated clusterSizing, separation, and redundancy for the networks a dedicated Ceph cluster needs.→
- Monitoring a dedicated clusterWhat differs when the cluster is independent, and the signals that matter most.→
- Running a dedicated cluster as a serviceThe operational practices that make an independent storage cluster dependable for its consumers.→
- Comparing the architectures honestlyA decision framework rather than a preference, with the criteria that actually distinguish them.→
- Serving multiple consumers from one clusterIsolation, attribution, and fairness when several independent teams share a storage cluster.→
Part LXXXIV
Proxmox Failure Scenarios
6 checks
- Diagnosing VM disk latency on CephA guest reports slow disks. Working the layers from guest to OSD to find where the time goes.→
- An OSD failure and its effect on running VMsWhat guests experience when an OSD fails, and how to bound the impact.→
- Coordinating Proxmox and Ceph maintenancePlanning work that touches both layers, and the ordering that avoids self-inflicted incidents.→
- Ceph quorum loss with VMs runningWhat happens to running guests when the storage cluster loses quorum, and how long they survive.→
- Path redundancy with RBDWhy RBD needs no multipath, what it does need, and the SAN assumptions that do not transfer.→
- Total storage loss with VMs runningThe worst case: what guests do, what can be recovered, and how to prepare for it.→
Part LXXXV
Kubernetes Integration
6 checks
- The CSI architecture for CephWhat the driver components do, where they run, and which failures affect what.→
- Deploying and configuring Ceph-CSIThe deployment, the secrets, the configuration, and verifying it works end to end.→
- Designing StorageClasses for CephThe parameters that matter, how many classes to define, and the settings that cannot be changed later.→
- PersistentVolumeClaims and their lifecycleWhat happens at each stage, where volumes get orphaned, and how to manage the lifecycle deliberately.→
- Topology-aware volume placementAligning Kubernetes zones with Ceph failure domains so pods and their data stay together.→
- Volume encryption with Ceph-CSILUKS encryption at the CSI layer: what it protects, where keys live, and the operational consequences.→
Part LXXXVI
Kubernetes RBD
6 checks
- RBD volumes and access modesWhat ReadWriteOnce means for a block device, why ReadWriteMany does not work, and the exceptions.→
- RBD StorageClass parameters in depthThe less obvious parameters, what each changes, and the combinations that cause problems.→
- RBD image layout for Kubernetes volumesObject size, striping, and thick provisioning — what each changes for a PVC.→
- RBD image features and kernel compatibilityWhat each feature does, which clients support it, and how to choose a set that works everywhere.→
- Aligning Kubernetes and Ceph topologyMaking the two systems agree about where things are, and what breaks when they do not.→
- CSI snapshots and what they are notVolumeSnapshot mechanics, the restore path, and why a snapshot is not a backup.→
Part LXXXVII
Kubernetes CephFS
6 checks
- CephFS subvolumes as Kubernetes volumesWhat a subvolume is, how CSI uses it, and the layout that results in the filesystem.→
- CephFS snapshots through CSISubvolume snapshots, cloning, and the consistency and performance characteristics.→
- Shared CephFS volumes across podsWhat ReadWriteMany actually guarantees, the locking that works, and the patterns that do not.→
- Permissions and identity on CephFS volumesUID and GID handling, fsGroup, and the mismatches that produce permission errors.→
- CephFS StorageClass configurationThe parameters, the mounter choice, and the settings that affect performance and isolation.→
- CephFS failure behaviour in KubernetesMDS failover, client eviction, and what pods experience during each.→
Part LXXXVIII
Kubernetes Storage Failure Scenarios
6 checks
- Diagnosing a volume that will not mountA pod stuck in ContainerCreating: the sequence of checks that finds the cause.→
- CSI driver failures and their blast radiusWhat breaks when each driver component fails, and how to recover without making it worse.→
- Authentication failures between Kubernetes and CephThe failure modes of cephx from CSI, and how each presents.→
- Kubernetes workloads during a Ceph cluster problemHow each class of Ceph problem propagates to pods, and what to do about each.→
- Stale snapshots and orphaned referencesWhen Kubernetes objects and Ceph state diverge, how it happens, and how to reconcile.→
- Volume density limits per nodeHow many volumes a node can actually handle, what the limits are, and how to configure them.→
Part LXXXIX
Rook Concepts
6 checks
- Rook: Ceph as a Kubernetes operatorWhat Rook manages, the custom resources it exposes, and how the reconciliation model differs from imperative management.→
- Rook and cephadm comparedTwo orchestrators for the same storage system, and what each assumes about the environment.→
- When Rook is the right choiceThe situations where running Ceph inside Kubernetes is the better answer, and what it requires.→
- When cephadm is the right choiceThe situations where a standalone Ceph cluster is the better answer, and what cephadm provides.→
- Why this course teaches Ceph directlyWhat transfers between orchestrators, what does not, and why the underlying system is what matters.→
- Continuing beyond this courseWhere to go next for each direction the material points toward, and how to keep current.→
Part XC
Scaling Out
6 checks
- Adding OSDs to an existing clusterThe sequence, the verification at each step, and the mistakes that cost a second rebalance.→
- Adding a host to the clusterPreparation, adoption, placement into the CRUSH hierarchy, and the verification that matters.→
- Changing the monitor countWhen more monitors help, when they do not, and how to add or remove them safely.→
- Manager placement and redundancyWhat the manager does, what its failure costs, and how many to run.→
- Managing the rebalance an expansion causesHow much data moves, how long it takes, and how to shape it around the workload.→
- Expanding during degraded conditionsWhen adding capacity during a recovery helps, when it compounds the problem, and how to decide.→
Part XCI
Adding Storage Nodes
6 checks
- Pre-check: failure domain placementEstablishing where the node physically is and whether the CRUSH hierarchy can express it.→
- Pre-check: network configurationVerifying the new node matches the cluster's network configuration in every respect that matters.→
- Pre-check: devices and capacityConfirming the devices are present, healthy, and appropriate before they become OSDs.→
- Pre-check: CRUSH map preparationPreparing the hierarchy and verifying rules before the node contributes capacity.→
- Pre-check: capacity and impact for the backfillConfirming the cluster can absorb the movement the addition causes, and planning its impact.→
- Executing the node additionThe full sequence with the verification at each step, and what to do if a step fails.→
Part XCII
Removing Storage Nodes
6 checks
- Evacuating a nodeThe two ways to drain a node, what each does to placement, and which to use.→
- Verifying an evacuation is completeWhat "empty" means precisely, and the checks that confirm it before anything is removed.→
- Removing the OSDsThe commands, what each does, and the difference between removing and purging.→
- Removing the hostTaking the host out of the cluster once its OSDs are gone, and what remains afterwards.→
- CRUSH map hygiene after removalsThe residue removals leave, why it accumulates, and how to keep the map an accurate description.→
- The removal order and why it existsWhat each step protects against, and what happens when the order is not followed.→
Part XCIII
Changing CRUSH Topology
6 checks
- CRUSH weight and OSD reweightTwo different weights, what each controls, and when to use which.→
- Bulk weight correctionsWhen weights across the cluster are wrong, and how to correct them without an unbounded rebalance.→
- Moving buckets in the hierarchyRelocating hosts and OSDs within the CRUSH tree, and the movement each move causes.→
- Changing a CRUSH ruleWhich rule changes are safe, which move everything, and how to make either predictable.→
- Blast radius of a topology changeEstimating what a change affects, bounding it, and communicating it before it happens.→
- Monitoring a topology changeWhat to watch during the movement, what indicates a problem, and when to intervene.→
Part XCIV
Hardware Replacement
6 checks
- Replacing a failed diskThe sequence from failure to a restored OSD, with the physical identification step that goes wrong most often.→
- Replacing a failing disk before it failsThe planned version of the same operation, and what the planning buys.→
- Replacing a failed serverWhen a whole host is lost: assessing the situation, recovering, and deciding whether to restore or replace.→
- Bringing a replacement node into serviceAdding a replacement for a lost host, and the differences from an ordinary expansion.→
- Verifying replacement hardware before deploymentCompatibility, firmware, and the differences that cause problems after the hardware is in service.→
- Validating a replacement after deploymentConfirming the replacement is correct, performing as expected, and that the cluster returned to its intended state.→
Part XCV
Maintenance Flags
6 checks
- The noout flag in depthWhat it changes in the monitors' behaviour, when it is correct, and its per-OSD and per-host variants.→
- The norecover flag in depthWhat it stops precisely, how it differs from the backfill flags, and its narrow legitimate uses.→
- The norebalance flag in depthThe one suppression flag safe to use routinely, and exactly why.→
- The nobackfill flag in depthThe broader suppression, what it stops that norebalance does not, and when the difference matters.→
- The scrub suppression flagsWhat each stops, the resumption behaviour, and the coverage cost of extended suppression.→
- Flag lifecycle managementMaking flags temporary in practice: ownership, expiry, automation, and audit.→
Part XCVI
Node Maintenance
6 checks
- Reboots and the down-out intervalWhy a reboot needs noout rather than out, and how the interval determines the approach.→
- Kernel and host software upgradesUpgrading the host beneath Ceph, the version sensitivities that matter, and the rollback path.→
- What clients experience during node maintenancePredicting and measuring the client-visible effect, and bounding it.→
- Monitoring during a maintenance windowWhat to watch, what would change the plan, and how to keep the window bounded.→
- Rolling back a maintenance changeWhat can be rolled back, what cannot, and how to preserve the option.→
- Post-maintenance verificationConfirming the cluster returned to its prior state, not merely that the work completed.→
Part XCVII
Network Maintenance
6 checks
- Switch maintenance with redundant uplinksTaking a switch out of service without disrupting Ceph, and verifying the redundancy actually works first.→
- Individual link and NIC maintenanceReplacing a cable, transceiver, or NIC on a live Ceph host.→
- Changing MTU across a live clusterCoordinating an end-to-end MTU change without a window where the cluster is split.→
- Network maintenance during a degraded clusterWhy network work and recovery interact badly, and how to sequence them.→
- Client impact during network maintenanceWhat clients experience, what determines it, and how to bound it.→
- Network change rollbackPreserving the ability to reverse a network change, and executing it when the cluster is partly unreachable.→
Part XCVIII
Software Upgrades
6 checks
- Ceph release cadence and supported upgrade pathsHow releases are named and supported, which paths are permitted, and how to plan a version strategy.→
- Running a cephadm upgradeThe command, what it does, how to monitor it, and how to pause or stop it.→
- The upgrade order and why it existsWhat each stage depends on, and what happens when the order is violated.→
- Pre-upgrade readinessThe checks that must pass before starting, and what each protects against.→
- Client version compatibilityWhat clients need to support, how to check, and how to handle a fleet that cannot be upgraded together.→
- Upgrade rollback and the backup that replaces itWhat can and cannot be reversed, and how to prepare for the case where nothing can.→
Part XCIX
Upgrade Planning
6 checks
- Reading release notes as an operational documentHow to extract the parts that affect your cluster from a long changelog.→
- Establishing the pre-upgrade baselineCapturing what normal looks like so post-upgrade comparison is possible.→
- What to back up before an upgrade, and what a backup cannot doThe realistic scope of pre-upgrade backup and where its limits are.→
- Building a test cluster that predicts productionWhat a staging cluster must match to be predictive, and what it can safely differ in.→
- Planning and communicating the upgrade windowEstimating duration, defining the abort criteria, and setting expectations with stakeholders.→
- The post-upgrade watch periodWhat to check immediately, what emerges over hours, and what only shows up over days.→
Part C
Upgrade Failure Recovery
6 checks
- A daemon fails to start after upgradingDiagnosing why a single daemon will not come up on the new version.→
- Operating a cluster stuck in a mixed-version stateWhat works, what does not, and how to get out.→
- A manager module fails after upgradingDiagnosing module load failures, disabling safely, and what a module failure affects.→
- Manager failover during an upgradeWhat happens when the active manager restarts, what is briefly lost, and what to verify.→
- A daemon crash-looping after an upgradeBreaking the restart loop, capturing evidence, and deciding how to proceed.→
- The post-upgrade reviewCapturing what happened so the next cluster and the next upgrade go better.→
Part CI
Security Hardening
6 checks
- A threat model for a Ceph clusterWhat an attacker can reach at each position, and which control stops what.→
- Segmenting and firewalling a Ceph clusterThe port inventory, which traffic must cross which boundary, and how to verify the rules.→
- Capability lifecycle: issuance, drift, and reviewKeeping least privilege true over time rather than only at creation.→
- Verifying encryption is actually in effectConfirming secure mode is negotiated rather than merely configured, on every path.→
- Hardening a Ceph hostReducing what a host compromise reaches, for hosts that run containers as root.→
- Running a repeatable hardening auditA script that produces evidence rather than opinions, and what to do with its output.→
Part CII
Management Security
6 checks
- Securing the cephadm SSH pathHow cephadm reaches every host, and how to constrain that access.→
- Controlling where `client.admin` existsReducing the number of places full cluster control is available.→
- Securing the Ceph dashboardAuthentication, TLS, exposure, and what the dashboard permits.→
- Securing the monitoring stackWhat Prometheus, Grafana, and Alertmanager expose, and how to constrain them.→
- The manager REST API and programmatic accessHow automation authenticates, and how to scope it.→
- Manager modules as an attack surfaceWhat each enabled module exposes and reaches, and how to decide what to run.→
Part CIII
Secrets and Key Management
6 checks
- Finding every copy of a keyringKeys spread further than the hosts that need them; a sweep is how you find out where.→
- Distributing keys without copying files aroundGetting a key to a consumer through a path that can be audited and revoked.→
- Making key rotation operationally possibleThe prerequisites that decide whether rotation is routine or a project.→
- Backing up and restoring the auth databaseWhat a keyring backup protects against, how to take one, and how to restore selectively.→
- The monitor store as the cluster's secret storeWhat lives in the monitor store beyond the maps, and what protecting it means.→
- Recovering from a key change that broke clientsDiagnosing authentication failures after a key change and restoring service.→
Part CIV
Multi-Tenancy in Practice
6 checks
- Tenant onboarding as a repeatable procedureTurning tenant provisioning into something scripted, consistent, and reversible.→
- Operating CephFS subvolume tenancy over timeGrowth, quota changes, snapshots, and the client-enforced quota caveat.→
- RGW multi-tenancy in operationUsage accounting, key lifecycle, and the eventual consistency of RGW quotas.→
- Namespaces when pool-per-tenant stops scalingMoving to namespace isolation, what it costs, and what it cannot do.→
- What happens when a tenant hits their limitThe behaviour at each quota layer, and how to make the limit an event rather than an outage.→
- Continuous isolation verification and cross-tenant incidentsKeeping isolation true after launch, and handling one tenant affecting another.→
Part CV
Backup Strategy
6 checks
- Why replication is not backupWhat redundancy protects against, what it does not, and why the distinction is repeatedly lost.→
- Enumerating the threats a backup must answerNaming each threat precisely enough that a control can be matched to it.→
- File-level and object-level backup pathsBacking up through the access layer rather than the storage layer, and when that is right.→
- Snapshots as a control: what they cover and what they costWhere snapshots genuinely protect, their capacity behaviour, and their limits.→
- Crash consistency versus application consistencyWhat a snapshot actually captures, and what an application needs on top of it.→
- Setting RPO and RTO before designing anythingDeriving the backup design from the targets rather than the reverse.→
Part CVI
RBD Backup
6 checks
- Full RBD export: mechanics and costExporting an image to a file, what it produces, and where the time goes.→
- Incremental export with export-diffTransferring only what changed, and the snapshot chain that makes it work.→
- Coordinating backup with the guestGetting a quiesced snapshot from outside the VM, and what to do when you cannot.→
- Restoring an RBD image from an exportImporting, reattaching, and verifying — including the steps that are not the transfer.→
- Rollback versus clone: choosing the recovery shapeTwo ways to recover from a snapshot, with very different consequences.→
- Meeting RBD recovery targets at scaleWhat backup cadence and restore paths actually deliver for a fleet of images.→
Part CVII
CephFS Backup
6 checks
- CephFS snapshots: the .snap directory and its behaviourHow CephFS snapshots are created, where they appear, and what they cost.→
- Off-cluster file backup with deduplicating toolsUsing restic or similar against CephFS, and the tuning that makes it finish.→
- Subvolume snapshots and asynchronous clonesThe managed snapshot interface, cloning from a snapshot, and how the clone progresses.→
- Quiescing shared-filesystem workloadsGetting a consistent snapshot when many clients write to the same tree.→
- Restoring CephFS dataRecovery paths from snapshot, clone, and off-cluster backup — and their very different costs.→
- CephFS recovery targets and mirroringMeeting RPO and RTO for a filesystem, including snapshot mirroring to a peer.→
Part CVIII
RGW Backup and Replication
6 checks
- Copying buckets with the S3 APISynchronising a bucket to another target, and where the approach breaks down.→
- RGW multi-site replication as a DR mechanismHow multi-site works, what it protects against, and what it does not.→
- Building an RGW backup inventoryKnowing what exists, who owns it, and what actually needs protecting.→
- Bucket configuration that a data copy does not carryPolicies, ACLs, lifecycle rules, and the settings a restore must recreate.→
- Versioning and object lock as in-place protectionMaking a bucket resistant to deletion and encryption without copying it anywhere.→
- RGW recovery targets and failoverMeeting RPO and RTO for object storage, and executing a zone failover.→
Part CIX
Disaster Recovery
6 checks
- Classifying a failure by what it costsDegraded, inactive, and lost are three different situations needing three different responses.→
- The dependency order of recoveryWhat must be working before anything else can be, and why order determines outcome.→
- Scenario modelling: what survives each failureWorking out in advance which copies remain after each plausible failure.→
- Rebuilding a cluster from nothingThe full sequence from bare hosts to serving data, and where the time goes.→
- Writing a DR plan people can actually executeWhat a usable plan contains, and what makes plans fail at the moment they are needed.→
- Running DR drills that find somethingDesigning drills that produce findings rather than reassurance.→
Part CX
Monitor Recovery
6 checks
- The monmap and the cluster identityWhat the monmap records, how to extract it, and why rank is not a name.→
- Replacing a lost monitorRemoving the dead one first, rebuilding the store, and why the order is not arbitrary.→
- Backing up and restoring the monitor storeHow to take a consistent copy, what it is actually good for, and when restoring it is wrong.→
- What the monitor store holds and how it is actually protectedCeph does not encrypt the monitor store. Where the secrets are and what protects them.→
- Moving a monitor to a different addressWhy monitors are found by address, the safe path, and the in-place path when there is no safe one.→
- How a monitor rejoins quorumThe states a monitor passes through, what blocks each one, and why a running daemon proves nothing.→
Part CXI
Manager Recovery
6 checks
- The manager active and standby modelOne active, the rest idle — what promotion actually costs and why the standby is cold.→
- Replacing a manager daemonFailing over before you operate, the cephadm path, and the path when there is no orchestrator.→
- Module state across a manager failoverWhat survives because it lives in the monitors, and what is lost because it lives in a timer.→
- Operating a cluster with no managerWhat keeps serving, what stops, and why the orchestrator cannot deploy its own daemon.→
- Verifying manager stateThe three views that matter, and why a healthy manager can be serving nothing.→
- CephFS through a manager outageMDS failover needs no manager; CephFS provisioning does. Where the two standby models diverge.→
Part CXII
OSD Host Loss
6 checks
- How the cluster learns a host is goneHeartbeat detection, the reporter rules, and what down means before anything is marked out.→
- Deciding whether to let recovery startThe down-out interval, the cost of waiting, and setting noout at the right scope.→
- Marking out and the two data movementsWhat out changes in the map, why removal is a different and larger operation, and what a returning host costs.→
- How much more you can afford to loseReading remaining tolerance from the CRUSH rule and the PG states, and what ok-to-stop does not tell you.→
- Whether the recovery will fitHeadroom arithmetic, the backfillfull stall, and why average utilisation is the wrong number.→
- Controlling how fast recovery runsmClock profiles, why the classic backfill throttles are ignored, and estimating completion.→
Part CXIII
Multiple OSD Failure
6 checks
- Below min_size and still holding the dataWhat a PG does when its acting set drops beneath min_size, and what lowering it actually buys.→
- When a PG has no surviving copyReading down and incomplete, probing for the data, and the order of the last resorts.→
- Reading the shape of a correlated failureGrouping down OSDs by CRUSH position, dating the failure from the osdmap, and stabilising a flapping fabric.→
- Tolerance is a per-pool propertyDifferent rules, different min_size, and why the smallest pool usually fails first.→
- The imbalance recovery leaves behindWhy survivors end up uneven, when the balancer will act, and how read load follows the primaries.→
- When more domains fail than the rule allowsThe arithmetic of a two-domain loss, what to prioritise while most PGs are inactive, and testing the design afterwards.→
Part CXIV
Complete Storage Node Loss
6 checks
- When the node is never coming backThe cascade from a permanently lost storage node, and the decision that starts the redundancy clock.→
- Removing a host that cannot be drainedThe teardown path when the orchestrator cannot reach the daemons it is supposed to stop.→
- How long the rebuild actually takesEstimating re-replication time from measured rate, and the levers that are no longer where you left them.→
- Whether the rebuild can finish at allThe CRUSH arithmetic that decides if redundancy returns, and what a second failure would cost meanwhile.→
- Bringing the replacement into the clusterAdding replacement hardware without paying for a second uncontrolled data movement.→
- Two nodes gone at onceTriage when the loss exceeds the failure domain budget and some PGs will not peer.→
Part CXV
Cluster-Wide Capacity Incident
6 checks
- Finding where the cluster is actually fullLocating fullness by OSD, pool, and device class while the cluster average is still comfortable.→
- Mapping the blast radiusFrom a full OSD to the clients that notice, and the second incident growing behind the first.→
- The first thirty minutesStopping the inflow, running the workstreams in parallel, and knowing whether you are winning.→
- When the imbalance is the emergencyDiagnosing distribution rather than volume, and moving data with a dry run before anything commits.→
- Adding capacity while the cluster is already at the limitWhy new OSDs raise usage before they lower it, and why they can stay empty entirely.→
- Raising the ratios as a bridge, not a fixWhat a threshold change actually grants, the ceiling no ratio can move, and the bookkeeping that makes it survivable.→
Part CXVI
Network Partition
6 checks
- Reading the first ninety seconds of a partitionThe symptom set that identifies a split, and the timers that produce it.→
- One-way reachability and the flap it producesWhy an OSD that keeps coming back up is reporting a directional fault.→
- Why a partition does not corrupt your dataThe mechanisms that stop two sides of a split from both accepting writes.→
- Which network partitioned, and how the symptoms differFront and back heartbeat failures point at different networks and different client experiences.→
- The cost of the partition healingThe second data movement, the mClock ceiling, and estimating the catch-up.→
- Reconstructing the partition from evidenceThe timeline you owe afterwards, and the artefacts that can still produce it.→
Part CXVII
Lost Monitor Quorum
6 checks
- Three monitors and what the arithmetic actually buysQuorum is computed from the monmap, and a synchronising monitor does not vote.→
- Five monitors and the placement that makes them countFailure tolerance comes from where the monitors sit, not from how many there are.→
- Getting one monitor back is the whole jobRestoring a failed monitor to service in preference order, and proving quorum held.→
- Forcing quorum by editing the monmapThe last-resort surgery, exactly, and the state you are agreeing to lose.→
- Triaging why quorum is goneDistinct causes with distinct signatures, because the recovery differs for each.→
- The artefacts and the drill that make the recovery routineWhat must be stored outside the cluster, and what the drill has to prove.→
Part CXVIII
Data Integrity Incident
6 checks
- Recognising a data integrity incidentTelling an isolated scrub error apart from corruption with a systemic cause.→
- Choosing which copy is authoritativeReading the shard record, and what to do when no copy can be shown to be right.→
- Deciding between repair and restoreThe asymmetry between the two, and why the corruption window decides which backup is usable.→
- When repair does not repairRequests that never start, repairs that recur, and counters that lie about the outcome.→
- Restoring corrupt data from backupMapping a damaged RADOS object to the thing that owns it, and putting the bytes back safely.→
- Finding the component that produced the corruptionSeparating a bad drive from a bad path from bad memory, and replacing it without losing evidence.→
Part CXIX
Disaster Recovery Architecture
6 checks
- Choosing the failure domain a second copy must surviveWhat CRUSH, stretch mode and an external copy each protect against, and where every in-cluster design ends.→
- Tiering copies for two different recovery timesA local tier for the restores that actually happen, and an offsite tier for the one that ends the company.→
- Recovery paths that do not depend on what failedAuditing the restore path for shared equipment and circular dependencies on the cluster itself.→
- Measuring the restore time you actually haveTurning a claimed RTO into a measured one, phase by phase, under realistic constraints.→
- Encrypting backups and keeping the keys usableWhere encryption belongs, what OSD encryption does not cover, and why rotation never destroys a key.→
- Separating cluster operations from backup operationsPush versus pull, least-privilege identities, and making cluster compromise stop at the cluster.→
Part CXX
Multi-Site Concepts
6 checks
- What RGW multi-site does that block and file mirroring cannotActive-active zones, per-bucket sync policy, and the asymmetric topologies only the object interface supports.→
- Delivered RPO across three replication mechanismsMeasuring the lag each mechanism actually delivers, and why a multi-interface service inherits the worst one.→
- Synchronous cross-site replication and its latency ceilingCeph can replicate synchronously between sites; the arithmetic of why that is rarely the right choice.→
- Stretch mode: what enabling it actually changesThe peering barrier, the forced pool geometry, degraded stretch mode, and the way back to healthy.→
- What a replicated copy is consistent withCrash consistency, snapshot boundaries, and the multi-volume recovery point no mechanism gives you.→
- Costing a second site, and what to replicate to itThe multipliers beyond raw capacity, and using each mechanism at its real granularity.→
Part CXXI
Storage Architecture Decision-Making
6 checks
- Testing a workload against Ceph before committing to itThe structural properties that make a workload a good fit, and how to measure the fit rather than assert it.→
- Where Ceph is the wrong answerThe latency floor, the metadata bottleneck, the size below which overhead dominates, and the team test.→
- Three hosts: the utilisation ceiling and the repair windowWhat a three-host cluster can absorb, the free-space arithmetic it imposes, and what has to be accepted in writing.→
- What the sixth and seventh hosts buySelf-healing, erasure coding that can actually heal, rack failure domains, and the cost of promoting one.→
- Ceph on cloud infrastructure: when it is a real choiceWhich durability you are paying for twice, what the cost line items become, and the instance lifecycle that changes OSD operations.→
- The architecture decision record and its as-built captureRecording the requirement, the rejected alternatives, the accepted risks, and what would invalidate the decision.→
Part CXXII
Small Cluster Risks
6 checks
- Recovery when there are only two survivorsWhat bounds re-replication on a small cluster, and when there is no re-replication at all.→
- Maintenance with no spare failure domainEvery reboot on a three-host cluster is a window with no fault tolerance left in it.→
- Usable capacity when there are few OSDsWhy the fullest OSD sets the ceiling, and what a three-host cluster can usefully reserve.→
- What three and four hosts actually tolerateCounting the failures a small cluster survives, separately for data and for quorum.→
- Patterns that help, and ones that only look like they doDevice classes and physical spares against the changes that trade correctness for a green health status.→
- The triggers that mean another host is dueRecognising when honest operation needs more failure domains rather than more drives.→
Part CXXIII
Capacity and Failure Planning
6 checks
- The capacity budget: what may actually be committedSplitting usable capacity into live data, reserves, and the allowance that runs until the next delivery.→
- Growth composition and the real lead timeSeparating drift from step changes, and counting every stage between the decision and a usable OSD.→
- Sizing for one host down, or for twoWhat each level of failure tolerance costs as a share of usable capacity, and when the host is the wrong unit.→
- Making a reserve realA reserve lives on individual OSDs and has to be defended, not recorded.→
- The maintenance reserve is a scheduling decisionWhether maintenance may overlap a degraded cluster is what determines how much capacity you buy.→
- Working the numbers to an orderA worksheet that ends in a date and a quantity, plus the placement group budget that constrains it separately.→
Part CXXIV
Production Reference Architecture
6 checks
- The six-node reference topologyA concrete node specification, the ratios behind every number, and what two racks actually survive.→
- Pool and daemon layout for RBD, CephFS, and RGWThe whole storage design on one page — pools, device classes, erasure coding, and where each daemon runs.→
- One identity per consumerProxmox, Kubernetes and Linux clients, each with its own cephx entity, cap set and blast radius.→
- The network the design assumesPublic and cluster networks, LACP hashing, MTU, and the failure mode a split network adds.→
- The monitoring stack that ships with the designPrometheus, Grafana and Alertmanager under cephadm, and thresholds derived from how long a fix takes.→
- The design as a repository, and the acceptance testService specs in Git, drift detection, the recovery kit, and how to prove a build matches its design.→