Ceph & Distributed Storage for Production Sysadmins — Final Theory Assessment

1. Q1. A client writes a 4 MiB object to a size=3 replicated pool. Which sequence correctly describes how that write is placed and acknowledged?

   The client computes placement from CRUSH and writes one copy to each of the three OSDs in parallel, acknowledging as soon as the first of the three commits and letting the other two converge in the background The client sends the write to a monitor, which looks up the acting set in the current OSDMap, forwards the object to the primary OSD on behalf of the client, and acknowledges once the primary has committed locally The client computes placement from CRUSH, sends the write to the primary OSD of the acting set, and the primary writes locally and forwards to the two replicas, acknowledging the client only after all copies have committed The client sends the write to the active manager, which picks the least loaded OSD from its device telemetry, stores the object there, and schedules asynchronous replication onto two further OSDs chosen by the balancer

2. Q2. The single active ceph-mgr daemon in a production cluster crashes and no standby was ever deployed. Which statement best describes what the cluster does next?

   Client I/O continues normally, but the dashboard, the Prometheus metrics endpoint, the balancer, the PG autoscaler, and ceph orch all stop working until a manager runs again Client I/O blocks immediately, because every librados write is brokered by the manager, which maps the object name to an acting set before the client contacts an OSD The monitor leader takes over the manager modules automatically, so the dashboard and the metrics endpoint keep serving and nothing is lost until an operator redeploys a manager Every PG drops out of active and stops serving, because peering requires the manager to publish the current PG map to the acting sets and OSDs have no authoritative mapping without it

3. Q3. Because CRUSH placement is deterministic, a Ceph client must still ask a monitor for the location of each object before it can read or write that object.

   True False

4. Q4. A colleague argues that Ceph is really just an object store, and that RBD, CephFS, and RGW are thin protocol translators bolted on top. Explain which part of that is accurate, and state what each of the three services adds that RADOS itself does not provide.

5. Q5. A size=3 replicated pool uses a CRUSH rule containing step take default class ssd, step chooseleaf firstn 0 type rack, step emit. The cluster has three racks, but SSD OSDs were only ever installed in two of them. What is the most likely observed result?

   CRUSH retries the failed step against the whole default root, so the third replica lands on an HDD OSD in the remaining rack and the pool still reaches active+clean with a mixed device class Pool creation is rejected, because the monitors test the rule against the current OSDMap and refuse any pool whose rule cannot return size distinct OSDs from the map as it stands All PGs place three replicas, but CRUSH relaxes the rack constraint for the third slot, so two of them share a rack and the failure domain is silently weakened A large number of PGs sit in active+undersized+degraded, because CRUSH can only find two racks that contain candidate OSDs and returns two OSDs where three were requested

6. Q6. One OSD in a 60-OSD cluster is persistently much fuller than its peers. Which command changes that OSD weight in the CRUSH map itself, as opposed to setting an override that is reset back to 1 when the OSD is marked out and then in again?

   ceph osd reweight osd.17 0.9 ceph osd crush reweight osd.17 3.4 ceph osd out osd.17 ceph osd primary-affinity osd.17 0.5

7. Q7. Your cluster has 12 hosts spread evenly across 3 racks, and every pool is size=3 with a host-level failure domain. Management asks whether the cluster survives losing a rack. Describe what you would check, give the honest answer, and explain what you would change and what that change would cost.

8. Q8. A colleague changed the failure domain of the production replicated rule from host to rack during business hours by editing and re-injecting the CRUSH map. Client latency has tripled and the cluster reports tens of thousands of misplaced objects. Describe the evidence you would collect, how you would stabilise the cluster, and the rollback you would keep ready.

   A 4-rack, 24-host cluster running a size=3 replicated pool that holds RBD images for roughly 300 virtual machines. The new rule is valid and CRUSH can satisfy it, so no PG is undersized and no data is at risk, but around 60 per cent of PGs are now remapped and backfilling. Client I/O is degraded but not blocked, and the change was not in the change calendar.

9. Q9. A cluster runs five monitors. Two of them are lost simultaneously when a rack loses power. Which statement is correct?

   Quorum is lost, because a five-monitor cluster requires four members to commit, and client I/O stops across the whole cluster as soon as the majority is broken Quorum is lost, but already-connected clients keep reading and writing indefinitely from their cached OSDMap, and only new connections and new mounts fail Quorum is retained by the remaining three monitors, and the cluster continues to elect a leader and commit map changes normally The surviving monitors shrink the monmap to three members automatically, so the majority requirement drops to two and the original failure margin is restored

10. Q10. A three-monitor cluster loses two monitor hosts permanently to a hardware disaster. One monitor survives with an intact store, and all OSD hosts are healthy. What is the correct recovery path?

    Extract the monmap from the surviving monitor, remove the two dead monitors from it, inject the edited monmap back into the surviving monitor, start it so it forms a quorum of one, and then add replacement monitors normally Start the surviving monitor with the configuration option that lowers the majority requirement to a single member, keep the cluster serving in that state while replacement hardware is procured, and restore the setting once three monitors are running again Delete the monitor store on the survivor so that it rebuilds the full map history from the OSDs automatically at the next start, then bring up two fresh monitors that synchronise their own stores from it Rebuild the two lost monitors with the same hostnames and addresses so that the existing monmap still matches reality; the three members then form a majority again and synchronise their stores from the survivor that holds every map

11. Q11. A monitor whose data filesystem fills up will drop out of quorum, and the monitor store tends to grow fastest precisely while the cluster is unhealthy - which makes free space on the monitor hosts a first-class production alert.

    True False

12. Q12. A network partition splits a cluster into a group of three monitors with half the OSD hosts and a group of two monitors with the other half. Applications on both sides report problems. Describe the evidence you would collect, what each side can and cannot do, and how you would recover.

    Two racks with redundant top-of-rack switches and a failed inter-rack link. Rack A holds mon.a, mon.b, mon.c and 12 OSD hosts; rack B holds mon.d, mon.e and 12 OSD hosts. All pools are size=3 with a host-level failure domain, so replicas are spread across both racks and most PGs have members on each side.

13. Q13. An OSD backed by a 16 TB HDD was created with a 30 GB block.db partition on a shared NVMe device. Over two years the RocksDB metadata for that OSD grows beyond what the 30 GB partition can hold. What happens?

    The OSD stops accepting writes as soon as RocksDB cannot allocate on the DB device, and the monitors mark it down until an operator enlarges the partition and restarts the daemon RocksDB places the levels that no longer fit on the slow block device instead, the OSD keeps working, and metadata-heavy operations on that OSD get noticeably slower BlueStore migrates the whole database back onto the slow block device and releases the NVMe partition, so the OSD performs as though it had never been given a DB device at all The manager reweights the OSD downwards through the balancer until enough PGs have moved away that the metadata left behind fits inside the DB partition again

14. Q14. BlueStore stores a checksum alongside every data extent it writes. What does that give Ceph that a backend without per-extent checksums could not offer?

    A guarantee that ceph pg repair always restores the correct bytes on a replicated pool, since the checksum proves which replica is right no matter where the corruption was introduced Freedom from deep scrubbing, since every extent is verified against its checksum as it is written, so corruption is caught at write time and cold data never needs re-reading The ability to reconstruct a corrupted extent in place from the checksum itself, so a replicated pool can repair bit rot without reading a second copy from another OSD Detection of silent corruption at read time, and the ability to identify which copy is bad rather than merely observing that the copies disagree

15. Q15. Explain what block, block.db, and block.wal are in BlueStore, and describe how you would decide whether to split block.db and block.wal onto separate devices for (a) an all-NVMe RBD pool serving VM disks and (b) an HDD-backed RGW workload with a very large number of small objects.

16. Q16. During an incident ceph -s reports a PG in the state active+undersized+degraded+remapped+backfilling. Which single reading of that state is correct?

    The PG is serving client I/O, currently holds fewer copies than the pool size, and is already copying data towards the OSDs that CRUSH now wants to hold it The PG has stopped serving client I/O because it is undersized, and reads will not complete until an operator restores enough OSDs to bring the acting set back up to the pool size The PG has permanently lost the objects that the degraded flag counts, and those will have to be restored from backup once backfill has finished placing the copies that survive The PG is remapped but blocked, waiting for an operator to unset norecover before the missing copies can be rebuilt onto the OSDs CRUSH has selected

17. Q17. During an incident you have two groups of unhealthy PGs: 240 PGs are active+degraded, and 3 PGs are stuck peering with one of them reported as down. Which do you work first, and for what reason?

    The 240 degraded PGs, because they hold far more data at reduced redundancy and a second device failure while they recover would cost you objects outright Neither, because both classes clear on their own once the down-out interval expires and the cluster starts backfilling the affected placement groups onto new OSDs The 3 inactive PGs, because degraded PGs are still serving client I/O while an inactive PG blocks every request that maps to it The 3 inactive PGs, because a degraded PG still holds the full number of copies and therefore carries no durability risk of its own while it recovers

18. Q18. Increasing pg_num on a large, busy pool is a safe and near-instantaneous operation, because it only updates pool metadata and no data moves until new objects are written.

    True False

19. Q19. A cluster has grown from 12 OSDs to 96 over two years, but its main RBD pool still carries the pg_num it was created with. Utilisation across OSDs is badly uneven and a handful of OSDs are logging slow ops. Describe the evidence you would collect, the remediation, and the rollback.

    A size=3 replicated pool named vms with pg_num 256 on a 96-OSD all-NVMe cluster serving about 300 virtual machines. The fullest OSD is at 84 per cent and the emptiest at 41 per cent. The PG autoscaler is in warn mode and has been emitting a recommendation that nobody has actioned. The cluster is at 63 per cent raw utilisation overall.

20. Q20. A pool uses an erasure code profile with k=4 and m=2 and was created with default settings. Two OSDs holding shards of the same PG fail at the same time. What happens to that PG?

    I/O continues unaffected, because m=2 means the profile tolerates two simultaneous shard losses and the four survivors are enough to reconstruct the missing pair on read Reads continue and only writes block, because a read is satisfied from any k shards while a write must update all k+m shards before it can be acknowledged to the client The PG is permanently unrecoverable and must be restored from backup, because an erasure-coded stripe cannot be decoded once more than one of its shards has been lost The PG goes inactive and blocks I/O, because the default min_size for an erasure-coded pool is k+1, which is 5 here, and only 4 shards remain

21. Q21. You plan a k=8, m=3 erasure-coded pool with a host-level failure domain for an archive workload. What is the minimum number of OSD hosts required for the pool to reach active+clean, and what is the practical minimum you would actually build?

    8 hosts minimum, because only the k data shards need distinct hosts, and 9 in practice so that the loss of one host still leaves somewhere for the parity to be recomputed 11 hosts minimum, and 12 or more in practice so that a host can fail or be taken down for maintenance and the missing shards still have somewhere to be rebuilt 3 hosts minimum, because m=3 sets the number of independent failure domains required, and CRUSH will place several shards on each host once the rule has that many buckets 11 hosts minimum, and 11 in practice, because m=3 already supplies the maintenance margin and the survivors have spare capacity to absorb a rebuild

22. Q22. Overwriting a small region of a large object on an erasure-coded pool costs more than the same overwrite on a replicated pool, because the affected stripe has to be read, re-encoded, and written back.

    True False

23. Q23. A team proposes moving the VM disk pool from size=3 replication to a k=4, m=2 erasure-coded pool in order to reclaim capacity. Give the capacity argument in their favour, at least three production reasons to refuse, and state the conditions under which you would agree.

24. Q24. A cluster is configured with a separate public network and cluster network. The cluster network switch fails while the public network stays entirely healthy. What do you expect to see?

    Nothing immediately, because the cluster network carries only recovery and backfill traffic; replication for live writes travels on the public network, so client I/O is unaffected until a device fails The monitors lose quorum immediately, because Paxos traffic between monitors runs on the cluster network, while the OSDs keep serving reads from their cached maps until it returns OSDs can no longer replicate to or heartbeat with each other, so they report one another down, the OSDMap churns, and client writes stall even though clients can still reach every OSD Ceph notices the back-end failure through its heartbeats and fails all OSD replication traffic over to the public network automatically, logging a warning until the cluster network returns

25. Q25. Jumbo frames were enabled on the Ceph hosts by setting MTU 9000 on the bonds, but one leaf switch port was left at 1500. Which symptom pattern best matches this?

    Small operations such as heartbeats, monitor traffic, and ceph -s all succeed, while large I/O involving the affected host stalls, times out, or retries intermittently The affected host drops off the network entirely, because a bond negotiating 9000 bytes against a port fixed at 1500 never links up and the interface stays down Ceph compares the MTU of every interface it binds against the rest of the cluster at daemon start, raises a health error naming the host, and refuses to bring up the OSDs on it Throughput falls evenly across every host in the cluster, because the smallest MTU on any path becomes the effective frame size for all Ceph traffic once path MTU discovery converges

26. Q26. Application owners report that a database VM on RBD now sees 40 ms write latency, up from 2 ms. The cluster is HEALTH_OK, no OSD is nearfull, and no recovery is running. Describe the evidence you would collect, how you would isolate the layer at fault, and what you would do.

    A 6-node hyper-converged Proxmox and Ceph cluster with all-NVMe OSDs and a bonded 25 GbE network carrying both public and cluster traffic. The problem appeared the morning after a scheduled switch firmware upgrade. ceph -s is clean, and rados bench from an admin host reports normal throughput with slightly elevated latency.

27. Q27. You are issuing a cephx key for a Kubernetes CSI provisioner that must create, map, and delete RBD images in the pool kube, and do nothing else. Which capability set is the right shape?

    mon allow *, osd allow *, mgr allow * - because the provisioner performs administrative operations mon profile rbd, osd profile rbd pool=kube, mgr profile rbd pool=kube mon allow r, osd allow rwx across all pools - because an RBD client must read the OSDMap for every pool in the cluster Reuse client.admin, because the CSI driver runs inside the cluster and is therefore already inside the trust boundary

28. Q28. A compliance requirement states that all storage traffic must be encrypted in transit within the datacentre. Which is the correct Ceph-native answer?

    Deploy every OSD with dm-crypt enabled, since ceph-volume then encrypts each object as it is handed to the OSD and the payload stays encrypted for the whole journey across both networks Confirm that cephx is enabled, since the session keys it negotiates already encrypt every message exchanged between clients, monitors, and OSDs and no messenger configuration is needed Move the public and cluster networks onto dedicated VLANs that no other tenant can reach, which satisfies an in-transit requirement inside the datacentre through isolation rather than cost Set the messenger v2 mode options for cluster, service, and client traffic to secure, accepting the CPU cost and the requirement that every participating client can speak msgr2 in secure mode

29. Q29. Two years ago client.admin was copied onto six administrator laptops and three CI runners. Describe how you would return the cluster to a least-privilege position, in order, and state what would break at each step if you got it wrong.

30. Q30. rbd map fails on a bare-metal host with an error reporting an RBD image feature set mismatch, while the same image attaches without trouble through librbd under QEMU on another host. What is the correct reading?

    The image has features that the kernel client on that host does not implement, so either the host needs a newer kernel or the unsupported features must be disabled on the image The cephx key is missing the profile rbd capability on the monitor, and only the kernel client enforces that at map time, which is why librbd under QEMU opens the same image without complaint The image is already open under QEMU on the other host, and the exclusive-lock feature is refusing to transfer the lock to a second client, which the kernel reports as a feature mismatch The image data lives in an erasure-coded pool with overwrites enabled, which librbd can reach through the metadata pool but which the kernel client has never supported

31. Q31. A CephFS filesystem with one active MDS and one standby serves a workload of many small files. Users report multi-second stalls listing large directories, while ceph -s stays HEALTH_OK. What is the most likely cause to check and act on first?

    The data pool, since the MDS must open every file object in the directory to return its size and modification time, so a large listing costs one read per entry from the slowest OSDs The PG count of the metadata pool, since each directory fragment is hashed to a placement group and a listing has to be gathered from every PG that the directory happens to span The MDS cache: if the memory limit is too small for the metadata working set, the MDS must fetch metadata from the metadata pool on almost every operation instead of answering from memory The client mount options, since the kernel client caps how many directory entries it will read ahead and drops back to one round trip per entry beyond a few thousand files

32. Q32. RGW multisite replication is asynchronous, so an object written in one zone can be readable there while not yet present in the other zone, and that is expected behaviour rather than a fault.

    True False

33. Q33. After adding a third RGW gateway behind the load balancer, a fraction of S3 requests began failing with 403 AccessDenied, apparently at random, while the same request succeeds on retry. Describe the evidence you would collect, the likely causes, and the remediation.

    Three radosgw daemons behind a layer 7 load balancer serving a single zone. The two original gateways were deployed 18 months ago by hand; the third was deployed this week through cephadm. Clients use SigV4 signatures with virtual-hosted-style bucket addressing over HTTPS, terminated at the load balancer.

34. Q34. A single OSD is reporting device health failures but is still up and in. The cluster is at 55 per cent raw utilisation with ample hosts. Which order of operations replaces it with the least risk and the least data movement?

    Stop the daemon so the OSD is marked down and then out when the interval expires, pull the disk, insert the replacement, and let cephadm create a fresh OSD that CRUSH fills from the surviving replicas, since a failing device was never a copy worth draining Set noout so that nothing rebuilds, pull the disk while the daemon is still running, insert the replacement into the same slot, and unset noout once the new OSD is up and in, which holds the data movement down to a single backfill onto the new device Purge the OSD from the CRUSH map first so that nothing new is placed on it, wait for the resulting backfill to finish while the failing device is still readable, then stop the daemon, swap the hardware, and create a new OSD with the next free ID Use the orchestrator OSD removal with the replace option, so the OSD is marked out and drained while its ID and CRUSH position are preserved, wait for the drain to complete and for the safe-to-destroy check to agree, then swap the device and let cephadm recreate the OSD into the same slot

35. Q35. You want recovery to run faster during a maintenance window on a Tentacle cluster left at defaults. You raise osd_max_backfills and osd_recovery_max_active, but nothing changes. Why?

    Those options are read only at daemon startup, so a runtime config set stores the values and shows them in the config dump without changing behaviour until every OSD in the cluster has been restarted in turn, which is why recovery tuning belongs in the rolling-restart plan The mClock scheduler is the default OSD operation queue and it allocates recovery and client bandwidth itself, so those legacy options are ignored unless overrides are explicitly enabled; the intended control is the mClock profile Those options were removed in a recent release and the only remaining recovery control is osd_recovery_sleep, which inserts a delay between recovery operations and has to be driven towards zero before a rebuild will go any faster Recovery throughput is bounded by the pool replication factor and by how many PGs each OSD holds, so the only way to make a rebuild finish sooner is to add OSDs or raise pg_num on the affected pool

36. Q36. Describe how you would take a single OSD host out of service for a two-hour memory upgrade on a size=3 cluster with a host-level failure domain. State which flags you would set and why each one, and what you would verify before and after.

37. Q37. A cluster has 10 hosts, each with twelve 16 TB HDD OSDs, and every pool is size=3 with a host-level failure domain. Which figure should you hand a capacity planner as space they may actually fill?

    About 1920 TB, because that is the raw figure ceph df reports and thin provisioning means the replicas of a sparse RBD image consume nothing until the guest actually writes to those blocks About 640 TB, which is the raw capacity divided by the replica count and is exactly what ceph df already reports as the maximum available space for a size=3 pool on this hardware Meaningfully less than 640 TB: after holding back a host worth of capacity for rebuild and staying under the nearfull threshold, something in the region of 480 to 500 TB is the defensible number About 213 TB, because the 640 TB of usable space is itself shared between the three replicas, so each copy may occupy no more than a third of it once the nearfull threshold is respected

38. Q38. A cluster has hit the full ratio and writes have stopped. An engineer proposes raising the full ratio from 0.95 to 0.98 to restore service while more disks are on order. What is the correct assessment?

    It is the wrong move: that reserve is what lets recovery and rebalancing still have somewhere to write, and consuming it means a subsequent OSD failure cannot be repaired - the correct emergency actions are to delete or move data, add capacity, and even out the fullest OSDs It is safe, because the full ratio only governs when the cluster raises the OSD_FULL health error; the threshold that actually stops client writes is the backfillfull ratio, which has not been reached, so raising the full ratio buys headroom without touching the mechanism that is blocking I/O It is safe provided it is reverted within a day, because the ratio is re-evaluated on every OSD statistics report, so nothing is carried forward once the value is put back, and recovery keeps priority over client writes for whatever headroom remains in the meantime It has no effect, because ceph osd set-full-ratio only changes the value the monitors use for the health check; each OSD enforces its own copy of the threshold from the OSDMap and keeps refusing writes until the daemons have been restarted, so the proposal cannot restore service

39. Q39. A rados bench result of 2 GB/s demonstrates that the cluster can comfortably serve a database VM whose workload is 4 KiB synchronous writes at queue depth 1.

    True False

40. Q40. A cluster takes hourly RBD snapshots retained for seven days, runs every pool at size=3 across three racks, and has versioning enabled on every RGW bucket. Which failure does this combination NOT protect against?

    A single OSD device failing, since the two surviving replicas keep every affected placement group serving while the missing copy is backfilled onto another OSD Loss of the cluster itself, or an operator or attacker holding cluster credentials deleting a pool along with all of its snapshots A user accidentally deleting an object from a versioned bucket, since the delete only writes a marker and the previous version can still be restored from the bucket A rack losing power, since a size=3 pool spread across three racks keeps two copies of every object available and rebuilds the third once the rack comes back

41. Q41. You back up a 4 TB RBD image nightly using incremental exports. Which sequence is correct and safe?

    Run the incremental export nightly with no snapshots at all, letting the object-map and fast-diff features compute the delta since the last successful run Snapshot nightly, export the difference between the newest snapshot and the live image head, then delete that snapshot as soon as the export file has been written so the image carries no snapshot overhead Give up on increments and export the whole 4 TB image every night, because an incremental stream can only be applied onto the exact snapshot it was taken from and that snapshot has to survive on the source Snapshot nightly, export the difference from last night snapshot to tonight snapshot, keep both snapshots until the export has been verified, and only then retire the older one

42. Q42. The primary datacentre is destroyed. You have an off-site RGW multisite secondary in a second datacentre, nightly incremental RBD exports to object storage in a third location, and deduplicating file backups of CephFS. Executives want a restoration timeline. Describe the evidence you would gather, the recovery sequence, and what you would keep as a fallback.

    A 24-host production cluster served 400 RBD virtual machine disks, a 90 TB CephFS home directory tree, and 200 TB of RGW objects. The DR site has hardware for a 12-host cluster, sized for the RBD and CephFS workloads but not for a full copy of the RGW data. The last successful DR drill was 14 months ago.

Passing score: 80%. Answers are checked in this browser.