Ceph & Distributed Storage for Production Sysadmins — Final Practical Assessment · Final assessment · Ceph

You have joined Northwind Logistics as the engineer responsible for storage. The person who built the Ceph estate left in May. You have four nodes, a cluster called nwl-prod running Ceph Tentacle, a hypervisor and a Kubernetes cluster that both depend on it, an S3 service, a backup pipeline, a Grafana dashboard, and a production workload that has been running on all of it since last November.

Nobody has told you anything is wrong. The dashboards are green, ceph -s says HEALTH_OK, and the last incident anyone connects to this cluster was months ago. The customer’s words, on your first morning, were “everything looks fine on our dashboards”.

Your job is to find out whether that is true, and to say so in a form somebody can act on.

Format

Part A — 10 auto-scored questions, in the set above this text. Machine-marked, 25% of the total. Most of them cannot be answered without having read the captures below properly.
Part B — the risk register you write yourself. 45% of the total. The marking key is in this page, under a heading that says not to read it yet.
Part C — five scenario write-ups in which you defend the most important findings on an evidence-first basis. 30% of the total. Rubrics near the end.
Open-book, four hours. You may look up anything, including the Ceph documentation. You have no shell on the estate; the captures below are what you have.
Pass: 75% overall, plus all four mandatory findings. Both bars are justified where the pass criteria are set out.

Access and constraints

You are given the estate as it sat at 09:00 this morning: the configuration files as they exist in the cluster filesystem, the CRUSH map as exported, the orchestrator service specs, the backup automation, the monitoring configuration, and the output of read-only commands run from ceph01.

You are not given a shell during the assessment, and you may not assume you can run anything else. If a finding needs a command you were not given, say so and name the command — an audit that identifies the evidence it still needs is a better audit than one that guesses.

Assume every file and every capture is current and honest. Assume the four nodes are in service and carrying production workload right now, and that any change you propose is a change to a live system. Where a question asks what a command would print or what a subsystem would do, it is asking about Ceph Tentacle 20.2.x, Proxmox VE 9.x and Kubernetes 1.31 or later.

The handover you did not get

There is no handover note. What exists is a page in the customer’s wiki, last edited in March, reproduced exactly.

Northwind storage - Ceph

Cluster:  nwl-prod, ceph01/02/03/04, Ceph Tentacle
Pools:    rbd-vm (Proxmox), k8s-data (Kubernetes), cephfs-data (CTs),
          rgw-data (S3 for the invoicing app)
Backup:   Ansible role ceph_backup, runs 01:00, writes to nfs01

Notes:

- k8s-data is size 2 for now, we were tight on space when the platform
  team onboarded. Revisit when the new disks land.
- Backups go to nfs01. Restore has not been tried.
- Grafana is at grafana.internal, dashboard "Ceph cluster overview".
- Do not delete anything in rgw-data, the invoicing app has no other copy.

That is the entire documentation of a system holding 33 TiB of production data.

The estate as you received it

Cluster, versions and daemons

Read-only / Safeceph -s— Illustrative. Read the services block line by line rather than skipping to the health line.

$ ceph -s

  cluster:
  id:     6f3a1b28-9c4d-4f7a-8b21-3d5e6f0a7c19
  health: HEALTH_OK

services:
  mon: 3 daemons, quorum ceph01,ceph02,ceph03 (age 12w)
  mgr: ceph01.hbkqrv(active, since 12w)
  mds: 1/1 daemons up, 1 standby
  osd: 24 osds: 24 up (since 12w), 24 in (since 9M)
  rgw: 2 daemons active (2 hosts, 1 zones)

data:
  volumes: 1/1 healthy
  pools:   8 pools, 889 pgs
  objects: 9.28M objects, 33 TiB
  usage:   95 TiB used, 44 TiB / 140 TiB avail
  pgs:     889 active+clean

io:
  client:   41 MiB/s rd, 118 MiB/s wr, 2.91k op/s rd, 4.22k op/s wr

Illustrative output

Read-only / Safeceph versions— Illustrative. One release across every daemon, which is what you want to see.

$ ceph versions

{
  "mon": {
      "ceph version 20.2.1 (a2f5c1d9e73b0f4a1c8d6b2e9f70a3c5d84e1b62) tentacle (stable)": 3
  },
  "mgr": {
      "ceph version 20.2.1 (a2f5c1d9e73b0f4a1c8d6b2e9f70a3c5d84e1b62) tentacle (stable)": 1
  },
  "osd": {
      "ceph version 20.2.1 (a2f5c1d9e73b0f4a1c8d6b2e9f70a3c5d84e1b62) tentacle (stable)": 24
  },
  "mds": {
      "ceph version 20.2.1 (a2f5c1d9e73b0f4a1c8d6b2e9f70a3c5d84e1b62) tentacle (stable)": 2
  },
  "rgw": {
      "ceph version 20.2.1 (a2f5c1d9e73b0f4a1c8d6b2e9f70a3c5d84e1b62) tentacle (stable)": 2
  },
  "overall": {
      "ceph version 20.2.1 (a2f5c1d9e73b0f4a1c8d6b2e9f70a3c5d84e1b62) tentacle (stable)": 32
  }
}

Illustrative output

Read-only / Safeceph mon dump— Illustrative. Note what this file is, and where the only copy of it lives.

$ ceph mon dump

epoch 3
fsid 6f3a1b28-9c4d-4f7a-8b21-3d5e6f0a7c19
last_changed 2025-11-14T09:12:41.882345+0000
created 2025-11-14T08:41:07.113920+0000
min_mon_release 20 (tentacle)
election_strategy: 1
0: [v2:198.51.100.11:3300/0,v1:198.51.100.11:6789/0] mon.ceph01
1: [v2:198.51.100.12:3300/0,v1:198.51.100.12:6789/0] mon.ceph02
2: [v2:198.51.100.13:3300/0,v1:198.51.100.13:6789/0] mon.ceph03
dumped monmap epoch 3

Illustrative output

Hosts, devices and CRUSH

Four hosts. Each has 256 GiB of memory, two NVMe devices and four SATA SSDs, all six presented as OSDs, and a pair of 10 GbE ports bonded with LACP. The hosts also run Proxmox VE 9.x and the production guests.

Read-only / Safeceph osd crush tree— Illustrative. ceph02, ceph03 and ceph04 repeat ceph01 exactly; osd.6 to osd.23 are omitted.

$ ceph osd crush tree

ID   CLASS  WEIGHT     TYPE NAME
-1         139.72552  root default
-3          34.93138      host ceph01
0   nvme    3.49309          osd.0
1   nvme    3.49309          osd.1
2    ssd    6.98630          osd.2
3    ssd    6.98630          osd.3
4    ssd    6.98630          osd.4
5    ssd    6.98630          osd.5
-5          34.93138      host ceph02
-7          34.93138      host ceph03
-9          34.93138      host ceph04

Illustrative output

Read-only / Safeceph osd crush class ls— Illustrative. Two classes exist. The question is whether anything uses them.

$ ceph osd crush class ls

[
  "nvme",
  "ssd"
]

Illustrative output

Read-only / Safeceph osd crush rule dump— Illustrative. Four rules, trimmed to the steps. Read item_name in each take step.

$ ceph osd crush rule dump

[
  {
      "rule_id": 0,
      "rule_name": "replicated_rule",
      "type": 1,
      "steps": [
          { "op": "take", "item": -1, "item_name": "default" },
          { "op": "chooseleaf_firstn", "num": 0, "type": "host" },
          { "op": "emit" }
      ]
  },
  {
      "rule_id": 1,
      "rule_name": "rbd-vm",
      "type": 1,
      "steps": [
          { "op": "take", "item": -1, "item_name": "default" },
          { "op": "chooseleaf_firstn", "num": 0, "type": "host" },
          { "op": "emit" }
      ]
  },
  {
      "rule_id": 2,
      "rule_name": "k8s-data",
      "type": 1,
      "steps": [
          { "op": "take", "item": -1, "item_name": "default" },
          { "op": "chooseleaf_firstn", "num": 0, "type": "host" },
          { "op": "emit" }
      ]
  },
  {
      "rule_id": 3,
      "rule_name": "cephfs",
      "type": 1,
      "steps": [
          { "op": "take", "item": -1, "item_name": "default" },
          { "op": "chooseleaf_firstn", "num": 0, "type": "host" },
          { "op": "emit" }
      ]
  },
  {
      "rule_id": 4,
      "rule_name": "rgw",
      "type": 1,
      "steps": [
          { "op": "take", "item": -1, "item_name": "default" },
          { "op": "chooseleaf_firstn", "num": 0, "type": "host" },
          { "op": "emit" }
      ]
  }
]

Illustrative output

The customer’s rack elevation, which you found in a spreadsheet rather than in any Ceph artefact, places ceph01 through ceph04 in rack R14, positions 12, 14, 16 and 18, on the same pair of PDUs and behind the same top-of-rack switch pair. nfs01 is at position 22 in the same rack.

Pools and placement

Read-only / Safeceph osd pool ls detail— Illustrative. Eight pools. Read size, min_size, crush_rule and autoscale_mode on every line.

$ ceph osd pool ls detail

pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 118 flags hashpspool stripe_width 0 pg_num_min 1 application mgr
pool 2 'rbd-vm' replicated size 3 min_size 2 crush_rule 1 object_hash rjenkins pg_num 512 pgp_num 512 autoscale_mode off last_change 214 lfor 0/0/198 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd
pool 3 'k8s-data' replicated size 2 min_size 1 crush_rule 2 object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode off last_change 231 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd
pool 4 'cephfs-metadata' replicated size 3 min_size 2 crush_rule 3 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 246 flags hashpspool stripe_width 0 pg_autoscale_bias 4 recovery_priority 5 application cephfs
pool 5 'cephfs-data' replicated size 3 min_size 2 crush_rule 3 object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode off last_change 246 flags hashpspool stripe_width 0 application cephfs
pool 6 '.rgw.root' replicated size 3 min_size 2 crush_rule 4 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 259 flags hashpspool stripe_width 0 application rgw
pool 7 'rgw-index' replicated size 3 min_size 2 crush_rule 4 object_hash rjenkins pg_num 16 pgp_num 16 autoscale_mode off last_change 259 flags hashpspool stripe_width 0 application rgw
pool 8 'rgw-data' replicated size 3 min_size 2 crush_rule 4 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 259 flags hashpspool stripe_width 0 application rgw

Illustrative output

Capacity

Read-only / Safeceph df— Illustrative. The two class rows are worth as much attention as the TOTAL row.

$ ceph df

--- RAW STORAGE ---
CLASS     SIZE    AVAIL     USED  RAW USED  %RAW USED
nvme    28 TiB  8.9 TiB   19 TiB    19 TiB      68.13
ssd    112 TiB   36 TiB   76 TiB    76 TiB      68.22
TOTAL  140 TiB   44 TiB   95 TiB    95 TiB      68.20

--- POOLS ---
POOL             ID  PGS   STORED  OBJECTS     USED  %USED  MAX AVAIL
.mgr              1    1   41 MiB       11  123 MiB      0    9.8 TiB
rbd-vm            2  512   24 TiB    6.29M    72 TiB  71.01    9.8 TiB
k8s-data          3  128  4.2 TiB    1.10M   8.4 TiB  21.88     15 TiB
cephfs-metadata   4   32   47 GiB    1.31M  141 GiB   0.47    9.8 TiB
cephfs-data       5  128  3.6 TiB     944k    11 TiB  26.87    9.8 TiB
.rgw.root         6    8   18 KiB       12   72 KiB      0    9.8 TiB
rgw-index         7   16   18 GiB    2.05k   54 GiB   0.18    9.8 TiB
rgw-data          8   64  1.3 TiB     901k  3.9 TiB   11.71    9.8 TiB

Illustrative output

Read-only / Safeceph osd df— Illustrative. Six of twenty-four rows, chosen to show the range. Read the %USE column against VAR.

$ ceph osd df

ID  CLASS  WEIGHT   REWEIGHT     SIZE  RAW USE     DATA     OMAP     META    AVAIL   %USE   VAR  PGS  STATUS
0   nvme  3.49309   1.00000  3.5 TiB  2.0 TiB  2.0 TiB  1.1 GiB  4.9 GiB  1.5 TiB  57.94  0.85   54      up
1   nvme  3.49309   1.00000  3.5 TiB  2.5 TiB  2.5 TiB  1.4 GiB  5.8 GiB  1.0 TiB  71.02  1.04   66      up
6   nvme  3.49309   1.00000  3.5 TiB  2.7 TiB  2.7 TiB  1.5 GiB  6.1 GiB  808 GiB  77.42  1.14   72      up
4    ssd  6.98630   1.00000  7.0 TiB  4.2 TiB  4.2 TiB  2.1 GiB  9.0 GiB  2.8 TiB  60.11  0.88  112      up
9    ssd  6.98630   1.00000  7.0 TiB  4.8 TiB  4.8 TiB  2.4 GiB  9.9 GiB  2.2 TiB  68.31  1.00  127      up
17    ssd  6.98630   1.00000  7.0 TiB  5.5 TiB  5.5 TiB  2.7 GiB   11 GiB  1.5 TiB  78.71  1.15  146      up
                     TOTAL  140 TiB   95 TiB   95 TiB   47 GiB  212 GiB   44 TiB  68.20
MIN/MAX VAR: 0.85/1.15  STDDEV: 6.34

Illustrative output

Read-only / Safeceph balancer status— Illustrative. One field decides what the spread above means.

$ ceph balancer status

{
  "active": false,
  "last_optimize_duration": "",
  "last_optimize_started": "",
  "mode": "upmap",
  "no_optimization_needed": false,
  "optimize_result": "",
  "plans": []
}

Illustrative output

Read-only / Safeceph osd dump | grep ratio— Illustrative. The three thresholds, at their defaults.

$ ceph osd dump | grep ratio

full_ratio 0.95
backfillfull_ratio 0.9
nearfull_ratio 0.85

Illustrative output

CephFS

Read-only / Safeceph fs status cephfs— Illustrative. The STANDBY MDS block is the part to read closely.

$ ceph fs status cephfs

cephfs - 26 clients
======
RANK  STATE           MDS              ACTIVITY     DNS    INOS   DIRS   CAPS
0    active  cephfs.ceph01.tqjkma  Reqs:   41 /s  1284k  1281k  48.2k   912k
    POOL           TYPE     USED  AVAIL
cephfs-metadata    metadata   141G  9.8T
cephfs-data        data      11T  9.8T
   STANDBY MDS
cephfs.ceph02.wzpnbx
MDS version: ceph version 20.2.1 (a2f5c1d9e73b0f4a1c8d6b2e9f70a3c5d84e1b62) tentacle (stable)

Illustrative output

Read-only / Safeceph fs get cephfs— Illustrative, trimmed. Read the flags line for what is present and for what is not.

$ ceph fs get cephfs

Filesystem 'cephfs' (1)
fs_name	cephfs
epoch	61
flags	12 joinable allow_snaps allow_multimds_snaps
created	2025-11-14T11:02:18.447213+0000
modified	2026-05-21T14:38:02.119640+0000
max_mds	1
in	0
up	{0=24103}
standby_count_wanted	1
data_pools	[5]
metadata_pool	4

Illustrative output

Read-only / Safeceph fs subvolume ls cephfs— Illustrative. One subvolume serves every container on the platform.

$ ceph fs subvolume ls cephfs

[
  {
      "name": "ct-shared"
  }
]

Illustrative output

There is no snapshot schedule on the filesystem, no quota on ct-shared, and no entry for CephFS anywhere in the backup automation shown below.

RGW

Read-only / Saferadosgw-admin bucket stats— Illustrative, trimmed to one representative bucket of the eleven that exist.

$ radosgw-admin bucket stats --bucket invoicing-archive

{
  "bucket": "invoicing-archive",
  "num_shards": 11,
  "tenant": "",
  "zonegroup": "default",
  "placement_rule": "default-placement",
  "owner": "invoicing",
  "ver": "0#418223",
  "mtime": "0.000000",
  "creation_time": "2025-11-28T16:04:11.229817Z",
  "usage": {
      "rgw.main": {
          "size": 1189426688000,
          "size_actual": 1189843046400,
          "num_objects": 812446
      }
  },
  "bucket_quota": {
      "enabled": false,
      "max_size": -1,
      "max_objects": -1
  }
}

Illustrative output

Read-only / Saferadosgw-admin metadata get— Illustrative. One bucket instance of eleven. No versioning flag is set on any of them.

$ radosgw-admin metadata get bucket.instance:invoicing-archive

{
  "key": "bucket.instance:invoicing-archive:5f0e2a17-8c9b-4b41-a0f5-2c7d9e13b8a4.14203.1",
  "ver": { "tag": "_qYnRk2xLpTfV8gH", "ver": 2 },
  "data": {
      "bucket_info": {
          "flags": 0,
          "num_shards": 11,
          "requester_pays": "false",
          "has_website": "false",
          "swift_versioning": "false",
          "object_lock_enabled": false
      }
  }
}

Illustrative output

No bucket on this zone has versioning enabled and no lifecycle policy exists. There is one zone, one zonegroup and one realm: radosgw-admin sync status reports no peers.

Orchestrator and service specs

Read-only / Safeceph orch ls— Illustrative. Read the RUNNING and PLACEMENT columns together.

$ ceph orch ls

NAME                       PORTS   RUNNING  REFRESHED  AGE  PLACEMENT
crash                                  4/4  6m ago     9M   *
mds.cephfs                             2/2  6m ago     9M   ceph01;ceph02
mgr                                    1/1  6m ago     9M   count:1
mon                                    3/3  6m ago     9M   ceph01;ceph02;ceph03
osd.all-available-devices               24  6m ago     9M   *
rgw.default                ?:8080      2/2  6m ago     9M   ceph01;ceph02

Illustrative output

The service specs, as ceph orch ls --export would print them. This output has never been captured to a file, a repository or a backup; the copy below was produced for the assessment.

service_type: mon
service_name: mon
placement:
  hosts:
  - ceph01
  - ceph02
  - ceph03
---
service_type: mgr
service_name: mgr
placement:
  count: 1
---
service_type: osd
service_id: all-available-devices
service_name: osd.all-available-devices
placement:
  host_pattern: '*'
spec:
  data_devices:
    all: true
  filter_logic: AND
  objectstore: bluestore
---
service_type: mds
service_id: cephfs
service_name: mds.cephfs
placement:
  hosts:
  - ceph01
  - ceph02
---
service_type: rgw
service_id: default
service_name: rgw.default
placement:
  hosts:
  - ceph01
  - ceph02
spec:
  rgw_frontend_port: 8080
  rgw_realm: default
  rgw_zone: default

Configuration database

Read-only / Safeceph config dump— Illustrative. Seven lines. Two of them are the estate's network design and two are decisions somebody made once.

$ ceph config dump

WHO     MASK  LEVEL     OPTION                   VALUE                                              RO
global        basic     container_image          quay.io/ceph/ceph@sha256:0f9c4a1e7b3d... (20.2.1)  *
global        advanced  public_network           198.51.100.0/24                                    *
global        advanced  cluster_network          198.51.100.0/24                                    *
global        advanced  mon_allow_pool_delete    true
global        advanced  osd_deep_scrub_interval  5184000
osd           advanced  osd_scrub_begin_hour     1
osd           advanced  osd_scrub_end_hour       6

Illustrative output

Read-only / Safeceph config get— Illustrative. Four options that are absent from the dump above, and therefore at their defaults.

$ ceph config get osd osd_memory_target

4294967296

$ ceph config get global ms_cluster_mode
crc secure

$ ceph config get global ms_client_mode
crc secure

$ ceph config get osd osd_mclock_profile
balanced

$ ceph config get osd osd_mclock_override_recovery_settings
false

Illustrative output

Identity and client keys

Read-only / Safeceph auth ls— Illustrative, trimmed to the client entities. Keys redacted; capabilities are not.

$ ceph auth ls

client.admin
key: AQD...redacted...
caps: [mds] allow *
caps: [mgr] allow *
caps: [mon] allow *
caps: [osd] allow *
client.backup
key: AQD...redacted...
caps: [mon] allow *
caps: [osd] allow *
client.k8s
key: AQD...redacted...
caps: [mon] allow *
caps: [osd] allow *
client.bootstrap-osd
key: AQD...redacted...
caps: [mon] allow profile bootstrap-osd

Illustrative output

There is no client.pve, no client.cephfs, and no client.rgw-app. The keys above were created on 2025-11-14 and none has been rotated since. The engineer who left in May created and held all of them.

The consumers

Proxmox VE, on all four nodes, /etc/pve/storage.cfg:

rbd: ceph-vm
	content images,rootdir
	krbd 0
	monhost 198.51.100.11 198.51.100.12 198.51.100.13
	pool rbd-vm
	username admin

The matching keyring is /etc/pve/priv/ceph/ceph-vm.keyring, which is replicated to every node by the Proxmox cluster filesystem.

Kubernetes, the ceph-csi secret in namespace ceph-csi:

apiVersion: v1
kind: Secret
metadata:
  name: csi-rbd-secret
  namespace: ceph-csi
stringData:
  userID: k8s
  userKey: AQD...redacted...

The csi-rbd StorageClass points at pool k8s-data, and 47 PersistentVolumes are bound through it. The invoicing application reaches S3 at http://ceph01:8080 — a single hostname, configured in the application’s environment, on both of its replicas.

The backup pipeline

The Ansible role ceph_backup, run by a systemd timer at 01:00 daily from the management host. Reproduced exactly.

- name: Prune backups older than a week
  ansible.builtin.command:
    cmd: find /mnt/backup -name '*.img' -mtime +7 -delete
  changed_when: true

- name: Export RBD images to the NFS share
  ansible.builtin.command:
    cmd: >-
      rbd export rbd-vm/{{ item }}
      /mnt/backup/{{ item }}-{{ ansible_date_time.date }}.img
  loop: "{{ rbd_images }}"
  changed_when: true

rbd_images is a static list in group_vars/all.yml, last edited 2025-12-09, holding 33 image names.

Read-only / Saferbd ls rbd-vm— Illustrative. Count the lines against the length of the static list above.

$ rbd ls rbd-vm | wc -l

Illustrative output

/mnt/backup is an NFS mount of nfs01.internal:/export/ceph-backup. nfs01 is the host at position 22 of rack R14. There is no restore playbook in the repository, no record of a restore ever having been performed, and no stated RTO or RPO anywhere in the customer’s documentation.

Monitoring

prometheus.yml, reproduced exactly:

global:
  scrape_interval: 30s

scrape_configs:
  - job_name: 'ceph'
    honor_labels: true
    static_configs:
      - targets: ['198.51.100.11:9283']

Prometheus and Grafana run in VMs 210 and 211. Both VMs have their disks in pool rbd-vm, on nodes ceph02 and ceph03. Grafana holds two dashboards: a cluster overview and a per-pool latency panel. There is no Alertmanager deployed, no rule_files stanza, and no alert has ever been delivered to a human from this estate.

What you must produce

Six deliverables. They are marked together, and an answer that produces findings without severities, or remediation without validation, is incomplete rather than partially correct.

Findings. A risk register. One row per finding: where it is, what it is, and what it can do. Findings that only exist in combination get their own row and name the components they need.
Evidence. For each finding, the specific artefact or the read-only command that establishes it. “The pool looks wrong” is not evidence. ceph osd pool ls detail next to ceph osd crush rule dump is.
Severity. A rating per finding, from the scale below, with one sentence of reasoning. The reasoning is what is marked; the label on its own is not.
Remediation. What you change, and in what order. The order is marked, and it is marked hardest. A plan that starts with the most interesting defect has chosen wrong.
Estimated impact. For every change that moves data, the volume of data it moves and the effect on capacity, in the cluster’s own units. “Backfill will take a while” is not an estimate.
Verification. For each remediation, the read-only command whose output demonstrates it worked. Not “the pool was updated” — the pool detail line, the recalculated post-failure capacity, the restored image mounted and read.

The severity scale

Use this scale, and use all of it.

Severity	Meaning
Critical	Can cause data loss, credential disclosure, or an estate-wide outage. Blocks the next production change.
High	Can cause an outage of a subset of workload, a silent loss of protection, or a serious loss of assurance. Fix before the next maintenance window closes.
Medium	Degrades reliability, capacity or diagnosis. Scheduled work with an owner.
Low	Hygiene and maintainability. Backlog.
None	Looks like a finding and is not. Record it as examined so nobody re-raises it.

How the marks are allocated

The total is 100. Nothing else carries marks.

Part	What is marked	Marks
A	Ten auto-scored questions, 2.5 each	25
B	The risk register	45
C	Five scenario write-ups, 6 each	30
	Total	100

Part B is broken down as follows.

Ref	Component	Marks
B1	Findings and evidence — half a mark for each register row that names the finding and the artefact establishing it, over the 36 rows in the key	18
B2	Severity discipline — bands below	9
B3	Interactions — one mark for each of the six named with its component findings	6
B4	Remediation sequence and estimated impact	7
B5	Verification plan	5
	Part B total	45

B2, severity discipline (9 marks). Marked in bands; award the highest band whose description your register satisfies, then adjust by one within the band.

Band	Description
8–9	Every row carries a severity and one sentence naming the harm. The four mandatory findings are Critical. At least three rows are Low or None with reasoning. No more than nine rows are Critical.
6–7	Severities are within one band of the key almost everywhere, and the reasoning names a harm rather than restating the finding.
3–5	Severities are present, but the reasoning restates the finding, or more than half the register is Critical or High.
0–2	No severities, or every row rated the same, or the register contains no row below High.

B4, remediation sequence and estimated impact (7 marks).

Marks	Awarded for
2	The first action is `min_size 2` on `k8s-data`, with a reason that names what it costs — nothing, no data movement, and the single-copy acknowledged write stops immediately
2	The move from `size 2` to `size 3` is costed in raw TiB and sequenced against the capacity finding, with the arithmetic shown
2	The change window is expressed as client latency under the default mClock scheduler, naming `osd_mclock_profile` and the fact that `osd_max_backfills` is ignored unless `osd_mclock_override_recovery_settings` is true
1	The remaining plan is ordered and its dependencies hold: backup relocation before CRUSH restructuring, capacity before any change that adds copies

B5, verification plan (5 marks).

Marks	Awarded for
2	Every remediation names a read-only command whose output demonstrates the change, rather than an assertion that it was made
2	The backup finding is verified by an actual restore into a scratch pool, mapped and read by something, with the elapsed time recorded and compared against a stated RTO
1	The capacity finding is re-verified by re-running the post-failure arithmetic against `ceph osd df tree`, not by “we added disks”

Marking key: the risk register

Thirty-six entries, in two tables. The thirty-two below are the substantive findings; the four after them are judgement calls a complete register still has to make.

Score half a mark for each row you produced independently, with its evidence named. Rows you produced only after reading this key score nothing.

#	Where	Finding	Severity
F1	`ceph osd pool ls detail`	`k8s-data` is `size 2 min_size 1`: a write is acknowledged while exactly one copy exists, and 47 PersistentVolumes depend on it	Critical
F2	`ceph df` + `ceph osd crush tree`	Four hosts, `size 3`, failure domain `host`: after one host loss each survivor must hold a full copy — 31.8 TiB against 34.9 TiB of raw, past `backfillfull_ratio` before the rebuild finishes	Critical
F3	`ceph balancer status` + `ceph osd df`	The balancer is off. OSD utilisation spans 57.94% to 78.71% at a 68.20% mean, so the fullest OSD reaches `full_ratio` while the cluster average is still ten points below `backfillfull_ratio`	High
F4	`ceph osd crush rule dump` + `ceph osd crush class ls`	Two device classes exist and no rule selects one. Every `take` step is `item_name: default`, so NVMe and SATA carry PGs in proportion to weight and both classes sit at 68% — the NVMe buys nothing it was bought for	High
F5	`ceph osd pool ls detail`	`cephfs-metadata` shares `crush_rule 3` with `cephfs-data`, so the small latency-critical metadata pool is placed indifferently across SATA alongside bulk file data	Medium
F6	`ceph osd crush tree` + rack elevation	The failure domain is `host` and there is no chassis or rack bucket. All four hosts are in rack R14 on one PDU pair behind one ToR pair, so a rack event takes every copy of every PG	High
F7	`ceph -s` + `ceph orch ls`	One manager, `placement: count: 1`, with no standby. It carries the orchestrator, the dashboard, the balancer module and the Prometheus exporter, so one host loss takes all of them at once	High
F8	`ceph mon dump`	No monmap has been exported and no copy of any monitor store exists off the cluster, so a two-monitor loss has no documented recovery path	High
F9	`ceph fs get cephfs`	`allow_standby_replay` is absent from the flags. The standby is cold and must read the journal from RADOS and rebuild its cache before serving, blocking clients for tens of seconds on failover; the fix is `ceph fs set cephfs allow_standby_replay true`	High
F10	`ceph orch ls`	Both MDS daemons, both RGW daemons, two of three monitors and twelve OSDs are on `ceph01` and `ceph02`. Losing either host removes more service classes at once than the placement suggests	Medium
F11	`ceph fs subvolume ls`	One subvolume serves every container, with no quota and no snapshot schedule, so one runaway writer fills `cephfs-data` for everybody	Medium
F12	`rgw` service spec	The RGW frontend is plain HTTP on port 8080 with no certificate configured, so S3 credentials and object payloads cross the network in clear	High
F13	Application environment	Two RGW daemons exist and every client is configured with the single endpoint `http://ceph01:8080`, so one host reboot takes S3 down despite the redundancy that was paid for	High
F14	`radosgw-admin metadata get`	No bucket has versioning enabled and no lifecycle policy exists, so a client-side delete on `invoicing-archive` is unrecoverable	High
F15	`ceph_backup` role	The pipeline copies RBD images from one pool. CephFS, RGW, the monitor store, the CRUSH map, the service specs and the cephx keys have no copy anywhere	Critical
F16	`/mnt/backup` mount	The NFS target is `nfs01` at position 22 of rack R14 — the same rack, PDU pair and switch pair as the cluster it protects	Critical
F17	Ansible repository	No restore playbook exists and no restore has ever been performed, so no claim about recovery has been tested and RTO is unmeasured	Critical
F18	`ceph_backup` role	The prune task runs before the export task. A failing export therefore deletes an old copy and does not create a new one, and the estate silently loses coverage from the older end	High
F19	`group_vars/all.yml` + `rbd ls rbd-vm`	`rbd_images` is a static list of 33 names last edited 2025-12-09; the pool holds 41 images. Eight images have never been copied anywhere	Critical
F20	`ceph_backup` role	`rbd export` takes a crash-consistent full copy with no guest quiesce, so database guests restore to the equivalent of a power cut and the copy grows with the image rather than with the change	High
F21	Customer documentation	The 24-hour cadence has never been agreed with the business and no RPO or RTO is written down, so nobody has ever compared what the pipeline delivers against what is expected of it	Medium
F22	`ceph auth ls` + `/etc/pve/storage.cfg`	Proxmox authenticates as `client.admin`, and the admin keyring is replicated to all four nodes by the cluster filesystem. Every hypervisor node holds a key that can delete every pool	Critical
F23	`ceph auth ls`	`client.k8s` and `client.backup` hold `mon allow ` and `osd allow ` instead of `profile rbd` scoped to their pool, so the CSI driver and the backup job each have cluster-wide authority	High
F24	`ceph auth ls`	No key has been rotated since 2025-11-14. The engineer who left in May created and held all of them and they remain valid	High
F25	`ceph config dump`	`public_network` and `cluster_network` are the same `/24` on the same LACP bond, so replication, client I/O and hypervisor traffic share one bandwidth budget and one failure	High
F26	`ceph config get global ms_cluster_mode`	`crc secure` is a preference order, not a guarantee: the cluster prefers CRC, so replication traffic is integrity-checked and not encrypted, and no document says otherwise	Medium
F27	`ceph config dump`	`mon_allow_pool_delete true` has been set since deployment, so the safety interlock that stops an accidental `ceph osd pool rm` is disarmed permanently	Medium
F28	`ceph config dump`	`osd_deep_scrub_interval` is 5184000, sixty days, against a seven-day default. Latent corruption on a SATA OSD can sit unverified for two months, and the overdue-scrub health check derives its threshold from this same interval, so raising it also raised the alarm	High
F29	`ceph config get osd osd_memory_target`	At the 4 GiB default. Six OSDs per host reserve roughly 24 GiB that appears in no Proxmox capacity plan, on hosts where VM allocation and OSD demand peak together	Medium
F30	`prometheus.yml`	A single static scrape target — the address of the only manager — with no `rule_files` and no `alerting` stanza. The estate has visualisation and no alerting at all	High
F31	Guest placement	Prometheus and Grafana are VMs with their disks in `rbd-vm`, on the cluster they monitor. When Ceph blocks client I/O the monitoring blocks with it	High
F32	Customer wiki + Ansible repository	No documented, rehearsed recovery procedure, no handover, and `ceph orch ls --export` has never been captured to version control. The estate exists in exactly one place	High

And four items that must appear in the register with the right verdict.

#	Where	Item, and the verdict a complete register reaches
J1	`rgw-data` size 3	Replication rather than erasure coding on the S3 pool. None — with four hosts and `failure_domain host`, `k=3 m=1` leaves no spare host to rebuild onto after a host loss and `k=2 m=2` costs 2.0x raw for a 33% saving over replication, paid for in read-modify-write on hosts that also run the hypervisor. Replication is the correct call and belongs in the register as examined
J2	cephadm	Deployment by cephadm rather than a configuration-management tool. None — cephadm is the supported path and needs no justification. The real defect is that its output is never exported, which is F32
J3	`autoscale_mode off`	The autoscaler is off on all eight pools. Low — 2,539 PG replicas across 24 OSDs is 106 per OSD, inside the healthy range today. It is drift risk rather than a present defect, and it becomes a High the first time a pool grows without anyone recalculating
J4	`ceph fs status`	One standby MDS rather than two. None — one standby satisfies `standby_count_wanted 1` and is a supported configuration. The defect is that it is not `standby-replay`, which is F9

Why the four judgement rows are in the key

Four things in this estate look like findings to a fast reader and are not, and all four are worth recording as examined rather than omitting.

“Replication where erasure coding would be cheaper” is a real anti-pattern and this is not an instance of it. The host count decides it: k+m must fit inside the failure domain, and on four hosts the only profiles that fit leave the pool unable to heal a host loss or cost almost as much as replication for a workload that also runs virtual machines on the same CPUs. A candidate who raises EC here has read a table and not the topology.

“The autoscaler is off” is the same shape. It sounds like negligence until you do the arithmetic, and the arithmetic says the pools are correctly sized for the cluster they are on today. Recording it Low, with the number, is what separates an auditor from an alarm.

An auditor who records all four as examined, with a sentence each, saves the next person a morning and demonstrates the judgement the other thirty-two rows are being marked on.

The interactions

Findings in isolation understate this estate. Six combinations matter more than their parts, and the register is expected to name them. One mark each.

The host you cannot afford to lose (F2 + F3 + F1 + F6). Take any one of the four hosts away and CRUSH must place three replicas across three survivors, which means each survivor holds one full copy of every size-3 pool. That is 31.8 TiB against 34.9 TiB of raw per host. The average OSD lands at 90.9%, past backfillfull_ratio, so the rebuild stalls with PGs in backfill_toofull — and because the balancer is off, the fullest OSDs are already ten points above the mean and cross full_ratio first, at which point writes stop for every pool with a PG on them. Those pools back every virtual machine on the same four hosts. k8s-data is the one pool that keeps writing through it, onto a single copy, because min_size is 1. And the whole calculation assumes the failure is one host: with every host in one rack on one PDU pair, the more likely large failure takes all four.

The backup that is not one (F15 + F16 + F17 + F18 + F19 + F20). Six independent mechanisms in series. The copy lives in the rack it is protecting against. It covers one pool of eight, so the S3 archive the invoicing application has no other copy of is not in it, nor is CephFS, nor is the cluster’s own configuration. Eight images in the pool it does cover are not in the static list. The prune runs before the export, so a failure shortens coverage rather than leaving it unchanged. The copies are crash-consistent with no quiesce. And no restore has ever been performed, so none of this was discovered. Any one of these would be a finding; in series they mean the loss of this cluster is the loss of the company’s data. This is the interaction that decides the production recommendation.

One credential, one command (F22 + F23 + F24 + F27 + F26). client.admin is on every hypervisor node, client.k8s and client.backup hold osd allow *, mon_allow_pool_delete is true, and no key has been rotated since the cluster was built by an engineer who left in May. Any one of those keys can remove every pool in one command, and the interlock that exists to prevent exactly that is switched off. The traffic those keys authenticate on is integrity-checked and not encrypted, on the same subnet as the guests. There is no second copy of the data anywhere — see the interaction above.

Failure without a signal (F30 + F31 + F7 + F28). The exporter lives inside the only manager. The single static scrape target is that manager’s address, so a manager failover — if there were anywhere to fail over to — would stop the metrics. There are no alert rules and no Alertmanager, so nothing evaluates the metrics that do arrive. The Prometheus that would evaluate them is a virtual machine whose disk is in rbd-vm, so when Ceph blocks, the monitoring blocks with it. And deep scrub, the mechanism that finds corruption nobody has asked about, runs on a sixty-day interval that also moved the health check meant to warn that it is late. Every one of these converts a signal into silence, and they are arranged so that the first notification anybody receives about this cluster will be a person telephoning.

The metadata path (F5 + F9 + F10 + F4). CephFS metadata is small, latency-critical and served by a single active MDS. Its pool shares a CRUSH rule with bulk file data, and that rule selects no device class, so metadata lands on SATA next to whatever else is there. The MDS serving it runs on ceph01, alongside a monitor, an RGW, six OSDs and the hypervisor workload, with a cold standby on ceph02 that also runs a monitor, an RGW and six OSDs. There is nothing wrong with any of those daemons individually. Together they mean the metadata path has no fast device, no warm failover and no host that is not already busy.

The estate nobody can rebuild (F32 + F8 + F15 + F24). The CRUSH map, the service specs, the pool definitions, the cephx keys and the monitor store exist in exactly one place, which is the cluster. There is no monmap export, so the documented path out of a two-monitor loss cannot be followed. There is no runbook, so nothing tells the next engineer what the estate is meant to look like. There is no handover, so the only person who knew left in May. The backup does not contain the configuration, so even a successful restore of every RBD image would leave someone rebuilding a cluster from a rack elevation and a wiki page.

Working the arithmetic

Three findings in this estate are arithmetic rather than judgement, and the register is expected to show the working rather than assert the conclusion.

Read-only / Safeestablish the post-failure position before believing anything about it— READ-ONLY: four commands that between them give raw capacity, per-host distribution, the stored figure and the pool replication the arithmetic needs.

ceph df
ceph osd df tree
ceph osd crush tree
ceph osd pool ls detail

Capacity, after losing one host. Do not start from %RAW USED. Start from what each surviving host must hold.

With failure_domain host and size 3 on four hosts, every placement group has a copy on three of the four. Lose one host and the three survivors each hold one copy of everything:

size-3 stored total = 24.0 (rbd-vm)
                    +  3.6 (cephfs-data)
                    +  1.3 (rgw-data)
                    + 0.047 (cephfs-metadata)
                    + 0.018 (rgw-index, .rgw.root, .mgr)
                    = 28.97 TiB per surviving host

k8s-data is size 2 on three hosts = 4.2 x 2/3 = 2.80 TiB per host

per-host requirement after one host loss = 31.77 TiB
per-host raw                             = 34.93 TiB
                                         = 90.9%

90.9% is past backfillfull_ratio 0.90, so the rebuild does not complete: it stops with placement groups in active+undersized+degraded+remapped+backfill_toofull. And because the balancer is off, the individual OSDs are not at the mean. The fullest today is at 78.71% against a 68.20% mean, so it arrives at full_ratio 0.95 while the cluster average is still below backfillfull. When an OSD reaches full_ratio, writes stop for every pool with a placement group on it — and those pools carry every guest on the same four hosts.

Check the same arithmetic against today’s figures to confirm it is the right model: 28.97 x 3/4 plus 4.2 x 2/4 is 23.82 TiB per host, which is 68.2% of 34.93 TiB, and ceph df reports %RAW USED 68.20. The model agrees with the capture, so the projection can be trusted.

What the fix costs. Raising k8s-data from size 2 to size 3 creates one more copy of 4.2 TiB:

raw used today          = 95.30 TiB  (68.20% of 139.73 TiB)
raw used after size 3   = 99.50 TiB  (71.21%)

per-host requirement after one host loss, with k8s-data at size 3
  = 28.97 + 4.20 = 33.17 TiB of 34.93 TiB = 94.9%

94.9% is full_ratio to within a rounding error, which means that after the change a host loss does not merely stall the rebuild — it arrives at the threshold where the cluster stops accepting writes. The change is safe today and it makes an already-unsurvivable host loss worse. That is not a reason to defer it — an acknowledged single-copy write is a worse exposure than a rebuild that was going to stall anyway — but the register has to say so rather than pretend the two findings are independent.

What capacity actually resolves it. Two options, both costed against the same post-failure test, with k8s-data at size 3:

Option	Post-loss per host	Result
Add one 7.68 TB SATA SSD to each of the four hosts (raw per host 41.92 TiB)	33.17 of 41.92 = 79.1%	Under `nearfull_ratio`, but with six points of margin and still four hosts in one rack
Add a fifth identical host (four survivors, so each holds three quarters of the stored total)	24.87 of 34.93 = 71.2%	Under `nearfull_ratio` with room, and it adds a member to the failure domain

The fifth host is the better buy and the honest recommendation. The four drives are the cheaper stopgap. Either is a purchase order; “we need more disk” is not.

How long the rebuild would take, if it could finish. Each survivor must gain 31.77 minus 23.82, so 7.95 TiB, and the cluster must write 23.82 TiB in total.

23.82 TiB = 24,396 GiB

at 1 GiB/s aggregate   = 24,396 s = 6.8 hours
at 300 MiB/s aggregate = 83,271 s = 23.1 hours

Neither number is a prediction. They are the two ends of a range that depends entirely on what the recovery is allowed to take from a 2x10 GbE bond that is also carrying every client read and write, and the point of writing them down is that the operator must measure the achieved rate during the event rather than assume one. ceph -s reports the recovery line; ceph osd pool stats reports it per pool.

The change window, correctly. Under the default mClock scheduler the classic recovery throttles do not apply. osd_mclock_profile is balanced and osd_mclock_override_recovery_settings is false, which means osd_max_backfills and osd_recovery_max_active are accepted by ceph config set, visible in ceph config get, and ignored by the OSD.

Configuration changethe lever that works on this cluster— CONFIGURATION: the profile takes effect live through the configuration database, with no daemon restart. Return it to balanced before client load resumes.

ceph config get osd osd_mclock_profile
ceph config set osd osd_mclock_profile high_client_ops
ceph config set osd osd_mclock_profile balanced

A candidate who plans the size 3 change around osd_max_backfills has planned around a setting this cluster will not read. On a hyperconverged estate the profile choice is the whole of the change window: high_client_ops protects the guests and lengthens the exposure, high_recovery_ops shortens the exposure and is felt by every VM on the same hosts, and the decision belongs to whoever owns the workload, not to whoever is running the command.

Part C: the scenarios

Five write-ups, six marks each, thirty in total. Each is a short piece of prose — three or four paragraphs — in which you defend a position with evidence from the captures rather than with a rule of thumb. Items marked REQUIRED are pass or fail on their own: a scenario answer that misses its required element scores zero for that scenario regardless of the other marks.

C1 — The pool that acknowledges a write it cannot keep

The platform team asks why you want a change window for k8s-data when the cluster is HEALTH_OK and their 47 volumes are working. Answer them.

Marks	Awarded for
2	REQUIRED — the mechanism, not the label: at `min_size 1` the pool accepts and acknowledges a write while exactly one copy of it exists, so the acknowledgement is a promise the cluster cannot keep if that OSD fails before recovery re-replicates. “size 2 is risky” is not this answer
2	The availability cost of `min_size 2` on a `size 2` pool is stated honestly — a single OSD failure blocks I/O to the affected PGs until the second copy exists — and the answer explains why that is still the right trade against silent loss
2	The move to `size 3` is costed at 4.2 TiB of new raw and placed against the capacity finding, rather than proposed as if it were free

C2 — The host you cannot afford to lose

The customer wants to know whether they can take ceph03 out for a firmware update next Tuesday. Give them an answer with a number in it.

Marks	Awarded for
2	REQUIRED — the post-failure figure is derived from stored data and pool size, not read off `%RAW USED`. The answer states 31.8 TiB required per survivor against 34.9 TiB available, and shows how it got there
2	`nearfull_ratio`, `backfillfull_ratio` and `full_ratio` are named with what each one does when crossed, and the answer distinguishes a rebuild that stalls from writes that stop
2	At least one costed remedy with the resulting post-loss percentage, and a distinction between the maintenance case — where `noout` and a short window avoid the rebuild entirely — and the failure case, where nobody chose the timing

C3 — The backup that is not one

The finance director signs off the storage budget and believes the estate is backed up nightly. Write the paragraph that changes their mind, and the one that tells them what to do about it.

Marks	Awarded for
2	REQUIRED — the scope gap is identified, not only the location: the pipeline covers one pool of eight, so the S3 archive that the invoicing application has no other copy of, the CephFS share, and the cluster’s own configuration are not in any backup at all
2	The prune-before-export ordering and the static 33-of-41 image list are both named, with what each one does on a bad night
2	The remedy is a procedure with a measured number in it: restore one image into a scratch pool, map it, read it, record the elapsed time, and compare that against the RTO the business believes it has

C4 — 02:40, and `ceph03` stops answering

You are paged. ceph03 is unreachable, six OSDs are down, placement groups are degraded and the guests that were running on it have restarted elsewhere. You have ten minutes before the cluster makes a decision for you. Say what you do and why.

Marks	Awarded for
2	REQUIRED — the answer acts before `mon_osd_down_out_interval` expires, and explains why an automatic `out` is the wrong outcome on this estate: the rebuild it starts cannot complete, and it will end with OSDs at `full_ratio` and writes blocked for every guest. `ceph osd set noout` buys the time to decide deliberately
2	The exposure while `noout` holds is stated correctly per pool: the `size 3` pools are at two copies and still above `min_size`, and `k8s-data` is at one copy and still accepting writes, which is the pool to watch
2	The mClock lever is named correctly for when the rebuild is allowed to run, with the knowledge that `osd_max_backfills` is ignored at the default profile, and the answer says what is watched to decide — client latency, `ceph -s` recovery rate, and the fullest OSD in `ceph osd df`

C5 — The production recommendation

May this estate keep carrying the workload? Write what you would put in front of the customer.

Marks	Awarded for
2	REQUIRED — a blocking set small enough to be credible, naming which findings must be cleared before the next production change rather than listing everything that is wrong
2	Interim controls that hold until the blocking set is cleared, and that somebody can actually operate: what is not changed, what is watched, what is escalated, and by whom
2	What is said to a customer whose dashboards are green — an explanation of why `HEALTH_OK` and these findings are both true at once, without either alarming them into a bad decision or reassuring them into no decision

Pass criteria

75% overall: Part A auto-scored at 25%, the risk register at 45%, the scenarios at 30%.
All four mandatory findings present in Part B, each with a defensible severity and its evidence named:
- F1, k8s-data at size 2 min_size 1, identified as an acknowledged single-copy write rather than as a general comment about redundancy.
- F2, the post-host-loss capacity position, with the arithmetic shown. A statement that the cluster is “getting full” does not satisfy this.
- F15 with F16 and F17, the backup, identified as one finding about a pipeline that is neither off-site, nor complete, nor ever tested — not as three unrelated remarks.
- F19, the eight RBD images that have never been copied anywhere.
At least four of the six interactions named, with their component findings, because a register of isolated items misrepresents this estate even when every item in it is correct.
At least three findings rated Low or None with reasoning. An audit that cannot say what is not urgent has not made a judgement.
Four of the five scenarios passed against their rubrics, with no REQUIRED element missed in any scenario answer.

Why the bar is 75 and not 70

The rest of this course passes coursework at 70. This assessment is set higher deliberately, for three reasons.

The first is the marking scheme. Part B awards marks for findings a careless reader still stumbles into. size 2 min_size 1 on a production pool is not hard to see. Neither is a backup written to a box in the same rack, nor an allow * capability on a CSI driver. A candidate who skims the captures, reports the ten most visible items and writes nothing about severity, sequencing or interaction can assemble something close to 70% out of the obvious half of the register. The bar has to sit above that, or the result certifies the wrong thing.

The second is that the interactions carry the estate. Every one of the six is composed of findings that are individually survivable and jointly are not. An engineer who lists thirty-two items and never connects four of them has produced a document that will be triaged into a backlog and worked in the wrong order — which is materially worse than a shorter register that names the capacity machine first.

The third is what the credential claims. This estate holds a company’s operational data on a four-node cluster in one rack, backed up into the same rack, with the S3 archive not backed up at all, alerting to nobody. The question this assessment answers is not “does this person know Ceph” — it is “would I let this person sign off the storage platform”. That is a higher bar in real life and it should be a higher bar here.

The mandatory findings exist for the same reason. A candidate could reach 75% on breadth while missing that eight virtual machine images have had no copy since December, and breadth does not compensate for that, so the scoring does not let it.

Why this matters

Every estate you inherit will look like this one. Not identical, but the same shape: a cluster that works, built under time pressure by someone competent who meant to come back to it, with green dashboards, a wiki page that is confident about the parts nobody checked, and HEALTH_OK on the line everybody reads.

Notice how many of these defects were once reasonable. size 2 on k8s-data bought the capacity the platform team needed on the day they onboarded, and the wiki says so. A sixty-day deep scrub interval was somebody protecting client latency on hosts that were already busy. One manager was the right number on a cluster that had one workload. A static list of images was accurate in December. client.admin in storage.cfg is what you type when you are bringing a hypervisor up for the first time and mean to fix it later. Each was a sensible decision with a half-life, and every one of them outlived the situation that justified it. That is what an inherited estate is: a stack of expired compromises, none of them documented as temporary.

Notice too what HEALTH_OK was ever able to tell you. Every health check in Ceph was written to catch a fault — a device dying, a monitor out of quorum, a placement group that cannot peer, a threshold crossed. Not one of them was written to catch a design decision, and every finding in this estate is a design decision. min_size 1 is a legal setting. A CRUSH map without rack buckets is a legal map. Ceph does not know the backup exists, so it cannot have an opinion about which rack it is in. The distinction between “nothing is firing” and “nothing is wrong” is the whole of this assessment, and it is the distinction that will decide how your first month somewhere new actually goes.

You will almost never be handed a labelled list of what is wrong. You will be handed the output and a deadline, and asked whether it is safe. The skill this measures is the one that answers that question: read what is actually there, do the arithmetic instead of accepting the summary, separate the four findings that could lose the company its data from the twenty-eight that are untidy, notice which ones are only dangerous together, and then say clearly what must be fixed before the next change and what can wait until there is budget.

The engineer who lists every defect and rates them all Critical has not done that job. Neither has the one who says the estate is basically fine, because the dashboards agree with them. The judgement in between is the whole of the work, and it is why the interactions section carries as many marks as six rows of the register that feeds it.

Ceph & Distributed Storage for Production Sysadmins — Final Practical Assessment

Objectives

Assessment questions

Format

Access and constraints

The handover you did not get

The estate as you received it

Cluster, versions and daemons

Hosts, devices and CRUSH

Pools and placement

Capacity

CephFS

RGW

Orchestrator and service specs

Configuration database

Identity and client keys

The consumers

The backup pipeline

Monitoring

What you must produce

The severity scale

How the marks are allocated

Marking key: the risk register

The interactions

Working the arithmetic

Part C: the scenarios

C1 — The pool that acknowledges a write it cannot keep

C2 — The host you cannot afford to lose

C3 — The backup that is not one

C4 — 02:40, and `ceph03` stops answering

C5 — The production recommendation

Pass criteria

Why the bar is 75 and not 70

Assessment questions

Format

Access and constraints

The handover you did not get

The estate as you received it

Cluster, versions and daemons

Hosts, devices and CRUSH

Pools and placement

Capacity

CephFS

RGW

Orchestrator and service specs

Configuration database

Identity and client keys

The consumers

The backup pipeline

Monitoring

What you must produce

The severity scale

How the marks are allocated

Marking key: the risk register

The interactions

Working the arithmetic

Part C: the scenarios

C1 — The pool that acknowledges a write it cannot keep

C2 — The host you cannot afford to lose

C3 — The backup that is not one

C4 — 02:40, and ceph03 stops answering

C5 — The production recommendation

Pass criteria

Why the bar is 75 and not 70

C4 — 02:40, and `ceph03` stops answering