Proxmox final assessment - production readiness of an inherited estate

1. Q1. Read the pool definition and the CRUSH rule together: pool 2 'ceph-vm' replicated size 2 min_size 1 crush_rule 1 rule by-osd: step chooseleaf firstn 0 type osd pve-02 loses power. What happens to the ceph-vm pool?

   Nothing of consequence. A replicated pool at size 2 places its two copies in separate failure domains, so the two surviving nodes hold a complete copy of every placement group between them. The pool stays active, the health line reports the four down OSDs as a warning, and once mon_osd_down_out_interval elapses the cluster backfills the missing replica onto the survivors and returns to HEALTH_OK inside the hour. The only cost is reduced read parallelism while that runs. The pool stops accepting writes immediately. min_size is the number of copies a placement group must still be able to place before Ceph will serve I/O on it, so a pool at size 2 with min_size 1 keeps no spare copy at all: losing one replica drops every affected placement group under the threshold and the primary OSD blocks client writes. Every RBD image on ceph-vm hangs on its first write until pve-02 returns, which is why min_size 1 makes a two-copy pool fail closed. Some placement groups lose both replicas at once, because the CRUSH rule chooses leaves by OSD rather than by host, so nothing prevents both copies of a PG from landing on the same node. Those PGs go inactive and every RBD image with an extent in them stalls. The PGs that keep one copy stay writable, because min_size 1 accepts a single replica - which is the more dangerous outcome, since production continues with no redundancy and nothing says so. All placement groups go degraded but stay writable. chooseleaf guarantees that the replicas of a placement group are taken from distinct leaves, so no placement group can lose both of its copies to a single node however the rule is written; the affected PGs report degraded, the cluster marks the pve-02 OSDs out, backfill rebuilds the second copy across the remaining eight OSDs, and HEALTH_OK returns without operator action. The finding here is a capacity one, not an availability one.

2. Q2. The handover note states the estate is N+1 on memory: 506 GiB of guest memory is allocated across three nodes with 256 GiB each, so 506 fits inside the 512 GiB that two surviving nodes provide. Is that arithmetic sound?

   No. 256 GiB of node memory is not 256 GiB of guest capacity. On these nodes the ZFS ARC is capped at 16 GiB, four OSDs at the default 4 GiB osd_memory_target account for roughly 16 GiB more, and the host, the monitors and the PVE services take several more - call it 40 GiB reserved, leaving about 216 GiB per node for guests. Two survivors offer roughly 432 GiB against 506 GiB allocated. Worse, the smallest node holds 121 GiB of guests while the two largest have 18 and 29 GiB of headroom between them, so no single node failure is absorbable at all. Yes. Ballooning turns the allocated figure into an upper bound rather than a commitment: with a minimum set on each guest the balloon driver hands pages back to the host as soon as host pressure crosses the threshold, so the sum of the minimums is what actually has to fit. The guests on this estate carry roughly half their maximum as the floor, which puts the real commitment on the surviving pair nearer 260 GiB than 506 GiB, and two nodes at 256 GiB absorb that with room left over for the OSDs. Yes, provided KSM is running. ksmtuned begins merging identical pages once host memory use passes its threshold, and across a fleet of near-identical Debian and Windows guests the deduplication ratio on an estate of this shape settles between 20 and 30 per cent of allocated memory. That recovers well over the shortfall on its own, which is why the sizing convention for a hyperconverged cluster is to plan against the post-merge figure rather than the raw allocation total. It cannot be established from what you have been given. Allocated memory per node is not recorded anywhere durable: each guest configuration holds a memory line, but nothing in the cluster filesystem associates a guest with the node it is currently running on, so a per-node total can only be read from the live resources endpoint at the moment you ask for it. Without a shell on the estate the arithmetic is unauditable, and the correct finding to record is that the handover note cannot be checked from the material supplied rather than that its conclusion is wrong.

3. Q3. A register that lists this estate item by item understates it. Which findings combine into the mechanism that turns one node failure into a simultaneous reboot of all three nodes? Select all that apply.

   The ceph-vm pool is configured with size 2. Corosync ring0 runs over the single bond that also carries the Ceph public network, the Ceph cluster network and all VM traffic, with no second link configured. pve-01 carries quorum_votes 2, so the cluster has 4 votes and needs 3 for quorum. The cluster has one ceph-mgr and no standby. HA is enabled on five guests, the watchdog is softdog, and fencing has never been exercised on this estate. The .mgr pool is configured with size 3 and min_size 2.

4. Q4. Because pve-01 holds two votes in a three-node cluster, the loss of pve-01 leaves 2 votes of an expected 4, which is not a majority. The two surviving nodes therefore become inquorate and their HA watchdogs reset them, so a single node failure takes the whole cluster down.

   True False

5. Q5. The PBS backup job has completed with status OK every night for the last ninety-one nights. What does that green status actually establish about this estate?

   That the guests in the job can be restored. The PBS client validates every chunk against its digest as it writes it and the server refuses the snapshot if any chunk fails that check, so a job that reaches status OK has already proved the data on the datastore is readable and complete. Ninety-one consecutive successes additionally establish that the datastore has had room every night and that the client has held working credentials throughout, which between them cover the failure modes a restore would have exposed. The one component still unproven is the datastore hardware, and a datastore that has accepted ninety-one snapshots without an I/O error is not the part of this estate to worry about. That the guests in the job can be restored, provided the datastore passes garbage collection. Garbage collection walks the index file of every snapshot and touches each chunk it references, so a clean GC run performs the same integrity proof a verify job does and performs it over the whole datastore at once - which is why PBS schedules garbage collection by default and leaves verification as an option for the cautious. Nothing in this datastore is failing GC, so the chain from snapshot index to chunk is intact and the restore itself is a formality. Nothing at all. A completed backup job records that a task exited zero, and nothing more: the client reports success as soon as it has finished issuing its writes, without waiting for the server to acknowledge the chunks or to write the index, so a green line is entirely compatible with an empty snapshot on the datastore. Ninety-one green nights therefore establish exactly what one green night establishes, which is that a scheduled task ran. The only evidence a backup ever produces is a restore, and everything short of that is decoration. That chunks were written. It establishes nothing else here: no verify job exists, so nothing has ever re-read those chunks against their checksums; no restore has ever been performed, so nothing has ever proved a guest boots from them; the datastore lives on a disk of a VM running on the cluster it protects, on the same Ceph pool, so it shares the failure it exists to survive; retention is keep-last 3; garbage collection last ran in November so the datastore is at 91% and reclaims nothing; the encryption key exists only in /etc/pve/priv on the cluster; and the ERP database is excluded from the job entirely.

6. Q6. HA recovery starts a guest on a surviving node - it is a cold start, not a live migration. Which of these configurations will prevent an HA-managed guest from being recovered onto another node in this estate? Select all that apply.

   A disk on local-lvm, which exists only on the node that created it. hostpci0 naming a PCI address directly, with no cluster resource mapping behind it. agent 0, so the QEMU guest agent is not in use. An ISO mounted from the node-local local storage through ide2. cpu host, on a cluster whose third node is a different CPU generation.

7. Q7. vm 105, the ERP database, is configured with cpu host. pve-01 and pve-02 are Xeon Gold 6338; pve-03 is a Xeon Gold 6542Y. What is the finding, and what is it not?

   It blocks HA recovery. A guest configured with cpu host records the model of the CPU it was created on, and the HA stack refuses to start it on a node that cannot present the same feature set, so the loss of pve-01 or pve-02 leaves vm 105 stopped with a start error in the task log instead of recovered onto a survivor. Live migration is unaffected, because migration negotiates a common feature mask between source and destination before the transfer begins. The finding is therefore about the unplanned case rather than about rolling maintenance, and nothing in the handover note records it. It blocks live migration to pve-03, because cpu host presents the source CPU model and the destination cannot supply the same feature set. So the rolling-maintenance procedure - evacuate a node, patch it, return it - fails on this guest and only on this guest, which happens to be the ERP database. It does not block HA recovery: HA cold-starts the guest, and it will boot against whatever CPU it lands on. The second-order finding is that the estate has no evacuation path for its most important workload, and nobody discovered that because nobody has ever evacuated a node. Nothing worth recording. cpu host is the recommended setting for database workloads because it exposes the full instruction set of the physical CPU, and KVM handles the generational difference transparently: the destination masks off any flag the incoming guest is using that it cannot supply, and the guest carries on with the reduced feature set once the migration completes. The only visible effect is a modest loss of throughput while vm 105 happens to be resident on pve-03, which on this hardware is inside the noise. It corrupts the guest during backup. A snapshot-mode backup of a running guest writes the saved CPU state alongside the disk images, and that state carries the model and the feature flags of the host it was taken on, so a restore onto pve-03 resumes a guest holding flags the destination cannot supply and it faults on the first instruction that uses one. The remediation is to move any guest configured with cpu host onto stop-mode backups until the cluster is homogeneous again. It prevents the guest from starting after any kernel upgrade. cpu host pins the feature set discovered at first boot into the guest configuration, so as soon as a node takes a new kernel or a microcode update the recorded flags stop matching what the host advertises and qm start refuses the guest until the cpu line is rewritten by hand. That is why the ERP database has not been rebooted since March, and it is the loose end the previous engineer never got to the bottom of.

8. Q8. The noout flag has been set on this Ceph cluster since March. Given that the pool is at 71% raw usage, leaving it set is a defensible operational choice, because it prevents an expensive and possibly unsurvivable rebuild after a transient node failure.

   True False

9. Q9. cluster.fw carries policy_in DROP, enable 0, an IPSET named management holding 192.0.2.0/24, a rule permitting 8006 from that IPSET, and a rule permitting SSH from 0.0.0.0/0. The administrators moved to 198.51.100.0/24 during the March renumbering. Somebody sets enable 1. What happens?

   Nothing changes. The datacenter firewall is a template rather than a control: cluster.fw holds the rules and the IPSETs, but nothing reaches the kernel on a node until that node also carries enable 1 in its own host.fw, and only pve-03 does. Setting enable 1 at the datacenter level therefore arms one node out of three, leaves the two nodes the administrators reach the interface on untouched, and looks from the console like a change that did nothing at all. The finding is the inconsistency between the nodes, not a lockout. All three nodes lose corosync, become inquorate, and the HA watchdogs reset them. policy_in DROP applies to every packet arriving on a node, the corosync UDP ports included, and the rules Proxmox generates cover only the interface carrying the web interface, so the ring falls silent on the first token after the apply. Sixty seconds later the softdog on each node fires and the whole estate reboots together, which is the usual way an untested rule set takes a cluster down rather than merely locking its administrators out. Management access survives. Proxmox writes an implicit accept for port 8006 from every network a node holds an address on, ahead of the user rules, precisely so that a mistaken rule set cannot remove the interface an administrator would need in order to repair it. The stale management IPSET is therefore a tidiness finding rather than an availability one: it narrows nothing, because the built-in rule is evaluated first and matches before the IPSET rule is ever reached. Cluster traffic between the nodes keeps working - Proxmox generates rules for the detected local network, so corosync and node-to-node management survive the apply. What does not survive is every administrator. The management IPSET still holds the old node network, so 8006 is permitted from a range no workstation sits in, and the web interface becomes unreachable from every desk at once. The one path that stays open is SSH from the entire internet, which is the single rule in this file that should not exist. The apply is rejected. pve-firewall compiles the rule set before it loads it and refuses to install one that would drop the address the change was submitted from, the same protection the network stack applies to a bridge edit, and the API returns an error naming the rule that would have caused the lockout. The administrators see a failed task, correct the management IPSET to 198.51.100.0/24, and enable the firewall successfully on the second attempt, so the stale range never becomes an incident at all.

10. Q10. The API token monitoring@pve!checkmk exists so a monitoring system can read node status. It has privsep 0, expire 0, and monitoring@pve holds Administrator on /. Which statements about it are accurate? Select all that apply.

    With privsep 0 the token carries the full privilege of the user that owns it, so this token can destroy every guest, every storage definition and every backup job in the cluster. expire 0 means the token expires at the end of the current calendar year and must be reissued each January. The gap between the privilege the purpose needs - PVEAuditor, scoped to /nodes - and the privilege granted is the finding. The role name is a symptom of it, not the finding itself. API tokens cannot perform destructive operations, so the excessive role is a paperwork issue rather than a risk. The token secret sits in a monitoring configuration on a VM that runs on this cluster, so the credential and everything it can destroy share a failure domain and a blast radius.

11. Q11. vm 105, the ERP database, is excluded from the PBS job. In its place, a cron entry on pve-01 runs a qm snapshot nightly and discards its output. The guest has 41 snapshots, the newest dated 2026-03-19, and it has been running on pve-02 since the pve-01 kernel reboot that month. State what you would establish, what you would change, and in what order.

12. Q12. A qm snapshot of a guest whose disk is an RBD image is stored in the same Ceph pool as the disk it protects, so it cannot survive the loss of that pool; and rolling back to one discards every write made since it was taken. Neither property is what the word backup means.

    True False

13. Q13. vm 101 and vm 102 are backed up nightly by PBS in snapshot mode. Both have agent 0. What consistency does the resulting backup have?

    Crash-consistent only. Without the guest agent, Proxmox cannot ask the guest to freeze its filesystems, so the backup captures the disks in the state a power cut would have left them. A journalling filesystem will replay and mount; a database will run its own crash recovery, which usually works and is not something to find out during a restore. The fix is the agent, plus a database-native dump alongside the image for anything transactional. Application-consistent. QEMU holds the guest I/O queue for the duration of the snapshot regardless of the agent, so no write is in flight when the dirty bitmap is taken and every application sees the disk exactly as it stood at that instant. The guest agent adds a clean shutdown signal and the IP addresses on the summary page, but it plays no part in backup consistency - which is why a backup job raises no warning about agent 0, and why stateless and transactional guests are treated identically by the scheduler. Filesystem-consistent. Snapshot mode takes a qcow2 snapshot underneath the running guest, and a qcow2 snapshot is by construction taken between two completed writes, so what lands on the datastore is always a point at which the filesystem metadata is coherent and no journal replay is needed. What it cannot promise is application consistency, because a database may have committed to its own write-ahead log without having flushed it, and closing that last gap is what the guest agent is for. Identical to stop mode. Both modes end with a static image of the disk on the datastore, and the difference between them is only how long the guest is unavailable while it is produced: stop mode shuts the guest down first, and snapshot mode performs the same freeze internally without the guest noticing it happened. The resulting chunks are byte-for-byte the same, so agent 0 costs nothing here, and the only reason to prefer stop mode is that it finishes sooner on a busy guest.

14. Q14. An audit that rates everything Critical has not audited anything. Which of these would you record with a severity of Low or None - real observations that change nothing about whether this estate survives a node failure? Select all that apply.

    The local dir storage present on all three nodes, holding ISO images and container templates. The pve-no-subscription repository configured on pve-01 and pve-02. The pvetest repository enabled on pve-03, which is running a kernel and a qemu-server package ahead of the other two nodes. ceph -s reporting 1 daemons have recently crashed, from an OSD crash in March that was investigated at the time and closed. The ceph-vm pool at 128 placement groups with the autoscaler in warn mode, giving roughly 21 PGs per OSD.

15. Q15. You have one four-hour change window before month-end freeze. Which sequence of work reduces the most risk, and why is the ordering the answer rather than the list?

    Rebuild the CRUSH rule to failure domain host first, since replica placement is the root of the storage risk and nothing else about the storage is worth correcting while both copies of a placement group can still land on one node - every later change simply inherits the defect. Raise size to 3 inside the same window so the two data movements coalesce into a single backfill instead of two consecutive ones, and let the remap run overnight against the freeze while nobody is making changes on top of it. Quorum and HA are single configuration lines that can be edited from any node at any time without a window at all, so they belong in the next one, once the storage underneath them is sound. Clear noout first so the cluster can self-heal. The flag has been suppressing recovery since March, so the estate is carrying an unknown quantity of deferred repair, and clearing it lets Ceph mark the stale OSDs out and rebuild while somebody is watching the health line - which is what a change window is for. Once the cluster is telling the truth about its own state, work through the backup findings, a verify job and a restore test, and finish with the corosync link, because network changes are the ones most likely to need a rollback and should sit at the end where there is least to unwind. Correct the corosync vote count and set min_size 2 on ceph-vm first: neither moves a byte of data, both take seconds, and between them they close the two paths that turn one node failure into total loss. Then remove HA from the three guests it cannot recover, because HA that cannot act is worse than no HA - it produces error states and an assurance nobody has audited. Then add the unused 25 GbE pair as corosync link1. The CRUSH rule change and size 3 are the correct fixes and both move tens of terabytes, so they need capacity and a window this one is not. Clearing noout waits until after them, because the automatic rebuild it re-enables is the thing that fills the cluster. Restore a guest from PBS first to establish whether the backups work at all, then decide the rest of the plan from what the restore shows. Nothing else in the register can be sequenced honestly until you know whether a recovery path exists: if the datastore turns out to be unreadable then the storage and quorum work becomes urgent for a different reason, and if it restores cleanly then the estate can defensibly carry its current risk for another month. Spending the window on the one test that reprices every other finding is the disciplined choice. Enable the firewall and rotate the API token first, since a credential holding Administrator on the whole cluster and an SSH rule open to the entire internet are the only findings here that somebody outside the organisation can act on tonight. Reliability findings need a failure to happen before they cost anything, and a compromised token needs nobody at all. Enable the firewall from the console with the rule set as it stands, replace the token with a scoped one, and take the storage and quorum work into the next window once the perimeter is closed.

16. Q16. Which single change most reduces the probability that one compromised credential destroys both production and every copy of it?

    Encrypt the PBS datastore, so an attacker who reaches the chunks cannot read them. Client-side encryption puts the contents beyond an intruder who obtains filesystem access to the datastore, and it is the control this estate is closest to already having, since a key exists in /etc/pve/priv and only needs applying to the job. Confidentiality is the property under attack when a single credential owns both production and the copy of it. A pull-mode sync job on a second PBS instance outside this cluster, holding its own credentials, its own storage and its own retention, with a copy of the encryption key kept off the cluster. Pull means the source never holds a credential for the destination, so an identity that owns production cannot reach the copy at all. Raise retention from keep-last 3 to keep-daily 30, so there is a version old enough to predate the compromise. Three days is shorter than the dwell time of most intrusions, so the estate as it stands would restore a compromised guest onto itself and call it a recovery; thirty daily snapshots put a known-good point behind any realistic dwell time and cost only the deduplicated delta on the datastore. Schedule garbage collection weekly so the datastore stops filling and backups keep completing. At 91 per cent with the last run in November this datastore will refuse writes within days, and a backup job that has stopped running is a copy that no longer exists - which makes the continued availability of the copy the first thing to secure, before any question of who is able to reach it. Weekly collection restores the reclaim cycle. Enable two-factor authentication on every human account in the cluster, so a stolen password on its own can no longer reach the API. Interactive credentials are how an attacker arrives, and enforcing a second factor at the realm closes that route for every administrator at once without touching a single backup job or storage definition. It is the only option here that reduces the chance of the compromise happening at all.

17. Q17. PBS verification reads every chunk of a snapshot and checks it against its recorded checksum. A datastore in which all snapshots are verified is therefore proof that a restore will succeed.

    True False

18. Q18. Every node has an unused pair of 25 GbE ports. Describe the traffic separation you would build with them, state which finding each part of it closes, and name what it does not close.

Passing score: 75%. Answers are checked in this browser.