Idempotency and drift in Proxmox automation · Proxmox VE

Part XX’s blurb promises idempotency, and every automation tool in this part — Terraform, Ansible, Proxmoxer — claims to provide it. What none of them can do is get it from the API, because the Proxmox API does not offer it.

The API is a set of remote procedure calls. POST creates a thing. PUT modifies a thing that exists. DELETE removes it. There is no endpoint that means “make the cluster look like this”, no reconciliation loop on the server side, and no notion of desired state anywhere in pvedaemon. Every tool that presents a declarative interface is implementing that on the client, by reading current state and computing the difference — and it is worth understanding how, because the places where that abstraction leaks are the places your automation will hurt you.

What “not idempotent” actually costs

Run this twice:

Read-only / Safethe naive create, run twice— Read-only here because it is deliberately not executed — read it and predict the second run's behaviour before continuing. The answer is different depending on one detail.

# run 1
pvesh create /nodes/pve-01/qemu --vmid 900 --name app-01 --memory 8192 --cores 4

# run 2, identical
pvesh create /nodes/pve-01/qemu --vmid 900 --name app-01 --memory 8192 --cores 4

The second run fails: unable to create VM 900 - VM 900 already exists. That is the good outcome — noisy, obvious, and safe.

Now consider the version most scripts actually contain, where the VMID is allocated rather than chosen:

Read-only / Safethe version that is worse for being convenient— Read-only here for the same reason. nextid returns the lowest free VMID, so a second run allocates a different one — and succeeds.

VMID=$(pvesh get /cluster/nextid)
pvesh create "/nodes/pve-01/qemu" --vmid "$VMID" --name app-01 --memory 8192 --cores 4

Run that twice and you have two virtual machines called app-01, on different VMIDs, both configured identically, both consuming memory, one of them unknown to whatever registered the first. It succeeds. Nothing warns you. And this is the pattern almost every getting-started guide shows.

/cluster/nextid is a race, and a retry loop turns it into duplicate guests

GET /cluster/nextid returns the lowest unused VMID. It does not reserve it. Between the moment it answers and the moment your POST lands, any other caller — a colleague in the GUI, a second pipeline run, a Terraform apply — can take the same number.

The observable failure is a create that fails with “already exists” for a VMID you were just told was free. That is confusing but survivable.

The unobservable failure is the one that matters. A script that treats a create failure as transient and retries — which is a sensible-looking default, since most API failures are transient — calls nextid again, gets a different number, and succeeds. You now have the guest you asked for, plus whatever the first attempt actually managed to create before it failed, plus no record connecting the two. On a busy provisioning system this accumulates: orphaned disks on shared storage, VMIDs nobody can account for, and a monthly storage bill that grows for no visible reason.

The fix is to stop asking. Derive the VMID from something stable — the ticket number, a hash of the hostname, a block allocated per environment — and use the same one every time that guest is created. A deterministic identity turns a duplicate into a collision, and a collision is an error you can see.

If you must allocate dynamically, nextid accepts an optional vmid parameter that checks a specific ID rather than returning the next free one, which lets you probe your derived choice before using it.

The three properties an idempotent operation needs

Deterministic identity. The same logical guest maps to the same VMID on every run.
Read before write. Fetch current state, compute the difference, write only what differs.
A guard against concurrent modification. So that “what differs” does not become stale between reading and writing.

Proxmox gives you the third for free, and almost nobody uses it.

The digest: optimistic concurrency you already have

Every guest configuration read returns a digest field. Every configuration write accepts one. The API documentation describes it precisely: “Prevent changes if current configuration file has different SHA1 digest. This can be used to prevent concurrent modifications.”

Read-only / Saferead the config and its digest— Read-only. The digest is a SHA1 over the configuration file's contents. Any change by anyone — GUI, another script, an operator at a shell — changes it.

NODE=pve-01
VMID=100

pvesh get "/nodes/$NODE/qemu/$VMID/config" --output-format json | jq -r '.digest, .memory, .cores'

Configuration changewrite with the digest as a guard— Changes the VM configuration only if it has not been modified since you read it. If someone else changed anything in between, the write is rejected rather than applied on top of their change. This is the single highest-value line in this lesson.

NODE=pve-01
VMID=100

DIGEST=$(pvesh get "/nodes/$NODE/qemu/$VMID/config" --output-format json | jq -r .digest)

pvesh set "/nodes/$NODE/qemu/$VMID/config" --memory 16384 --digest "$DIGEST"

Read-only / Safethe guard doing its job— Illustrative. This is a rejection, not a bug. Someone changed the configuration between your read and your write; re-read, recompute the difference, and try again.

# pvesh set /nodes/pve-01/qemu/100/config --memory 16384 --digest 4f2a...

400 Parameter verification failed.
digest: detected modified configuration - file changed by other user? Try again.

Illustrative output

Why this matters more than it sounds

Without the digest, a reconcile loop that runs every ten minutes will happily overwrite a change an operator made three minutes ago during an incident. The operator raised a guest’s memory to get it through a spike; the loop reads its own desired state, sees a difference, and puts it back. The guest OOMs again. The operator raises it again. Nobody connects the two events because the loop’s log says “converged”, which from its point of view is true.

The digest makes that impossible in the one way that matters: the write fails instead of succeeding wrongly, and a failure is something you can alert on.

This is the same pattern as an HTTP If-Match header or a database optimistic lock. Proxmox implemented it years ago and it is off by default because it is a parameter you have to pass.

Read before write: computing the difference

Configuration changean idempotent 'ensure' for guest configuration— Reads current state, compares against intent, and writes only the keys that differ — carrying the digest so a concurrent change aborts rather than being overwritten. Running it repeatedly on an already-correct guest performs no write at all.

#!/usr/bin/env bash
set -euo pipefail

NODE=pve-01
VMID=100

# intent
declare -A WANT=( [memory]=16384 [cores]=4 [cpuunits]=400 [onboot]=1 )

CUR=$(pvesh get "/nodes/$NODE/qemu/$VMID/config" --output-format json)
DIGEST=$(echo "$CUR" | jq -r .digest)

ARGS=()
for KEY in "${!WANT[@]}"; do
HAVE=$(echo "$CUR" | jq -r --arg k "$KEY" '.[$k] // "unset"')
if [ "$HAVE" != "${WANT[$KEY]}" ]; then
  echo "drift: $KEY is '$HAVE', want '${WANT[$KEY]}'"
  ARGS+=( "--$KEY" "${WANT[$KEY]}" )
fi
done

if [ "${#ARGS[@]}" -eq 0 ]; then
echo "no change needed"
exit 0
fi

pvesh set "/nodes/$NODE/qemu/$VMID/config" "${ARGS[@]}" --digest "$DIGEST"
echo "applied ${#ARGS[@]} change(s)"

Three things that script does deliberately.

It prints the drift before applying it. A reconcile step that silently fixes things is a reconcile step whose findings you never see. The log line is often more valuable than the fix.

It performs no write when nothing differs. That is not merely efficient. Every write to a guest configuration creates a task, appears in the task log, and — for a running guest — may create a pending change. A loop that writes unconditionally fills the task history with noise and makes real changes impossible to find.

It carries the digest read in the same breath as the values. Reading the config and the digest in two separate calls reintroduces exactly the race the digest exists to close.

Removing a setting: `--delete`

Setting a key to an empty value does not remove it. The API has a dedicated parameter, documented as “a list of settings you want to delete”.

Configuration changeremove settings rather than blanking them— Removes the named keys from the guest configuration entirely, so the guest falls back to the default. This is what a reconcile step needs when intent no longer includes a setting that the guest currently has.

NODE=pve-01
VMID=100

DIGEST=$(pvesh get "/nodes/$NODE/qemu/$VMID/config" --output-format json | jq -r .digest)

pvesh set "/nodes/$NODE/qemu/$VMID/config" --delete cpulimit,hookscript --digest "$DIGEST"

pvesh get "/nodes/$NODE/qemu/$VMID/config" --output-format json | jq 'has("cpulimit")'

This is the gap that makes most home-grown “declarative” wrappers wrong. They set every key they know about and never delete the ones they do not, so a cpulimit applied during an incident six months ago persists forever, invisible to a tool that believes it owns the guest’s configuration.

Pending changes: the state between intent and reality

A running guest cannot accept every configuration change live. Changing cores, machine or bios on a running VM records a pending change, applied at the next start. qm config shows the current values; the pending set is a separate view.

Read-only / Safewhat is configured versus what is running— Read-only. The pending endpoint returns key, value (in effect now) and pending (queued for next start). A drift check that reads only the config will report success while the running guest is still on the old value.

NODE=pve-01
VMID=100

pvesh get "/nodes/$NODE/qemu/$VMID/pending" --output-format json \
| jq -r '.[] | select(.pending != null) | [.key, (.value|tostring), (.pending|tostring)] | @tsv'

Read-only / Safea guest that is not what its config says it is— Illustrative. cores is 4 in effect and 8 pending; the change was accepted, is recorded, and has not happened. A monitoring check reading the config alone reports 8.

# pvesh get /nodes/pve-01/qemu/100/pending --output-format json | jq ...

cores	4	8
machine	pc-q35-9.2+pve0	pc-q35-10.0+pve0

Illustrative output

Configuration changediscard a pending change you no longer want— Reverts queued changes without restarting the guest. Useful when a reconcile run queued something that a later decision reversed, and you would rather not carry it to the next restart.

NODE=pve-01
VMID=100

pvesh set "/nodes/$NODE/qemu/$VMID/config" --revert cores,machine

pvesh get "/nodes/$NODE/qemu/$VMID/pending" --output-format json | jq -r '.[] | select(.pending != null) | .key'

Marking ownership

Automation that manages some guests on a cluster and not others needs to know which. Two mechanisms, both cheap.

Configuration changetag what you own, and protect what must not be deleted— Tags are described in the API as meta information only — they change no behaviour, which is exactly why they are safe to use as ownership markers. protection is different: it genuinely disables the remove VM and remove disk operations, including from a script.

NODE=pve-01
VMID=100

pvesh set "/nodes/$NODE/qemu/$VMID/config" --tags 'managed-by-ci;env-prod'
pvesh set "/nodes/$NODE/qemu/$VMID/config" --protection 1

# everything this pipeline owns, cluster-wide
pvesh get /cluster/resources --type vm --output-format json \
| jq -r '.[] | select(.tags != null and (.tags | test("managed-by-ci"))) | [.vmid, .node, .name] | @tsv'

protection is documented as “sets the protection flag of the VM. This will disable the remove VM and remove disk operations.” It is enforced by the API, so it protects against your own script as much as against a mis-click. For anything whose loss would be an incident, it costs one line and removes an entire class of accident.

A drift report that does not change anything

The most useful automation in this lesson is the one that writes nothing.

Read-only / Safereport drift across every managed guest— Read-only. Runs the comparison without applying anything, so it is safe on a schedule and safe during an incident. Run this daily for a fortnight before you let anything reconcile automatically.

#!/usr/bin/env bash
set -euo pipefail

pvesh get /cluster/resources --type vm --output-format json \
| jq -r '.[] | select(.tags != null and (.tags | test("managed-by-ci"))) | [.node, .vmid] | @tsv' \
| while IFS=$'\t' read -r NODE VMID; do
    CUR=$(pvesh get "/nodes/$NODE/qemu/$VMID/config" --output-format json)
    for KEY in cpuunits onboot protection; do
      HAVE=$(echo "$CUR" | jq -r --arg k "$KEY" '.[$k] // "unset"')
      echo "$NODE $VMID $KEY=$HAVE"
    done
  done

Under the hood

Why the API is imperative, and why that is a defensible design rather than an oversight.

A guest’s configuration is a plain text file in /etc/pve — for VM 100 on node pve-01, /etc/pve/nodes/pve-01/qemu-server/100.conf. That directory is pmxcfs, a FUSE filesystem backed by a SQLite database and replicated across the cluster by corosync. Every node sees the same contents within milliseconds of a write.

The digest is a SHA1 over that file’s contents. That is why it changes when anything about the guest changes, why it is cheap for the server to compute, and why it works identically for the GUI, qm, pvesh and a raw curl — they are all writing the same file through the same code path.

Two structural consequences follow.

The API has no place to keep desired state. A declarative API needs somewhere to store what you asked for, separately from what is currently true, plus a controller that continuously drives one toward the other. Proxmox has neither, deliberately: a cluster whose configuration is a replicated text file is one you can inspect, back up, diff and repair with ordinary tools, and adding a reconciliation controller would mean a second source of truth that can disagree with the file. Kubernetes made the opposite choice and pays for it with a much larger control plane.

Locking is per file, and it is real. Long-running operations set a lock field in the configuration — backup, migrate, snapshot, clone, rollback. Any write attempted while a lock is held fails. This is why an automation script that does not wait for its own tasks runs into lock errors rather than corruption: the file lock is the backstop for the race the script created. It is also why qm unlock exists and why it should be regarded as a last resort — clearing a lock whose operation is still running lets a second writer in, and that is how the configuration gets corrupted.

The practical synthesis for anyone writing automation: treat the configuration file as the source of truth, use the digest as your compare-and-swap primitive, keep your own desired state in version control where it belongs, and accept that the reconciliation loop is yours to write. Terraform and Ansible are doing exactly this on your behalf; understanding the mechanism is what lets you debug them when the abstraction leaks.

What happens if two pipelines reconcile the same guest at the same time?

Without a digest guard, you get last-writer-wins with no indication that anything happened — and the outcome depends on timing that nobody controls.

The mechanics. Pipeline A reads the config: memory 8192, cores 4. Pipeline B reads the same config a moment later. A writes memory 16384. B writes cores 8. Because a PUT to the config endpoint modifies only the keys you send, both writes land and the result is memory 16384, cores 8 — which is what both wanted, and everything is fine.

Now the version where it is not fine. A is reconciling toward a spec that says cpulimit 4. B is reconciling toward a spec that says no cpulimit at all, and issues --delete cpulimit. Both run every ten minutes, offset by a few seconds. The guest’s cpulimit now appears and disappears continuously. Each pipeline’s log says “converged” on every run, because each one verified its own change immediately after making it. Meanwhile the guest’s performance changes every ten minutes and no dashboard shows why.

The version that costs data. A is a reconcile loop. B is an operator at a shell during an incident, who has just set --memory 32768 to get a guest through a spike and gone back to watching the application. Six minutes later the loop reads its desired state, sees 32768 where its spec says 8192, and puts it back. The guest OOMs. The operator, reasonably, concludes their change did not take effect and applies it again.

What the digest changes. In every case above, the second writer’s call is rejected with detected modified configuration - file changed by other user? Try again. That is a visible failure with a specific message. It appears in the pipeline’s output, it can be alerted on, and it names the situation exactly.

The correct response to that rejection is not to retry blindly — that reintroduces the loop, just with extra steps. It is to re-read, recompute the difference, and decide whether the difference is still one you want to make. A reconcile loop that hits a digest conflict three times in a row on the same guest has found something a human should look at, and saying so is more valuable than converging.

The design rule. Exactly one system should own each guest’s configuration, and it should say so with a tag. Two owners is not a concurrency problem to be solved with better locking; it is an ownership problem, and the digest’s job is to make it visible rather than to make it safe.

Common mistakes

Allocating VMIDs with /cluster/nextid in automation. It does not reserve, and a retry after a failure produces duplicate guests. Derive the ID deterministically.
Writing without --digest. A reconcile loop then silently overwrites whatever an operator did during an incident.
Reading the config and the digest in two separate calls. That reopens the race the digest closes.
Setting a key to an empty value to remove it. Use --delete.
Never deleting anything. A tool that only sets keys leaves every historical override in place forever, invisible to itself.
Reading config and ignoring pending. A queued change reports as applied while the running guest is still on the old value.
Writing unconditionally when nothing differs. It floods the task log and hides the changes that mattered.
Turning on reconciliation before running the report. That is an unbounded change to production with no baseline and no rollback.
Retrying a digest conflict blindly. Re-read and recompute, and escalate if it recurs.

Key takeaways

The PVE API is imperative RPC. Declarative behaviour is implemented by the client, and every tool that offers it is doing read-diff-write.
Deterministic identity is the first requirement. /cluster/nextid does not reserve, so a retry after failure creates a duplicate guest.
The config digest is a SHA1 of the guest’s configuration file and is accepted as a write guard: “prevent changes if current configuration file has different SHA1 digest”.
Read the config and digest in one call, compute the difference, write only what differs, and print the drift you found.
--delete removes settings; setting an empty value does not.
pending is a separate view from config. A drift check that ignores it reports success for changes that have not happened.
Tag what you own; protection 1 genuinely blocks removal, including from your own script.
Report drift for weeks before reconciling it. A loop against an unknown baseline is an unbounded production change.

Knowledge check

Why this matters

Most Proxmox automation is written the way most automation is written: someone needed a VM created, they found the API call that creates one, and it worked. The script grew. It got a retry loop after a flaky night. It got a nextid call because hard-coding the VMID felt clumsy. Each step was reasonable and the result is a system that produces duplicate guests under load and undoes operators’ emergency changes on a ten-minute cycle.

None of that is a Proxmox defect, and it is worth being clear about why. An imperative API is a legitimate design, and the one Proxmox has is unusually honest about it — it gives you a compare-and-swap primitive, a delete parameter, a pending view and a protection flag, which is more than many imperative APIs bother with. What it does not do is pretend to be declarative. The pretending, when it happens, is done by the tool on top, and by the mental model of whoever wrote the script.

The transferable idea is that idempotency is a property of the caller, not of the endpoint. POST /qemu will never be idempotent. A script that derives its identity deterministically, reads before writing, guards with a digest, deletes what is no longer wanted and refuses to write when nothing differs is idempotent, and it is idempotent against an API that offers no help beyond one optional parameter. That is a skill that transfers to every infrastructure API you will meet, most of which offer less.

The last point is the one that costs teams the most and is the easiest to act on: report before you reconcile. Every drift-correcting system should spend its first few weeks writing nothing at all. The output of that period is the most accurate description of your estate anyone has ever had — the settings nobody remembers applying, the guests nobody claims, the deliberate exception that was never documented. Fixing those by hand, with a human deciding each one, is the work. The loop that keeps it fixed afterwards is the easy part.