REST API automation patterns · Proxmox VE

Why this matters in production

The Proxmox REST API is the foundation of every automation tool. Using it correctly means the difference between a reliable automation and a 3 AM incident.

The API calls themselves are the easy part. What separates automation that survives contact with production is everything around them: which identity it uses, whether it verifies who it is talking to, what it does when a call fails, and whether anyone can tell afterwards what it did.

Authentication

Proxmox supports two authentication methods:

Tickets (PVEAuthCookie): short-lived, returned by /access/ticket. Use for interactive API use.
API tokens: long-lived, sent in an Authorization header. Use for automation.

The wiki gives the token header format directly: PVEAPIToken=USER@REALM!TOKENID=UUID, and notes that “API tokens do not need CSRF values for POST, PUT or DELETE” — which is the practical reason to prefer them for scripts. A ticket-authenticated client must also fetch and send a CSRF token on every write; a token-authenticated one does not.

Read-only / Safea token-authenticated read— Read-only. The token secret lives in an environment variable rather than the command line, so it does not appear in the shell history or in the process list where any user on the host can read it with ps.

PVE_HOST=pve-01.example.com
PVE_TOKEN='automation@pve!ci=REPLACE_ME'

curl -sS \
-H "Authorization: PVEAPIToken=$PVE_TOKEN" \
"https://$PVE_HOST:8006/api2/json/nodes" | jq -r '.data[].node'

Scope the token to the job

API tokens are tied to a user. With --privsep 1, the token has its own ACLs; with --privsep 0, it inherits the user’s permissions.

Configuration changecreate a scoped automation identity— Creates a dedicated user, grants it access to one path only, and issues a token with privilege separation so the token cannot exceed what is granted to the token itself. The secret is printed once and cannot be retrieved again.

pveum user add automation@pve --comment "CI pipeline - ticket OPS-1423"

pveum acl modify /vms/100 --users automation@pve --roles PVEVMAdmin

pveum user token add automation@pve ci --privsep 1

pveum acl list

A `root@pam` token with `--privsep 0` is the default that everyone reaches for

It is the fastest thing that works. The documentation example uses root@pam, every forum answer uses root@pam, and creating a user, an ACL and a token takes three commands instead of one. So the token that ends up in the CI system is root@pam!automation with privilege separation off.

What that token can do is everything: destroy any guest, delete any storage, change any user’s permissions, dissolve the cluster. It is stored in a CI variable, visible to anyone who can edit a pipeline, and it will be copied into a second system within a month because it is the one that is known to work.

Why the usual reassurances do not hold. “It is only used for read operations” is a property of the current script, not of the token. “It is in a secret store” protects it at rest and not from the pipeline that legitimately reads it. And “we would notice” is undermined by the next section: without privilege separation, every action it takes is logged as root@pam, indistinguishable from a human administrator.

The fix is three commands and it is the ones above. A dedicated user, an ACL on the narrowest path that works, and --privsep 1. When that token is compromised the blast radius is one path, and when it does something unexpected the task log names it.

Verify TLS, do not disable it

Almost every Proxmox API example on the internet passes -k or --insecure. It is there because a fresh install has a self-signed certificate and the example needs to work without setup. Carrying it into automation means your script will authenticate to anything that answers on port 8006 — and it will hand over the token while doing so.

Configuration changetrust the cluster's own CA instead— Copies the cluster CA certificate to the client and points curl at it. The API is then verified normally, and a script that suddenly cannot connect is telling you something real rather than being suppressed.

PVE_HOST=pve-01.example.com

# on a cluster node:
cat /etc/pve/pve-root-ca.pem

# on the automation host, having placed that file:
curl -sS --cacert /usr/local/share/ca-certificates/pve-root-ca.crt \
-H "Authorization: PVEAPIToken=$PVE_TOKEN" \
"https://$PVE_HOST:8006/api2/json/version" | jq .

The better answer, where it is available, is an ACME certificate from a CA the client already trusts — Proxmox supports this natively through pvenode acme, and then no client-side configuration is needed at all.

Common API patterns

Read-only / Saferead guest configuration across the cluster— Read-only. /cluster/resources answers cluster-wide questions from any node, which qm list cannot. Always parse JSON rather than the human-readable text format.

curl -sS -H "Authorization: PVEAPIToken=$PVE_TOKEN" \
"https://$PVE_HOST:8006/api2/json/cluster/resources?type=vm" \
| jq -r '.data[] | [.vmid, .name, .node, .status] | @tsv'

Service impact possiblestart a VM, and capture the UPID it returns— Starting a guest is a state change. The response body is a UPID, not a result — the VM is not running when this returns. Poll the task to completion before doing anything that depends on it.

NODE=pve-01
VMID=100

UPID=$(curl -sS -X POST \
-H "Authorization: PVEAPIToken=$PVE_TOKEN" \
"https://$PVE_HOST:8006/api2/json/nodes/$NODE/qemu/$VMID/status/start" \
| jq -r .data)

echo "task: $UPID"

Idempotency and drift are large enough subjects to have their own lesson; xx-automation-idempotency covers the configuration digest, deterministic identity and reconciliation. Task polling is in xx-cli-tasks-and-upids. This lesson stops at the envelope.

Error handling: which failures are retryable

The API returns:

Status	Meaning	Retry?
200	Success	—
400	Bad request; validation failed	No. The request is wrong; retrying sends the same wrong request
401	Unauthenticated	No, unless a ticket expired — then re-authenticate once
403	Authenticated but forbidden	No. A permissions problem does not resolve itself
404	Not found	No, usually — but see below
500	Server error	Yes, with backoff
596	No route to host / node unreachable	Yes, with backoff

Retrying a 4xx is how automation causes damage

The instinct is to retry everything, because most transient failures are network failures and those do resolve. Applied to the 4xx class it produces two specific harms.

A retried 400 or 403 amplifies a failing job into a load problem. A pipeline retrying a permissions error every second, across a fleet, is a denial of service you inflicted on yourself — and it fills the task log, burying the events you need to diagnose it.

A retried create is how duplicate guests appear. If the create failed because the VMID was taken, retrying with a freshly allocated VMID succeeds and leaves the first attempt’s partial state orphaned. The general rule: retry only what a different outcome is possible for. A 403 will be a 403 forever.

The one 404 worth a single retry is immediately after a create, where the resource may not yet be visible cluster-wide. Even there, waiting for the task is the correct fix.

Configuration changea retry wrapper that only retries what is retryable— Separates the HTTP status from the body, retries 5xx and connection failures with exponential backoff and a cap, and fails immediately on 4xx. Print the response body on failure — the API's error messages are specific and discarding them is the commonest reason an incident takes hours.

#!/usr/bin/env bash
set -euo pipefail

api() {
local method="$1" path="$2"; shift 2
local attempt=0 max=5 delay=1 code body

while :; do
  body=$(curl -sS -X "$method" -w '\n%{http_code}' \
    --cacert /usr/local/share/ca-certificates/pve-root-ca.crt \
    -H "Authorization: PVEAPIToken=$PVE_TOKEN" \
    "https://$PVE_HOST:8006/api2/json$path" "$@" ) || body=$'\n000'

  code=$(printf '%s' "$body" | tail -n1)
  body=$(printf '%s' "$body" | sed '$d')

  case "$code" in
    2??) printf '%s' "$body"; return 0 ;;
    4??) echo "non-retryable $code on $path: $body" >&2; return 1 ;;
  esac

  attempt=$(( attempt + 1 ))
  if [ "$attempt" -ge "$max" ]; then
    echo "giving up after $attempt attempts, last $code: $body" >&2
    return 2
  fi
  echo "attempt $attempt got $code, retrying in ${delay}s" >&2
  sleep "$delay"
  delay=$(( delay * 2 ))
done
}

api GET /cluster/resources?type=vm | jq -r '.data[].vmid'

Three details worth keeping when you port this to another language.

Exponential backoff with a cap, not a fixed interval. A fixed one-second retry from fifty machines against a struggling cluster is a synchronised load spike. Doubling the delay spreads them out, and adding a small random jitter spreads them further.

A distinct exit code for exhaustion. “The call failed” and “we stopped trying” are different facts for the caller, because the second means the operation may still be in progress.

The response body on failure. The API’s error strings are specific — VM 900 already exists on node 'pve-02' rather than a generic code. Discarding them turns a five-minute diagnosis into an evening.

Make your automation attributable

Everything your script does appears in the task log as its identity

This is the payoff for using a dedicated user rather than root@pam. Every state change becomes a task, and the task records who requested it.

Read-only / Safewhat has the automation identity been doing?— Read-only. userfilter narrows the node's task history to one identity, which turns 'did the pipeline do this' from an argument into a query.

pvesh get /nodes/pve-01/tasks --userfilter 'automation@pve' --limit 20 \
--output-format json | jq -r '.[] | [.starttime, .type, .id, .status] | @tsv'

Add a ticket or run reference to the guest’s description or tags when your automation creates something, so the artefact carries its own provenance rather than relying on a task log that rotates.

Common mistakes

Using root@pam with --privsep 0. It is the fastest thing that works and the worst thing to leak, and it makes every action indistinguishable from an administrator’s.
Carrying -k from an example into production. Your script will hand its token to anything answering on 8006. Trust the cluster CA, use ACME, or pin.
Putting the token secret on the command line. It lands in shell history and in ps.
Retrying 4xx responses. They do not resolve, and a retried create makes duplicates.
Fixed-interval retries. Synchronised retries from a fleet turn a degraded cluster into an outage.
Discarding the response body on failure. The API’s error strings usually name the exact problem.
Treating a POST response as a result. It is a UPID; the work has only started.
Parsing the text output format. It is a human table and its columns are not an interface.

Key takeaways

Tokens for automation, tickets for interactive use. Tokens skip the CSRF requirement on writes, which is why scripts prefer them.
Create a dedicated user, an ACL on the narrowest path that works, and a token with --privsep 1. Three commands, and the blast radius becomes one path.
Verify TLS against the cluster CA (/etc/pve/pve-root-ca.pem) or use ACME. Pin if you must; never disable.
Retry only 5xx and connection failures. 400, 401 and 403 will not resolve.
Back off exponentially with a cap and jitter, and use a distinct exit code for exhaustion.
Always log the response body on failure.
A dedicated identity makes the task log attributable — --userfilter then answers “did the pipeline do this”.
Idempotency, drift and task polling have their own lessons; this one is the envelope around the call.