Tasks, UPIDs and asynchronous operations · Proxmox VE

Ask the Proxmox API to start a virtual machine and it does not tell you whether the machine started. It tells you that it has begun trying:

Read-only / Safethe API's answer to 'start this VM'— Illustrative. This is the complete response body. It is not an error, it is not a status, and it is not a result — it is a receipt.

# pvesh create /nodes/pve-01/qemu/100/status/start

UPID:pve-01:00006A2F:04C1B2E1:6A7C83E0:qmstart:100:root@pam:

Illustrative output

Every state-changing operation in Proxmox works this way. Starting a VM, taking a backup, migrating a guest, creating a container, moving a disk: each one forks a worker process, records it as a task, and returns its identifier immediately. The API viewer confirms it — POST to /nodes/{node}/qemu/{vmid}/status/start has a return type of string, and that string is the UPID.

At an interactive shell this is easy to miss, because qm start usually finishes so fast that the next thing you type happens after the VM is up anyway. In a script it is the single most common source of races, and the failures it produces are intermittent, timing-dependent, and worse on a busy cluster — which is to say, worst in production.

What a UPID encodes

A UPID is a colon-delimited string with a trailing colon, and every field is there for a reason.

UPID:pve-01:00006A2F:04C1B2E1:6A7C83E0:qmstart:100:root@pam:
     |      |        |        |        |       |   |
     |      |        |        |        |       |   +-- user@realm
     |      |        |        |        |       +------ task id (usually VMID)
     |      |        |        |        +-------------- task type
     |      |        |        +----------------------- start time, hex Unix epoch
     |      |        +-------------------------------- process start, hex jiffies
     |      +----------------------------------------- worker PID, hexadecimal
     +------------------------------------------------ node that owns the task

Three of those are immediately useful.

The node. A UPID belongs to the node that ran the task. You cannot query it from a different node’s /nodes/{other}/tasks/ path — the node name in the UPID and the node in the API path have to match. Scripts that migrate a guest and then poll the wrong node are a real category of bug.

The start time. 6A7C83E0 in hexadecimal is 1786545120, a Unix timestamp. That makes a UPID self-dating without any lookup, which is useful when you find one in a log six months later.

The type and id. qmstart:100 says what was attempted and to what. vzdump:141, qmigrate:102, imgcopy:, hastart:vm:100 — the type vocabulary is small and readable, and it is what you filter on when searching the task history.

Read-only / Safedecode a UPID's timestamp— Read-only. Field 5 is the start time in hexadecimal. This is worth knowing because a UPID in a six-month-old log tells you exactly when the operation ran without any lookup at all.

UPID='UPID:pve-01:00006A2F:04C1B2E1:6A7C83E0:qmstart:100:root@pam:'

HEXTIME=$(echo "$UPID" | cut -d: -f5)
date -d "@$(( 16#$HEXTIME ))"

Polling a task correctly

Two things have to be checked, and checking only one is the classic mistake.

Read-only / Safethe two fields that matter— Read-only. 'status' says whether the worker is still running. 'exitstatus' says whether it succeeded, and it only exists once the task has stopped. A task can be stopped and failed at the same time.

NODE=pve-01
UPID='UPID:pve-01:00006A2F:04C1B2E1:6A7C83E0:qmstart:100:root@pam:'

pvesh get "/nodes/$NODE/tasks/$UPID/status" --output-format json | jq .

Read-only / Safea task that finished, and failed— Illustrative. status is 'stopped', which many scripts treat as success. exitstatus is the field that says otherwise, and 'OK' is the only value that means success.

# pvesh get /nodes/pve-01/tasks/UPID.../status --output-format json

{
"exitstatus": "unable to create VM 900 - VM 900 already exists on node 'pve-02'",
"id": "900",
"node": "pve-01",
"pid": 27183,
"pstart": 79822817,
"starttime": 1786545120,
"status": "stopped",
"type": "qmcreate",
"upid": "UPID:pve-01:00006A2F:04C1B2E1:6A7C83E0:qmcreate:900:root@pam:",
"user": "root@pam"
}

Illustrative output

'status: stopped' is not success, and this is how automation silently corrupts state

This is the failure mode. It is quiet, it is common, and it produces incidents that get blamed on everything except the script.

A deployment script starts a VM, polls until status is stopped, and proceeds. It never reads exitstatus. When the start fails — no free memory on the node, a storage that is not active, a lock held by a running backup — the task still reaches stopped, the script still proceeds, and the next step configures, registers or points traffic at a guest that is not running.

Why the first diagnosis is wrong. The symptom shows up downstream: a load balancer with an unhealthy backend, a cluster member that never joined, a database replica that is not replicating. The script’s own log says every step succeeded, because from the script’s point of view every step did. Investigation goes to the application, to the network, to DNS — everywhere except the automation, which is reporting green.

The rule: exitstatus must equal exactly OK. Anything else is the error string, and the error string is frequently informative — as in the capture above, which names the actual problem precisely. A script that reads it can log it. A script that ignores it discards the one piece of evidence that would have made the investigation five minutes long.

Note the asymmetry that makes this so persistent: the failing path and the succeeding path look identical to a script checking only status. There is no exception to catch, no non-zero exit code, no timeout. It works in testing, where nothing fails, and it breaks in production, where things do.

Reading a task log

The task log is the same output the GUI shows in its task viewer, and it survives the task by a long way — Proxmox keeps completed task logs on disk, so a backup that failed last week can still be read in full.

Read-only / Safethe full log of a task, running or finished— Read-only. The log endpoint returns an array of objects with 'n' (line number) and 't' (text). Use --limit 0 to ask for all lines rather than the default page.

NODE=pve-01
UPID='UPID:pve-01:00006C88:04C41A02:6A7BD3A3:vzdump:141:root@pam:'

pvesh get "/nodes/$NODE/tasks/$UPID/log" --limit 0 --output-format json \
| jq -r '.[].t'

The pvenode wrapper is shorter for interactive use and does the same thing:

Read-only / Safethe interactive route— Read-only. pvenode task list is the node's history; task log and task status take the UPID from it. This is the fastest path from 'something failed overnight' to the actual error message.

pvenode task list --limit 20 --errors 1

UPID='UPID:pve-01:00006C88:04C41A02:6A7BD3A3:vzdump:141:root@pam:'
pvenode task status "$UPID"
pvenode task log "$UPID"

Read-only / Safea backup task log with the real cause in it— Illustrative. The last line before the failure is the one that matters, and it is specific enough to act on. This detail exists only in the task log; the task list shows 'job errors' and nothing more.

# pvenode task log UPID:pve-01:00006C88:...:vzdump:141:root@pam:

INFO: starting new backup job: vzdump 141 --storage backup-pbs --mode snapshot
INFO: Starting Backup of VM 141 (qemu)
INFO: Backup started at 2026-08-12 02:00:03
INFO: status = running
INFO: creating Proxmox Backup Server archive 'vm/141/2026-08-12T02:00:03Z'
INFO: started backup task 'a9f21c44-2b1e-4c8d-9f10-REDACTED'
INFO:   0% (1.1 GiB of 200.0 GiB) in 3s, read: 383.4 MiB/s
ERROR: job failed with err -5 - Input/output error
INFO: aborting backup job
ERROR: Backup of VM 141 failed - job failed with err -5 - Input/output error
INFO: Failed at 2026-08-12 02:00:41

Illustrative output

Finding tasks you do not have the UPID for

This is the ordinary morning-after case: something failed, nobody recorded the UPID, and you need to find it.

Read-only / Safefilter the node's task history— Read-only. The tasks endpoint accepts errors, typefilter, vmid, userfilter, statusfilter, since and until. Filtering server-side is much faster than pulling everything and grepping, especially on a node with months of history.

NODE=pve-01

# every failed task in the last 24 hours
SINCE=$(date -d '24 hours ago' +%s)
pvesh get "/nodes/$NODE/tasks" --errors 1 --since "$SINCE" \
--output-format json | jq -r '.[] | [.starttime, .type, .id, .status] | @tsv'

# everything that touched one guest
pvesh get "/nodes/$NODE/tasks" --vmid 141 --limit 20 \
--output-format json | jq -r '.[] | [.type, .status] | @tsv'

For a cluster-wide view there is /cluster/tasks, which aggregates across nodes — useful precisely when you do not yet know which node the work happened on.

Read-only / Saferecent tasks across the whole cluster— Read-only. This is the endpoint to reach for after an HA failover, when the guest has moved and you do not know which node's history to search.

pvesh get /cluster/tasks --output-format json \
| jq -r '.[] | [.node, .type, .id, .status] | @tsv' | head -20

Making it synchronous in a script

Almost every script wants “start this VM and tell me when it is running, or fail”. That is a wrapper, and it is worth writing once properly.

Configuration changea task-waiting wrapper— Runs an API call, then polls until the task stops or the timeout expires. Returns non-zero on failure and prints the exitstatus, so the caller can react. Adjust the timeout per operation type: a VM start needs seconds, a 2 TB backup needs hours.

#!/usr/bin/env bash
set -euo pipefail

wait_for_task() {
local node="$1" upid="$2" timeout="${3:-600}"
local deadline=$(( $(date +%s) + timeout ))
local st exitstatus

while :; do
  st=$(pvesh get "/nodes/$node/tasks/$upid/status" --output-format json)
  if [ "$(echo "$st" | jq -r .status)" != "running" ]; then
    exitstatus=$(echo "$st" | jq -r '.exitstatus // "no exitstatus"')
    if [ "$exitstatus" = "OK" ]; then
      return 0
    fi
    echo "task failed: $exitstatus" >&2
    pvesh get "/nodes/$node/tasks/$upid/log" --limit 0 \
      --output-format json | jq -r '.[].t' | tail -20 >&2
    return 1
  fi
  if [ "$(date +%s)" -ge "$deadline" ]; then
    echo "timed out after ${timeout}s waiting for $upid" >&2
    return 2
  fi
  sleep 2
done
}

NODE=pve-01
VMID=100
UPID=$(pvesh create "/nodes/$NODE/qemu/$VMID/status/start")
wait_for_task "$NODE" "$UPID" 120

Four details in that wrapper are not decoration.

It polls status, then reads exitstatus. Both, in that order, because exitstatus is absent while the task is running.
It has a timeout, and a distinct exit code for it. A caller needs to distinguish “the operation failed” from “we stopped waiting”, since the second means the operation may still be in progress.
It dumps the tail of the task log on failure. The exitstatus string is often a summary; the log has the detail. Capturing it at the moment of failure is far more reliable than expecting someone to go looking later.
It sleeps two seconds rather than spinning. A tight poll loop against pvedaemon across a fleet is a self-inflicted load problem.

Stopping a running task

Service impact possibleask a running task to stop— Sends a stop request to the worker. It is a request, not a kill: the worker unwinds at its next safe point, which for a backup means finishing the current chunk. Some tasks leave partial state behind, so check what the operation was before cancelling it.

NODE=pve-01
UPID='UPID:pve-01:00006C88:04C41A02:6A7BD3A3:vzdump:141:root@pam:'

pvesh delete "/nodes/$NODE/tasks/$UPID"

pvesh get "/nodes/$NODE/tasks/$UPID/status" --output-format json | jq -r .exitstatus

A stopped task reports an exitstatus reflecting the interruption rather than OK, which is correct — it did not do what it was asked. Scripts that treat any completed task as done will read a cancelled backup as a successful one.

Under the hood

Where a task actually lives, and why the log outlives the process.

When a state-changing API endpoint is called, pvedaemon forks a worker process to do the work and returns the UPID to the caller straight away. The parent does not wait, which is what makes the API responsive: a migration that takes forty minutes does not hold an HTTP connection open for forty minutes.

The worker writes its output to a file under /var/log/pve/tasks/, in a directory sharded by the last hex digit of the task’s start time — which is why that field is in the UPID at all. Two index files sit alongside: one listing tasks currently running, and one recording finished tasks with their end time and exit status.

That layout explains several behaviours operators otherwise find arbitrary.

Logs survive reboots and the death of the worker. They are ordinary files. A node that crashed mid-backup still has the partial log, and it usually contains the last thing that happened before the crash.

Task history is per node and is rotated. The index is trimmed, so old tasks age out. If a task matters — a change record, an incident timeline — capture the log rather than relying on the UPID still resolving in six months.

A task’s state is derived from the index, not from the process. This is why a task whose worker was killed abruptly can sit in the running list until the daemon reconciles it, and why pstart is in the UPID: the process start time distinguishes a live worker from a recycled PID that happens to match. Without it, a node that has wrapped its PID space could mistake an unrelated process for a task worker.

Cluster-wide task views are aggregations, not a shared log. /cluster/tasks asks each node for its recent tasks and merges the results. A node that is down contributes nothing, which is worth knowing when reconstructing what happened during an incident: the missing node’s history is not lost, it is simply unreachable until it comes back.

One last consequence for automation. Because the UPID contains the node, the worker PID and the process start time, it is globally unique without any coordination between nodes — no shared counter, no lock, no allocation service. That is a deliberate design property of the format and it is why you can safely use a UPID as a key in your own systems: a ticket reference, a database row, an idempotency marker. It will never collide with one from another node.

What happens if a script does not wait, on a cluster under load?

You get a failure that is intermittent, correlates with load, and survives every attempt to reproduce it in a test environment. The mechanism is worth understanding because the same shape recurs in migrations, restores, disk moves and cloning.

The immediate version. A provisioning script creates a VM, then immediately issues qm set to attach a disk. On an idle test cluster the create finishes in under a second and everything works. On a busy production node the create is still running, the configuration file is locked, and the qm set fails with a lock error. The script’s error handling — if it has any — retries, which sometimes works and sometimes races differently. The team concludes the cluster is “flaky under load”.

The worse version. A script starts a VM, does not wait, and proceeds to register it in a service discovery system. The start fails for a reason that has nothing to do with the script — insufficient memory after an HA failover moved other guests onto the node. The registration succeeds regardless, because registration does not check anything. Now traffic is routed to a VM that does not exist, and the outage is attributed to the load balancer.

The one that damages data. A backup script issues vzdump, does not wait, and proceeds to delete the source guest because the backup “completed”. Every element of that sentence is a decision someone made deliberately; the only mistake was believing the returned UPID meant the work was done.

How to tell this is what you are looking at. Three signatures, and any one of them should send you to the task history rather than to the application.

The failure rate correlates with cluster load rather than with any input. The script’s own logs report success at every step. And the task log for the operation, retrieved by UPID, contains a clear error that appears nowhere in the script’s output — which is the confirmation, because it proves the information was available and discarded.

The fix is structural rather than a matter of adding sleeps. A fixed sleep 10 after each operation is the response most teams reach for first, and it fails in both directions: too short under load, and wasting minutes per operation when things are fast. Wait on the actual task, with a timeout, and check exitstatus. It is fifteen lines of shell, it is written once, and it removes an entire category of intermittent production failure.

Common mistakes

Treating the returned UPID as a result. It is a receipt for work that has begun.
Polling status and ignoring exitstatus. A task can be stopped and failed simultaneously. Only OK means success.
Querying a UPID against the wrong node. The node is in the UPID and must match the API path. After a migration this bites.
Discarding the error string. exitstatus frequently names the exact problem; the task log has more.
Polling every 100 ms. The load exceeds the work on a cluster of any size. Two seconds for a start, ten to thirty for a backup.
No timeout, or a timeout that reports the same failure as an error. A caller needs to distinguish “it failed” from “we stopped watching, and it may still be running”.
Treating a cancelled task as complete. Stopping a task produces a non-OK exitstatus for a reason.
Relying on task history for an audit trail. It rotates. Capture the log if the record matters.

Key takeaways

Every state-changing Proxmox operation returns a UPID immediately and does the work asynchronously.
A UPID encodes node, worker PID, process start, start time (hex Unix epoch), task type, task id and user — and is globally unique without any coordination.
Poll /nodes/{node}/tasks/{upid}/status; status says whether it is running, exitstatus says whether it worked, and exitstatus exists only once it has stopped.
exitstatus must be exactly OK. Anything else is the error.
/nodes/{node}/tasks/{upid}/log returns the full output and survives the task; pvenode task list --errors 1 finds failures without a UPID.
/cluster/tasks aggregates across nodes — the right endpoint when you do not know where the work ran.
DELETE /nodes/{node}/tasks/{upid} requests a stop; the worker unwinds at a safe point and reports a non-OK status.
Write one task-waiting wrapper with a timeout and reuse it. Fixed sleeps fail in both directions.

Knowledge check

Why this matters

The asynchronous task model is the single largest conceptual gap between using Proxmox and automating it. Everything else in this part — tokens, Terraform, Ansible, Proxmoxer, webhooks — is a layer over an API whose central property is that it answers before it acts.

That property is not a quirk. It is the reason a forty-minute migration does not hold an HTTP connection open for forty minutes, the reason the GUI can show live progress for an operation started from a script, and the reason the same task appears identically to every interface. The design is right. It just requires the caller to participate.

The reason this deserves a lesson of its own rather than a paragraph is the shape of its failure. Almost every other automation bug announces itself: a non-zero exit code, an exception, a timeout, a stack trace. This one produces no signal at all. The script proceeds, logs success, and the damage appears somewhere else, later, in a system that has no connection to the automation. That combination — silent, delayed, misattributed — is what makes it expensive out of proportion to how simple it is to prevent.

The fix genuinely is fifteen lines of shell, written once. What makes it worth the lesson is knowing why those fifteen lines check two fields instead of one, why they need a timeout with its own exit code, and why they capture the log at the moment of failure rather than trusting anyone to go and look. Every one of those details exists because of a specific production failure that someone spent a night on.

There is also a broader habit worth taking from this. Any API that returns a handle rather than a result has this shape, and modern infrastructure is full of them — cloud provisioning, CI pipelines, Kubernetes controllers, backup services. The question to ask of each is the same: what does this return, when does the work actually finish, and how do I distinguish finished from succeeded? Proxmox at least answers all three clearly, and puts the answer in a string you can decode with cut and date.