Collecting evidence: pvereport, logs and the task model · Proxmox VE

Most Proxmox incidents are diagnosed twice. Once badly, in the moment, with whatever was to hand; and once properly, three days later, from evidence that no longer exists.

The methodology lesson gave you an order to think in. This one is about the raw material: what to capture, from where, and — the part that is nearly always skipped — when.

Evidence has a shelf life

Evidence	Survives a reboot?	Survives a week?
The task log of a failed operation	yes, on disk	usually
`journalctl` output	yes, only if persistent journald is enabled	depends on retention
The current state of a hung process, its stack, its open files	no	no
`dmesg` ring buffer	no — the journal keeps a copy if persistent	no
Ceph health detail at the moment of the fault	no	no
Which storages were active, which mounts existed	no	no
The output of `pvereport`	yes, if you saved it	yes

The rows that say “no” are the reason for this lesson. A node that is behaving strangely right now contains information that a rebooted node does not, and the instinct under pressure is to reboot first.

`pvereport`

pvereport collects the state of a node into one stream: version information, storage configuration and status, network configuration, cluster state, disk and filesystem layout, and guest configuration. It is exposed on the API as /nodes/{node}/report, and in the GUI as Node → Summary → System Report.

Read-only / Safecollect a node report while the fault is present— Read-only. Writes the whole report to a timestamped file in /root. Run it on the affected node, not on a healthy one, and run it before restarting anything. It takes tens of seconds on a busy node because it enumerates storages and guests.

OUT="/root/pvereport-$(hostname)-$(date +%F-%H%M).txt"

pvereport > "$OUT"

wc -l "$OUT"
grep -n '^# ' "$OUT" | head -40

Read-only / Safe

$ grep -n '^# ' /root/pvereport-pve-a2-2026-08-12-0314.txt | head -20

12:# pveversion --verbose
58:# cat /etc/hostname
61:# cat /etc/hosts
72:# cat /etc/network/interfaces
118:# ip -details -statistics address
203:# dmesg
2891:# lscpu
2934:# pvesubscription get
2947:# cat /etc/pve/storage.cfg
2999:# pvesm status
3021:# lsblk --ascii
3078:# zpool status
3115:# pvecm status
3161:# ha-manager status
3190:# qm list

Which daemon answers which question

This is the map worth memorising, because picking the wrong log costs more time than any other mistake in Proxmox troubleshooting.

Symptom	Ask this unit	Why
Web interface unreachable, TLS error, 596	`pveproxy`	It terminates the public API and web UI on 8006
API call fails, task fails to start, permission error	`pvedaemon`	The local API worker that actually executes
Status figures stale or missing in the GUI	`pvestatd`	It collects node, guest and storage status
`/etc/pve` read-only, config write fails	`pve-cluster`	The pmxcfs daemon
Node left or rejoined the cluster, quorum changed	`corosync`	Membership and the totem ring
A guest migrated on its own, a node fenced	`pve-ha-crm`, `pve-ha-lrm`	The cluster and local HA managers
A backup or replication job did not run	`pvescheduler`	It triggers scheduled jobs
A VM died and PVE reacted	`qmeventd`	It watches QEMU process events
SPICE console fails	`spiceproxy`	The SPICE side of console access
Ceph anything	`ceph-mon@`, `ceph-osd@`, `ceph-mgr@`	Plus `ceph -s` and `ceph health detail`

Read-only / Saferead the right log, scoped to the incident window— Read-only. Always scope by time. An unscoped journalctl on a busy node returns tens of thousands of lines and the interesting one is not near either end.

SINCE="2026-08-12 02:50:00"
UNTIL="2026-08-12 03:30:00"

journalctl -u pveproxy --since "$SINCE" --until "$UNTIL" -o short-precise

journalctl -u pve-cluster -u corosync --since "$SINCE" --until "$UNTIL" -o short-precise

journalctl --since "$SINCE" --until "$UNTIL" -p err -o short-precise

The last of those — everything at priority err or above across all units — is the single best first command when you do not yet know which subsystem is at fault. It is short, it is rarely empty during an incident, and it usually names the unit you should have asked.

If the journal is not persistent, half this lesson does not work

Debian’s default journald configuration on some installations keeps the journal in memory only, which means a reboot destroys every log line you would have wanted.

Check it on every node, now rather than during an incident:

journalctl --list-boots shows one entry if the journal is volatile and many if it is persistent. /var/log/journal existing as a directory is the other tell.

Making it persistent is Storage=persistent in /etc/systemd/journald.conf plus a restart of systemd-journald, and then setting a size cap (SystemMaxUse=) so it cannot fill the root filesystem. Part XVI covers aggregation, which is better still — but persistent local journald is the floor, and a node whose journal dies with it is a node you can only ever diagnose live.

Task logs

Every operation PVE performs — a backup, a migration, a start, a snapshot — is a task with a UPID, and each task’s output is written to disk on the node that ran it, under /var/log/pve/tasks/. Part XX covers the UPID structure; what matters here is retrieval.

Read-only / Safefind a task and read its log— Read-only. The API route is the reliable one because it works from any node and understands the task index. Falling back to the filesystem is for when the API is the thing that is broken.

NODE="$(hostname)"

pvesh get "/nodes/$NODE/tasks" --limit 25 --output-format json \
| jq -r '.[] | [.starttime, .type, .id, (.status // "running"), .upid] | @tsv'

UPID="UPID:pve-a2:0000A1B2:0123CDEF:68A9B0C1:vzdump:141:root@pam:"
pvesh get "/nodes/$NODE/tasks/$UPID/log" --output-format json | jq -r '.[].t'

Read-only / Safewhen the API is down, read the task log from disk— Read-only. Task logs live under /var/log/pve/tasks/ on the node that ran the task, indexed by the active/index files. This is the path that still works when pveproxy or pvedaemon is the fault.

ls -lt /var/log/pve/tasks/ | head

grep -rl 'vzdump:141' /var/log/pve/tasks/ 2>/dev/null | head

tail -50 /var/log/pve/tasks/active

Under the hood

Why does Proxmox VE have so many daemons, and why does that help you?

Because each one owns a distinct failure domain, and the separation is what makes the symptom-to-log map above reliable.

pveproxy is a public-facing HTTP server. It terminates TLS, serves the web interface, and forwards API requests. It runs as an unprivileged user and proxies privileged work rather than doing it. So a TLS problem, a certificate expiry, a port conflict or a reverse-proxy misconfiguration lands here and nowhere else.

pvedaemon is the local API worker, listening on localhost only. It executes the request, forks a worker for anything long-running, and that worker writes the task log. So a permission failure, a bad parameter or a failing operation lands here.

pvestatd polls: node status, guest status, storage status. It is why the GUI shows numbers, and it is entirely separate from whether those subsystems work. A hung pvestatd produces a GUI full of question marks over a perfectly healthy cluster — which is one of the most misleading symptoms in the product, because it looks like everything is broken and nothing is.

pve-cluster runs pmxcfs. If it is unhappy, /etc/pve is read-only or absent, and every other daemon starts failing in confusing ways because they all read their configuration from there. That is why pve-cluster and corosync are worth checking early even when the symptom points somewhere else: they are upstream of everything.

The practical technique that falls out of this: when several unrelated things fail at once, look for the shared dependency rather than the most alarming message. Nearly always it is /etc/pve, quorum, or the root filesystem being full.

The support-case package

Whether the case goes to Proxmox, to a partner, or to your own second line, the same package removes a round trip.

Always:

pvereport from the affected node, collected while the fault was present.
pveversion -v from every node — the set of distinct answers matters more than any one of them.
The journal from the incident window, from the affected node: journalctl --since ... --until ... -o short-precise.
The task log of the failing operation, with its UPID, from the node that ran it.

A short narrative, in this shape:

What was expected, and what happened instead.
The exact time the problem started, in a stated timezone.
What changed in the preceding week — upgrades, network changes, hardware, configuration. “Nothing” is almost never true and is worth checking against the drift report and the change record before writing it.
What you have already tried, and what effect each attempt had.
Whether it is reproducible, and the steps if so.

Add where relevant: ceph -s and ceph health detail; zpool status -v; pvecm status and corosync-cfgtool -s from more than one node; ip -s link and ethtool -S for network issues; the guest configuration (qm config <vmid>) and qm showcmd <vmid> --pretty for guest issues.

Do not send screenshots of terminal output when you can send the text, and do not send a report you have not read.

What happens if you reboot the node before collecting anything?

You lose a specific and predictable set of evidence, and it is worth knowing exactly which so you can decide honestly.

Gone permanently: the state of any hung process and its kernel stack; the list of open files and mounts at the time; the contents of the dmesg ring buffer if the journal is not persistent; the memory pressure and slab state; any in-flight task’s live output; the exact set of active storages.

Gone if journald is volatile: everything logged during the incident, which is to say the incident.

Still available: task logs on disk, /etc/pve configuration, PBS backup task history, whatever your external monitoring recorded, and anything you had already written down.

What this means in practice. For a node that is down, reboot away — there is nothing live to preserve. For a node that is up and misbehaving, the two minutes of collection in the callout above is the highest-value time you will spend in the whole incident.

And the honest consequence of skipping it: the incident review concludes “resolved by reboot”, the same fault returns in six weeks, and the second occurrence has no more evidence than the first. That pattern — a recurring, never-diagnosed fault — almost always traces back to a decision made in the first five minutes of the first occurrence.

Key takeaways

Evidence has a shelf life. Process state, mounts, ring buffers and live task output do not survive a reboot; task logs and a saved pvereport do.
pvereport collects versions, network, storage, cluster, disk and guest state into one readable stream, sectioned by the command that produced each part. Run it while the fault is present, and save it to /root.
Review a pvereport before sending it outside the organisation — it describes your infrastructure in detail.
Map the symptom to the daemon: pveproxy for the web interface, pvedaemon for API and task execution, pvestatd for stale GUI figures, pve-cluster for /etc/pve, corosync for membership, pve-ha-* for fencing and automatic migrations, pvescheduler for jobs that did not run.
journalctl --since ... --until ... -p err across all units is the best first command when the subsystem is unknown.
Check that journald is persistent on every node now. A volatile journal means a reboot destroys the incident.
Task logs live under /var/log/pve/tasks/ on the node that ran the task. A migration has two, and the destination’s usually has the real reason. Use /cluster/tasks to find which node to look at.
A support package is pvereport, pveversion -v from every node, the scoped journal, the task log, and a narrative that includes what changed in the preceding week.

Knowledge check

Why this matters

There is a category of incident that recurs for years. It happens every few months, someone restarts something, it goes away, and the ticket is closed. Ask the team what causes it and you get a shrug and a theory.

Almost every one of those traces back to the first occurrence, where the decision was made — reasonably, under pressure — to restore service before capturing anything. That decision is correct in the moment and expensive in aggregate, because the fault now has a permanent supply of chances to happen again and no supply of evidence.

The fix is not heroism. It is two minutes and a habit: pvereport, the journal, and the four status commands, written to /root before the restart. Nothing in that sequence requires knowing what is wrong, which is exactly why it works at 3 a.m.

Collect first. You can always throw the evidence away; you cannot go back and take it.