When the GUI or the API is the problem

“The cluster is down” is, four times out of five, “I cannot reach the web interface”. Those are very different statements, and the first thing this lesson buys you is the ability to tell somebody which one is true within thirty seconds.

The four daemons and one port

The shape to remember: pveproxy is a thin, unprivileged front door that proxies to pvedaemon, which does the work. Almost every symptom in this lesson is explained by which of the two is failing.

Cannot reach the interface

Work outward from the node. Four causes, four tests, in the order that costs least.

systemctl status pveproxy pvedaemon pvestatd pve-cluster --no-pager

ss -ltnp | grep -E ':8006|:3128'

journalctl -u pveproxy -n 50 --no-pager -o short-precise

pvesh get /version

pvesh get /nodes --output-format json | jq -r '.[] | [.node, .status] | @tsv'

$ ss -ltnp | grep :8006

LISTEN 0      128          0.0.0.0:8006       0.0.0.0:*    users:(("pveproxy worker",pid=2431,fd=6),("pveproxy",pid=2428,fd=6))

Cause one: the daemon is not running

Usually because it failed to start, and usually because of a certificate. The journal names it. Restarting is safe:

systemctl restart pve-cluster
systemctl restart pvedaemon
systemctl restart pvestatd
systemctl restart pveproxy

systemctl status pveproxy --no-pager

Cause two: access control in /etc/default/pveproxy

pveproxy supports host-based access control with ALLOW_FROM, DENY_FROM and a POLICY of allow or deny, and LISTEN_IP to bind to a specific address.

These are excellent controls and they are a classic self-inflicted outage: a management network is renumbered, ALLOW_FROM still names the old range, and the interface becomes unreachable from everywhere at once — including from the machine of the person who changed it.

cat /etc/default/pveproxy 2>/dev/null || echo "no /etc/default/pveproxy - using defaults"

Cause three: the certificate

pveproxy uses /etc/pve/local/pveproxy-ssl.pem and /etc/pve/local/pveproxy-ssl.key if present, and falls back to /etc/pve/local/pve-ssl.pem and /etc/pve/local/pve-ssl.key.

That two-step fallback explains most certificate confusion in Proxmox:

• The pveproxy-ssl.* pair is where a custom or ACME certificate goes.
• The pve-ssl.* pair is the node’s cluster-CA-signed certificate, managed by PVE.
• If a custom certificate is broken, PVE does not silently fall back — the fallback applies when the file is absent, not when it is invalid.

ls -l /etc/pve/local/pveproxy-ssl.pem /etc/pve/local/pve-ssl.pem 2>&1

for f in /etc/pve/local/pveproxy-ssl.pem /etc/pve/local/pve-ssl.pem; do
[ -f "$f" ] && { echo "== $f"; openssl x509 -in "$f" -noout -subject -issuer -dates; }
done

echo | openssl s_client -connect 127.0.0.1:8006 -servername "$(hostname -f)" 2>/dev/null \
| openssl x509 -noout -subject -issuer -dates

mv /etc/pve/local/pveproxy-ssl.pem "/root/pveproxy-ssl.pem.$(date +%F)"
mv /etc/pve/local/pveproxy-ssl.key "/root/pveproxy-ssl.key.$(date +%F)"

systemctl restart pveproxy

echo | openssl s_client -connect 127.0.0.1:8006 2>/dev/null | openssl x509 -noout -issuer -dates

Cause four: the network, or the other end

Firewall rules — the PVE firewall’s own management IPSet, or something upstream — a proxy in front, a changed DNS record, or a client on a network that was never permitted. ss on the node plus a curl from the client’s network settles it quickly.

One dead node makes the whole GUI slow

This is the symptom that reads as a cluster-wide failure and is not.

The web interface presents the whole cluster. To do that, the pveproxy you are connected to must gather information from every node — and a node that is unreachable does not refuse quickly; it times out. The result is a resource tree that takes tens of seconds to populate, panels that spin, and an interface that feels broken on every node, because every node is waiting for the same missing peer.

pvecm status

pvecm nodes

pvesh get /nodes --output-format json | jq -r '.[] | [.node, .status, (.uptime // 0)] | @tsv'

for n in $(pvesh get /nodes --output-format json | jq -r '.[].node'); do
printf '%s ' "$n"
timeout 3 bash -c "echo > /dev/tcp/${n}/8006" 2>/dev/null && echo reachable || echo UNREACHABLE
done

Key takeaways

• Running guests do not depend on pveproxy, pvedaemon, pvestatd or spiceproxy. Say so early in the incident; it changes the severity.
• pveproxy serves the API and web interface on TCP 8006 over HTTPS, runs as www-data with very limited permissions, and forwards privileged work to pvedaemon, which listens on localhost only.
• pvesh get /version on the node splits the problem in half: if it works, the API is healthy and the fault is pveproxy, TLS, access control or the network.
• /etc/default/pveproxy holds ALLOW_FROM, DENY_FROM, POLICY and LISTEN_IP. A renumbered management network plus a stale ALLOW_FROM locks everyone out at once.
• Certificates: pveproxy-ssl.pem/.key if present, falling back to pve-ssl.pem/.key. The fallback applies when the file is absent, not when it is invalid — so move a broken custom certificate aside to recover.
• pvecm updatecerts regenerates a node certificate from the cluster CA.
• A GUI that is slow on every node is a membership problem, not a GUI problem. One unreachable node makes every node’s interface wait on timeouts.
• Never weaken TLS, remove access control, or copy certificates between nodes as a workaround.
• Have out-of-band console access and a local administrative account that does not depend on an external realm, arranged before you need them.

Knowledge check