Proxmox VE architecture and daemons

Why this matters in production

When something breaks, you triage by understanding which subsystem owns which responsibility. “Is the cluster filesystem stale?”, “Is the API daemon alive?”, “Is the QEMU process wedged?” — these are different questions, addressed by different daemons, with different log locations.

Mental model

A Proxmox node is a Debian system with a set of cooperating daemons layered on top. The GUI, CLI, and API all funnel through the same internal services.

flowchart TB
  subgraph CLIENT[Clients]
    GUI[Web GUI]
    CLI[qm / pct / pvesh]
    API[REST API]
  end
  subgraph MANAGEMENT[Management stack]
    PROXY[pveproxy + pveapi]
    DAEMON[pvedaemon]
    WORKER[pvefw — firewall worker]
  end
  subgraph CORE[Core cluster]
    PMXCFS[pmxcfs — cluster filesystem]
    STATUS[pvestatd — status collector]
    SCHED[pvescheduler]
  end
  subgraph HYPERVISOR[Hypervisor]
    QEMU[QEMU/KVM]
    LXC[LXC runtime]
  end
  subgraph OPTIONAL[Optional]
    HA[HA: pve-ha-crm + pve-ha-lrm]
    CEPH[pveceph services]
    BACKUP[vzdump / PBS client]
  end

  CLIENT --> PROXY
  CLI --> DAEMON
  API --> PROXY
  PROXY --> DAEMON
  DAEMON --> PMXCFS
  DAEMON --> QEMU
  DAEMON --> LXC
  DAEMON --> HA
  DAEMON --> CEPH
  STATUS --> PMXCFS
  SCHED --> DAEMON
  QEMU --> PMXCFS

The core daemons

pmxcfs — the cluster filesystem

The most distinctive piece of Proxmox architecture. pmxcfs is a FUSE filesystem mounted at /etc/pve/. It replicates configuration files across the cluster using Corosync, provides distributed locking, and becomes read-only if quorum is lost.

flowchart LR
  N1[pve-01 /etc/pve] --> C[Corosync]
  N2[pve-02 /etc/pve] --> C
  N3[pve-03 /etc/pve] --> C

Key files under /etc/pve/:

How a “Start VM” operation flows

sequenceDiagram
  participant User
  participant GUI as Web GUI
  participant Proxy as pveproxy
  participant Daemon as pvedaemon
  participant Cfs as pmxcfs
  participant Qemu as QEMU
  participant Stat as pvestatd

  User->>GUI: click "Start"
  GUI->>Proxy: POST /api2/json/nodes/pve-02/qemu/100/status/start
  Proxy->>Daemon: dispatch over local socket
  Daemon->>Cfs: write lockfile, update status
  Daemon->>Qemu: qm start (fork QEMU process)
  Qemu-->>Daemon: PID, monitor socket
  Daemon->>Cfs: release lock, update status
  Stat->>Cfs: periodic status updates

Important files (continued)

CLI walkthrough

systemctl list-units 'pve*' 'spiceproxy*' --no-pager

pmxcfs-tool status

ls -la /etc/pve/

cat /var/log/pve/tasks/active 2>/dev/null || ls /var/log/pve/tasks/

Production considerations

• Daemon health is observable via systemd. A node with degraded daemons will surface in the GUI dashboard. Do not silence alerts about service restarts.
• pmxcfs is the thing to inspect first when “the GUI is wrong.” Either a config change did not propagate (check the file), or the daemon is wedged (check journalctl).
• pvestatd writes can be heavy on large clusters. It refreshes VM status every few seconds. On very large clusters (1000+ VMs), consider tuning the cache TTLs in /etc/pve/datacenter.cfg.

Common mistakes

• Trying to “fix” a cluster-wide change by editing /etc/pve/foo.cfg on one node and not seeing the change on another. The change does propagate — but if pmxcfs has lost sync with that node, it will lag. Check pmxcfs-tool status.
• Hunting for logs in /var/log/syslog. Proxmox is opinionated about where logs go; check the table above.
• Treating /etc/pve/ as “just a directory.” It is a distributed filesystem with strong consistency semantics.

Key takeaways

• Proxmox is a set of cooperating systemd services on top of Debian. Knowing which daemon owns what operation is the difference between a 5-minute triage and a 2-hour triage.
• pmxcfs is the cluster filesystem that holds configuration. It is read-only on quorum loss.
• Logs are in journalctl -u <unit> and under /var/log/pve/.

Knowledge check