Pools, delegation and the limits of tenant boundaries · Proxmox VE

Splitting into more clusters is one answer to “several groups of people share this platform”. The other is to share one cluster properly, which is cheaper and is the right answer far more often than the cluster-per-tenant instinct suggests.

Proxmox VE’s tool for that is the resource pool. It is genuinely useful and it is routinely oversold, so this lesson does two things: builds a delegation that works, and then states plainly what pools do not do.

What a pool is

The documentation is precise and short. A pool is “a set of virtual machines, containers, and storage devices” that exists to simplify permission handling: rather than setting permissions on each resource, you set them on /pool/{poolid}, and they are inherited by all pool members.

That is the whole feature. A pool is:

A membership list of guests and storages.
An ACL path — /pool/<name> — that permissions can attach to.
A GUI grouping, so a tenant’s view is a folder rather than a search.

Proxmox VE does not support per-pool resource limits or quotas

This is the single most important fact in the lesson and it comes straight from the User Management chapter: Proxmox VE does not support per-pool resource limits or quotas, and pools function exclusively as a permission grouping mechanism.

There is no “this pool may use 200 GB of RAM”, no “this pool may allocate 4 TB”, and no admission control that refuses a guest creation because a pool is at its limit. A tenant with VM.Allocate and Datastore.AllocateSpace can create guests until the cluster runs out, not until their pool does.

If you have promised a customer a quota, you are enforcing it somewhere else — in a portal in front of PVE, in a periodic report with a human consequence, or by giving them dedicated storage. Do not discover this after signing.

Building a delegated tenant

The working pattern is a group, a role, a pool, and ACLs on the pool path plus the storages the tenant is allowed to consume.

Configuration changecreate the pool and its group— Changes access control and cluster configuration. Creates an empty pool and a group to hold the tenant's users. Nothing is delegated yet.

POOL="tenant-blue"

pveum pool add "$POOL" --comment "Blue team: delegated self-service"
pveum group add "$POOL-admins" --comment "Administrators for the blue tenant"

Configuration changea purpose-built tenant role— Changes access control. PVEVMAdmin is close to what a self-service tenant needs but grants more than most estates intend. Building the role explicitly means the privilege list is a document you can review. Note what is absent: VM.Migrate, VM.Config.HWType, Sys.* and anything under Permissions.

pveum role add TenantSelfService --privs "VM.Allocate VM.Audit VM.Clone VM.Config.CDROM VM.Config.Cloudinit VM.Config.CPU VM.Config.Disk VM.Config.Memory VM.Config.Network VM.Config.Options VM.Console VM.PowerMgmt VM.Snapshot VM.Snapshot.Rollback VM.Backup Datastore.Audit Datastore.AllocateSpace Pool.Audit SDN.Use"

Configuration changegrant the role on the pool, and only the storages the tenant may use— Changes access control. Two grants: the pool path covers every current and future member guest, and the storage paths are what allow disk allocation. Grant the storage narrowly — a tenant with Datastore.AllocateSpace on / can fill every storage in the cluster.

POOL="tenant-blue"
TENANT_STORAGE="blue-rbd"

pveum acl modify "/pool/$POOL" --groups "$POOL-admins" --roles TenantSelfService
pveum acl modify "/storage/$TENANT_STORAGE" --groups "$POOL-admins" --roles TenantSelfService

pveum acl list

Configuration changeput resources into the pool— Changes cluster configuration. --vms and --storage take comma-separated lists; --delete on the same command removes the listed members instead of adding them. A guest can belong to at most one pool.

POOL="tenant-blue"

pveum pool modify "$POOL" --vms 141,142,205 --storage blue-rbd

pveum pool list --poolid "$POOL"

Read-only / Safe

$ pveum pool list --poolid tenant-blue

┌──────────┬───────┬─────────┬──────────┬─────────┐
│ type     │ vmid  │ node    │ storage  │ status  │
╞══════════╪═══════╪═════════╪══════════╪═════════╡
│ qemu     │   141 │ pve-a2  │          │ running │
│ qemu     │   142 │ pve-a1  │          │ running │
│ lxc      │   205 │ pve-a3  │          │ running │
│ storage  │       │ pve-a1  │ blue-rbd │         │
└──────────┴───────┴─────────┴──────────┴─────────┘

One pool per guest, so use tags for the other dimensions

Pool membership is exclusive: a guest is in one pool or none. That is a good fit for tenancy, which is genuinely exclusive, and a bad fit for every other way you might want to slice the estate — cost centre, environment, application, change window.

Tags are the mechanism for those, and datacenter.cfg has controls for them: tag-style governs display and ordering, registered-tags names tags that require Sys.Modify on / to set or delete, and user-tag-access controls what tags ordinary users may apply — none, list, existing or free.

The useful combination for a shared cluster is: pool for tenancy and authorisation, registered tags for anything that drives billing or policy so tenants cannot set them, and free-form tags for whatever the tenants find useful. Part XXII’s inventory and chargeback material builds on exactly this.

The `allow-move` question

pveum pool modify takes --allow-move, defaulting to 0. It governs whether a resource already belonging to another pool may be moved into this one by the same call.

Leaving it at the default is the safer choice and it is worth understanding why. A guest silently changing pools changes which ACLs apply to it. With --allow-move enabled and a scripted pool assignment, a mistake in a VMID list does not error — it moves somebody else’s guest into your tenant’s pool, and with it, access.

Enable it deliberately, for a specific reorganisation, and turn it off again.

What pools do not isolate

This is the section to read to a stakeholder before agreeing to a multi-tenant design. A pool is an authorisation boundary. It is not any of the following.

Not isolated	What actually happens	Where the real control lives
CPU	Tenant A’s guests compete with tenant B’s for host cores	Per-guest `cores`, `cpulimit` and `cpuunits`; node placement (Part XVII)
Memory	Overcommit and ballooning are node-wide; one tenant’s pressure is everyone’s	Per-guest memory, no overcommit for shared clusters, node separation
Storage capacity	A shared storage is filled by whoever gets there first	Separate storages per tenant, monitored per storage
Storage IOPS	A noisy tenant degrades the shared backend	Per-disk `mbps`/`iops` limits, separate backends
Network	Guests on the same bridge/VLAN can reach each other regardless of pool	SDN zones and firewall — the next lesson
The node	A tenant with `Sys.Console` or node-level access sees everything on that node	Do not grant node-level privileges to tenants
Cluster visibility	`/cluster/resources` filtered by permission still reveals node names and structure	Accept it, or use separate clusters
Backups	A cluster backup job is cluster configuration, not tenant configuration	Per-pool backup jobs exist as a job selection mode; retention is still yours
The blast radius	A pmxcfs, quorum or upgrade event hits every tenant at once	Separate clusters — Part XXVII lesson 1

The pool that was supposed to be a quota

A hosting team sold “8 vCPU, 32 GB RAM, 2 TB storage” tiers and implemented them as pools, reasoning that the pool was the tenant’s boundary.

For eight months it worked, because tenants provisioned roughly what they had bought. Then one tenant automated their build pipeline, and their VM.Allocate right did exactly what it says: they created guests. Nothing refused them. The shared RBD pool went from 60% to 94% over a weekend, and the first anyone knew was a Ceph nearfull warning on Monday.

The diagnosis people reach for first is a Ceph capacity problem. It is not: Ceph did precisely what it was told, and the alerting worked. The defect was architectural — a limit had been sold and never implemented, because the product does not implement it and nobody had read the sentence that says so.

The remediation was not clever. Separate storages per tenant, sized to the tier and monitored individually, so that a tenant filling their allocation is a tenant-scoped event with a tenant-scoped alert. That is what “2 TB” had meant to the customer all along.

Under the hood

Why do pools carry permissions but not limits?

Look at where a pool lives. It is a small membership record in cluster configuration, and /pool/<name> is a node in the ACL path tree — the same tree that holds /nodes/<node>, /vms/<vmid>, /storage/<id> and /sdn/zones/<zone>. Permission resolution walks that tree, with deeper levels replacing inherited ones and user permissions replacing group permissions. Adding a pool to the tree is therefore free: it is one more path with members.

A quota is a completely different kind of object. Enforcing “this pool may allocate at most 2 TB” means every allocation path — guest creation, disk add, disk resize, clone, restore, migration inbound, snapshot on a thick backend — must consult a running total, and that total must be correct under concurrency across every node in the cluster. It is admission control, and admission control has to be transactional and complete or it is worse than nothing, because it gives an assurance it sometimes fails to keep.

Proxmox’s choice is to expose the accounting and leave the enforcement to you. pvesh get /pools/<poolid> and /cluster/resources give you consumption per pool; pvesm status gives you capacity per storage. Building a report from those is a small job, and a report with a human consequence attached is a real control. What you cannot build from outside is admission control, which is why the durable answer for a hard limit is a dedicated storage — a boundary the storage layer itself enforces.

Choosing the boundary for a tenant

Tenant looks like	Right boundary	Because
A team inside your organisation, cooperative, shared change window	Pool + SDN zone on a shared cluster	Cheapest to run; one upgrade, one backup regime, one on-call
A team that needs guaranteed capacity	Pool + dedicated storage, and node placement or HA groups for compute	Capacity is enforceable at the storage layer; compute is enforceable by placement
A customer with a contract, a change freeze, or a compliance boundary	Separate cluster	The upgrade unit and the blast radius are contractual, and neither is delegable inside a cluster
Tenants who must not be able to reach each other’s control plane at all	Separate cluster	A cluster’s nodes trust each other; there is no partial trust
A cost centre or a project	Tags, not pools	Pools are exclusive; a guest has one tenant but many attributes

A tenant onboarding checklist

Run through this every time. Steps 5 and 8 are the ones that get skipped and they are the ones that cost money.

Pool created, named for the tenant, with a comment naming the owner.
Group created and the tenant’s users added to it, from the realm they authenticate against — not local PVE accounts if an LDAP/AD realm exists.
Role reviewed, not copied. Read the privilege list aloud. VM.Migrate, Sys.*, Permissions.Modify and Realm.* should not be there.
ACLs granted on /pool/<name> and on the tenant’s storages only. Check with pveum user permissions <user> --path / that nothing broader applies.
Storage decided: shared or dedicated. If the tenant has been promised a capacity number, it is dedicated. Record which.
Network decided: which SDN zone or VLAN, and confirm it is not the one another tenant is on. Next lesson.
Backup job created for the pool, with retention that matches what the tenant has been told, and verified to have run once.
Monitoring and alerting scoped: a per-storage capacity alert if the storage is dedicated, and a per-pool consumption report if it is not. Without this, “how much are they using” is a question nobody can answer until it is urgent.
Offboarding written down at onboarding time: which pool, which storages, which group, which ACL entries, which backup job, which SDN objects. A tenant who leaves without this list leaves residue for years.

What happens if a tenant fills the shared storage?

Everyone stops. This is the failure mode a pool cannot prevent and it is worth walking through, because the response order matters.

Immediately. Guests on the affected storage begin failing writes. On a thin backend that means guests pausing or their filesystems going read-only; on Ceph it means the pool passing nearfull and eventually full, at which point writes block cluster-wide for that pool. Backups to the same storage fail. This is not tenant-scoped: every tenant on that storage is affected, including the ones who did nothing.

The first diagnosis is usually wrong. The alert says storage, so the instinct is a storage fault — a failed OSD, a stuck resilver, a snapshot that did not clean up. Check consumption by owner before you check hardware: pvesm status for the headline, then pvesm list <storage> to see the volumes and their sizes, and /cluster/resources to attribute guests to pools.

Do not free space by deleting the newest thing you find. Snapshots and backups are the usual quick wins and both are somebody’s recovery point. Free space in this order: orphaned volumes belonging to no guest, then expired backups under an agreed retention, then — with the owner’s explicit agreement — tenant data.

Then fix the class of problem, not the instance. If a capacity promise exists, move that tenant to a dedicated storage sized to it. If no promise exists, say so to the tenants in writing, and put a per-storage alert at a level that gives you days rather than hours: 75% is a conversation, 85% is a change, 95% is an incident.

The uncomfortable summary: on a shared storage, every tenant’s headroom is every other tenant’s discipline. That is an acceptable arrangement between cooperative teams who know it, and an unacceptable one between customers who were told otherwise.

Key takeaways

A pool is a membership list of guests and storages plus an ACL path /pool/<name>. Permissions set there are inherited by all members.
Proxmox VE does not support per-pool resource limits or quotas. Pools are exclusively a permission grouping mechanism. Any quota you have promised is enforced outside PVE or by dedicated storage.
Build the tenant role explicitly rather than reusing PVEVMAdmin, and grant it on /pool/<name> plus only the storages the tenant may consume.
pveum pool modify --vms --storage manages membership; --delete removes the listed members; --allow-move (default 0) should stay off except during a deliberate reorganisation.
A guest belongs to at most one pool. Use tags — with registered-tags and user-tag-access in datacenter.cfg — for cost centre, environment and application.
Pools do not isolate CPU, memory, storage capacity, IOPS, network reachability, node access or blast radius. Each of those has a different control, and two of them only have one: separate storage, and separate clusters.
Choose pool plus SDN zone for cooperative internal teams; pool plus dedicated storage where capacity is promised; a separate cluster where the contract, the change freeze or the trust boundary is real.

Knowledge check

Why this matters

Multi-tenancy on Proxmox VE works. Hosting providers run it at scale, and the pieces — pools, roles, ACL paths, SDN zones, firewall — are sound.

What does not work is inferring capabilities from a feature’s name. “Resource pool” sounds like a resource allocation. It is a permission grouping with a membership list, and the documentation says so in one sentence that is easy to read past.

The habit worth taking from this lesson is broader than pools: before you build a boundary into a commercial promise, find the sentence in the documentation that describes what the boundary enforces. If there is no such sentence, the boundary is organisational, not technical, and it will hold exactly as long as everyone behaves.

Sometimes that is fine. Between two internal teams who both understand the arrangement, a shared storage and a pool are the right engineering — cheap, simple, and honest about what they are. The failure is not sharing. It is sharing while telling someone you are not.