API tokens, privilege separation and secret rotation · Proxmox VE

An API token is a credential that authenticates as a user without that user’s password, without a login, and without any possibility of a second factor. That last point is the one that determines how tokens should be treated: a token is the one credential in Proxmox that TFA cannot protect, because there is no interactive session in which to challenge anybody.

Which makes scope and lifetime the only controls available. Both are optional, both default to the permissive answer, and neither leaves a trace when it is skipped.

Anatomy

A token belongs to a user. Its full identifier is three parts:

alice@corp!terraform

user, !, token ID. The secret is a UUID, shown once, and the credential sent on the wire is the two joined:

Authorization: PVEAPIToken=alice@corp!terraform=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

Creating one

Configuration changea scoped, separated, expiring token— Creates the token and prints the secret once. Privilege separation is the default; the expiry is not, and is the only thing that bounds the credential lifetime without human intervention.

EXPIRY=$(date -d '+90 days' +%s)

pveum user token add svc-backup@pve nightly \
--privsep 1 \
--expire "$EXPIRY" \
--comment 'PBS nightly job, owner platform-team, rotate quarterly'

Configuration changethe one time the secret appears— Illustrative. The value shown here is a placeholder; a real token secret is a UUID. Capture it into your secret store from this output or not at all.

# pveum user token add svc-backup@pve nightly --privsep 1 --expire 1786500000

┌──────────────┬──────────────────────────────────────┐
│ key          │ value                                │
╞══════════════╪══════════════════════════════════════╡
│ full-tokenid │ svc-backup@pve!nightly               │
├──────────────┼──────────────────────────────────────┤
│ info         │ {"privsep":"1","expire":"1786500000"}│
├──────────────┼──────────────────────────────────────┤
│ value        │ REPLACE_ME-token-secret-uuid         │
└──────────────┴──────────────────────────────────────┘

Illustrative output

The --comment is not decoration. A token with no comment is a credential with no owner, and the first question in every access review is who uses this and what breaks if it goes away. Put the owning team and the rotation cadence in it.

Privilege separation

--privsep 1 is the default, and it is the difference between a token that does one job and a token that is a copy of a person.

From the documentation, the two modes:

Separated privileges (default): the token needs to be given explicit access with ACLs. Its effective permissions are calculated by intersecting user and token permissions.

Full privileges: the token’s permissions are identical to that of the associated user.

The intersection is the important word. A separated token’s effective permission on a path is the smaller of what the user has there and what the token has there. Two consequences, and they pull in opposite directions:

A separated token can never exceed its user. “Privilege separated tokens can never have permissions on any given path that their associated user does not have.” Granting the token a role on a path the user cannot reach produces an ACL entry that resolves to nothing at all.

A separated token with no ACL entries can do nothing. It authenticates successfully and every subsequent call returns a permission error. This is the single most common “my token does not work” report, and it is the system behaving correctly: creating the token and granting the token are two operations.

Configuration changecreating a token is not granting it— Two ACL entries scoped to the two subtrees the job touches. Without these, the separated token created above authenticates and is refused by every endpoint.

pveum acl modify /pool/production --tokens 'svc-backup@pve!nightly' \
--roles BackupRunner
pveum acl modify /storage/pbs-main --tokens 'svc-backup@pve!nightly' \
--roles BackupTarget

Auditing what a token can do

The ACL list cannot answer this, because the intersection is not written anywhere. Ask:

Read-only / Safethe only honest answer— Read-only. Resolves the user permissions, the token ACLs and the intersection into the effective permission set.

pveum user token list svc-backup@pve
pveum user token permissions svc-backup@pve nightly

Run this on every token during an access review and compare the result against the comment describing what the token is for. The recurring finding in production estates is a token holding more privilege than its purpose, and this pair of commands is what surfaces it in seconds.

Expiry

--expire takes a Unix timestamp. After it passes, the token stops authenticating.

That is a blunt instrument and it is still worth using, because the alternative is a credential with no end. The failure mode is honest: the job breaks, loudly, on a date you chose, rather than the credential outliving the person who created it and the system it was for.

Two habits make it survivable:

Alert before the date, not on it. Nothing in Proxmox warns you that a token is about to expire. Export the expiry into whatever tracks certificate renewals — it is the same class of problem and the same consequence.

Give the expiry a value that matches the job. A token for a two-week migration gets a two-week expiry. A permanent backup job gets a quarterly one aligned with the rotation you were going to do anyway.

Where the secret ends up

The token is created once and then copied, and every copy is a place it can leak. The realistic inventory for a Proxmox estate:

Where	Exposure	What to do
Terraform `provider` block	Committed to git, forever, in history	Environment variable or a secret backend; never a `.tf` file
Terraform state file	Plaintext in state, including remote state	Encrypted remote state with restricted access
Ansible inventory or vars	Committed unless vaulted	`ansible-vault`, or lookup from the secret store
CI/CD variable	Visible to anyone who can edit the pipeline	Masked, protected, scoped to one project
Monitoring agent config	World-readable config files are common	`chmod 600`, dedicated user, verify after every deploy
Shell history	`~/.bash_history` on a shared jump host	Never paste a secret onto a command line
Backup of any of the above	Multiplies every row above	Note it in the rotation runbook
A screenshot in a ticket	Permanent, searchable, unrevoked	The reason the value goes to the secret store, not the terminal

Configuration changepass a token to a tool without putting it on the command line— Reads the secret from a file with restricted permissions into the environment for one command. Keeps the value out of shell history and out of the process list, where a command-line argument would be visible to any user on the host.

install -m 600 /dev/null /etc/pve-automation/token.env
# write the secret into that file with an editor, not with echo

set -a
. /etc/pve-automation/token.env
set +a

terraform plan

The process-list detail is easy to miss and worth stating: an argument passed on a command line is visible in ps to every user on the host for as long as the process runs. An environment variable is not, and a file read at startup is not.

Rotating without an outage

The naive rotation — delete the token, create a new one, update the consumer — has an outage in the middle of it and no way back if the new credential is wrong. The overlap procedure has neither.

A token ID is part of the identifier, so a user can hold two tokens at once with different IDs and different secrets. That is the whole trick.

Configuration changestep 1 — create the replacement alongside the current one— Adds a second token to the same user. The existing token is untouched and every consumer keeps working.

EXPIRY=$(date -d '+90 days' +%s)
pveum user token add svc-backup@pve nightly-2026q4 \
--privsep 1 --expire "$EXPIRY" \
--comment 'PBS nightly, rotation of nightly-2026q3, owner platform-team'

Configuration changestep 2 — grant it the same ACLs, then prove they match— A separated token starts with no permissions. The comparison at the end is the step people skip, and it is the one that catches a mistyped path before the cutover.

pveum acl modify /pool/production --tokens 'svc-backup@pve!nightly-2026q4' \
--roles BackupRunner
pveum acl modify /storage/pbs-main --tokens 'svc-backup@pve!nightly-2026q4' \
--roles BackupTarget

pveum user token permissions svc-backup@pve nightly-2026q3 > /tmp/tok-old.txt
pveum user token permissions svc-backup@pve nightly-2026q4 > /tmp/tok-new.txt
diff /tmp/tok-old.txt /tmp/tok-new.txt && echo 'permissions match'

Step 3 — update the consumer to the new token and let it run one full cycle. For a nightly backup that means one night. Do not skip the cycle: a permission that is only exercised during the job, such as writing to a second datastore on the monthly run, will not fail until the monthly run.

Step 4 — confirm the old token is idle before removing it. Task history is the evidence, and it is worth checking rather than assuming you found every consumer:

Read-only / Safestep 4 — is anything still using the old token?— Read-only. Filters recent cluster tasks by the old token identifier. An empty result over a full cycle is the go signal for removal.

pvesh get /cluster/tasks --output-format json \
| grep -c 'nightly-2026q3' || echo 'no recent tasks under the old token'

Service impact possiblestep 5 — remove the old token— Deletes the credential. Any consumer still using it fails immediately with an authentication error. Do this only after a full cycle with no observed use.

pveum user token remove svc-backup@pve nightly-2026q3

The rollback at every step before 5 is to point the consumer back at the old token, which still exists and still works.

Revocation under compromise

Rotation is planned and reversible. Revocation is neither, and the order of operations is different: remove the access first, investigate second.

Service impact possiblerevoke a compromised token— Immediate and unconditional. Every consumer of this token fails on its next call, including legitimate ones. That is the correct trade when the credential is known to be exposed.

pveum user token remove svc-backup@pve nightly

Then, in order: list what the token could reach, so the investigation has a scope; read the task log for anything it did; and check whether the same secret was reused anywhere else, because a secret in a git history is usually in more than one repository.

If the token was --privsep 0 on a privileged user, the blast radius is the user’s entire permission set and the incident is larger than the token.

Under the hood

Where tokens live, and why removal is instant across the cluster.

Token definitions are records in /etc/pve/user.cfg, the same pmxcfs file that holds users, groups, roles and ACLs. The secret itself is stored separately under /etc/pve/priv/, hashed rather than in plain text, which is why it genuinely cannot be retrieved after creation — the value is not kept anywhere to retrieve.

pmxcfs replicates both to every node synchronously. A token removed on node 1 is refused by node 3’s pveproxy on the next request; there is no propagation delay and no cache to expire. That is exactly the property you want during an incident, and it is worth knowing that the same property applies to the ACL changes you make while responding.

Two behaviours follow from tokens being an authentication mechanism rather than a session mechanism.

There is no token session and nothing to expire. Every API request carrying a token header is authenticated and authorised from scratch against user.cfg. Removing the token or its ACLs takes effect on the next request — unlike a ticket-based session, which remains valid until the ticket expires. Revoking a token is therefore more immediate than disabling the user who owns it.

Token authentication never touches the realm. A token owned by alice@corp is validated against user.cfg alone; the LDAP directory or OIDC provider behind corp is not contacted. Two consequences: tokens keep working while the directory is down, which is occasionally useful and generally a surprise; and disabling the user in the directory does not disable their tokens. Offboarding has to remove the tokens explicitly, and the sync job will not do it for you — remove-vanished acts on ACLs and user entries, not on the credential.

The token’s own ACL entries are ordinary ACL records with the token as the principal, which is why pveum acl list shows them and why they follow every rule from the ACL model lesson, including replacement at deeper paths. The intersection with the user’s permissions is computed at check time and is not stored anywhere — hence pveum user token permissions as the only way to see it.

What happens if a token is committed to a public repository?

Treat it as used, not as at risk. Public repositories are scraped for credentials continuously and automatically, and the window between a push and the first attempt is measured in minutes.

Remove the token first. Not after you have worked out what happened, not after the change ticket — the credential is public, and every minute of investigation is a minute it still works. pveum user token remove takes effect cluster-wide on the next request.

Then establish the blast radius, and note that you cannot do it after the removal in the way you might expect: pveum user token permissions requires the token to exist. Capture that output before deletion if you have thirty seconds, or reconstruct it from the ACL entries and the owning user’s permissions afterwards.

Then read the task log for the exposure window. The Proxmox task history records the principal that initiated each task, so tasks attributed to that token that your automation did not schedule are the actionable finding. Absence of unexplained tasks is weak evidence rather than proof: a read-only enumeration of the cluster leaves far less trace than a create or delete.

The two hard parts come next, and they are both about scope beyond Proxmox. Rewriting git history does not revoke anything. The commit is in forks, in clones, in CI caches, in the platform’s own event history. The credential is gone only because you removed it in Proxmox. And the same secret is rarely in one place: check whether the value was copied into a CI variable, a monitoring agent config, or a colleague’s local environment file, because those copies keep failing until somebody updates them, and each failure is a service that stopped.

The preventive control that actually works here is not care. It is a pre-commit secret scanner plus tokens that expire, so that the worst case is bounded even when care fails.

Common mistakes

--privsep 0 because the separated token returned a permission error. The error meant the token had no ACL. Grant it one.
Tokens owned by root@pam. Create a purpose-built user; a token cannot be narrower than its owner.
No --expire. A credential with no end date outlives its purpose, its owner and its documentation.
No --comment. An unattributed credential cannot be reviewed, because nobody will delete something they cannot prove is unused.
Auditing the ACL list instead of pveum user token permissions. The intersection is not written down anywhere else.
Rotating by deleting first. Create, grant, cut over, verify, then delete.
Assuming offboarding removes tokens. It does not. The realm sync does not touch them and disabling the directory account does not either.

Key takeaways

A token is the only Proxmox credential that cannot carry a second factor. Scope and expiry are the controls that remain.
The secret is shown once and is stored hashed. There is no recovery.
--privsep 1 means effective permissions are the intersection of user and token. A separated token with no ACL can do nothing, and that is correct behaviour.
pveum user token permissions is the only place the intersection is visible.
Rotate with overlap: create, grant, verify the permissions match, cut over, run a full cycle, confirm idle, delete.
Revoke first and investigate second. Removal is cluster-wide on the next request.
Tokens survive the offboarding of their owner unless someone removes them explicitly.

Knowledge check

Why this matters

Tokens are where least privilege quietly stops being enforced, and the reason is structural rather than careless.

Every other credential in Proxmox has a human attached to it. A person logs in, a second factor challenges them, a realm decides whether they still work here, and an offboarding process eventually removes them. A token has none of that. It cannot be challenged, no directory owns its lifecycle, and its owner leaving the company changes nothing about whether it works. The only things standing between a token and the whole cluster are the two flags on the command that created it, and both default in the direction of “keep working”.

That is why a token with more privilege than its purpose is the finding that recurs in every review of a mature estate. It is never the result of a decision to over-grant. It is the result of a permission error at 17:40 on a Friday, --privsep 0 making the error stop, and nobody returning to narrow it — because the narrow version and the broad version look identical from every angle except the one nobody checks.

The habits that close it are small and unglamorous. Every token gets a comment naming its owner. Every token gets an expiry, so the worst case has an end date even when everything else is forgotten. Every token gets pveum user token permissions run against it during a review, and the result gets compared to the comment. And rotation happens with overlap, because a rotation that requires an outage is a rotation that gets deferred until it is an incident instead.