medium riskservice affecting~120 min

Rotate cluster and API credentials, including tokens

1 · Prerequisites

Confirm every item is in place before any state change.

The scope of the rotation is defined: scheduled hygiene, an offboarding, or a suspected compromise - because a compromise removes the option of a gradual overlap
Every consumer of each credential is identified, or the identification method is agreed and its blind spots stated
A secret store exists to hold the new values, because a rotation that ends with a token in a chat message has not improved anything
Change windows are known for any automation that will need reconfiguring
The privilege each token actually needs is known, since rotation is the natural moment to reduce an over-broad one
Someone with an independent, working administrative login exists, so a mistake does not lock everyone out

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· pveum user list and pveum user token list enumerate the accounts and tokens that currently exist
· pveum acl list shows what each token and user is actually permitted to do
· The pveproxy access log confirms which tokens have been used recently, and which have not been used at all
· A second administrative session is open and confirmed working, on a different account from the one being rotated
· Scheduled jobs, external integrations and monitoring systems that authenticate to this cluster are listed
· It is confirmed that the PBS datastore encryption key is not in scope, because that key cannot be rotated without re-taking every backup

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Enumerate every credential in scope: API tokens, user accounts, automation SSH keys, and storage credentials in storage.cfg
2Determine the real consumer of each one from the access log, not from documentation
3For each token: create the replacement alongside the old one, with the minimum privileges it needs
4Store the new secret in the secret store before it is used anywhere, because it is displayed only once
5Deploy the new credential to each consumer and confirm that consumer works with it
6Watch the access log until the old credential shows no further use across at least one full scheduling cycle
7Revoke the old credential
8Confirm the old credential now fails authentication, by testing it rather than assuming
9Repeat for SSH keys: add the new key, verify, then remove the old one
10Rotate storage credentials in storage.cfg and confirm each storage returns to active
11Record what was rotated, when, and which consumers were touched

4 · Verification

Confirm the procedure actually fixed the problem.

✓A request using each old token returns 401, verified by making the request
✓A request using each new token succeeds and is limited to the intended paths
✓The pveproxy access log shows no further use of any revoked token after the revocation time
✓Every scheduled job that authenticates has run to completion at least once after the rotation, on its normal schedule
✓pvesm status shows every storage active after any storage credential change
✓SSH to every node using the new key succeeds, and using the old key fails
✓No authentication failures appear in the logs that correspond to a consumer nobody remembered
✓The new secrets exist in the secret store and nowhere in shell history, chat, or a ticket

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶While the old and new credentials both exist, rollback is simply pointing the consumer back at the old one
↶Revocation is the point of no return: a deleted API token secret cannot be recovered, only replaced
↶Recreating a token with the same identifier produces a NEW secret, so every consumer must be updated again - which is why the overlap period exists
↶If a forgotten consumer breaks after revocation, the fix is to issue it a new token, not to try to restore the old one
↶For SSH keys, keep the old key authorised until the new one is proven on every node; removing it first can lock automation out of a node that was unreachable during the change
↶In a suspected compromise there is no rollback and no overlap - the old credential is revoked immediately and broken consumers are repaired afterwards

6 · Escalation

When the runbook isn't enough, contact:

· Escalate to the security owner immediately if the rotation is a response to a suspected compromise, because the sequence and urgency change completely
· Escalate before revoking any credential whose consumer could not be positively identified
· Escalate to the service owner if a rotation will interrupt an integration during business hours
· Escalate if a credential is found with far broader privileges than its purpose requires, since that is a finding in its own right
· Escalate if any secret is discovered stored in plaintext in a repository, a wiki, or a ticket, because rotating it is only half the remediation

Verified against Proxmox VE 9.2.4 and Proxmox Backup Server 4.2.5.

Credential rotation fails in exactly one way: something nobody remembered was using the old secret, and it stops working at 02:00 three days later when its schedule next fires.

Everything in this runbook is built around that. Create the new credential before revoking the old one, prove each consumer works, watch the access log through a full scheduling cycle, and only then revoke. The overlap period is the entire safety mechanism, and the only case where you skip it is a compromise - where the risk of the old credential continuing to work outweighs the risk of breaking something.

When to use this runbook

Scheduled credential hygiene.
Someone with cluster access has left.
A token appeared somewhere it should not have - a repository, a ticket, a screenshot.
An audit requires it.
Suspected compromise. Read the callout in Step 2 first; the sequence changes.

Step 1: Enumerate what exists

Read-only / Safeaccounts, tokens, and what they may do

pveum user list
pveum group list
pveum role list

# Tokens per user
USER=automation@pve
pveum user token list "$USER"

# What everything is actually allowed to do
pveum acl list

Read-only / Safecredentials that live outside the user database

# Storage credentials: PBS passwords, iSCSI CHAP, CIFS
grep -vE '^\s*#' /etc/pve/storage.cfg | grep -iE 'username|password|fingerprint|datastore'
ls -la /etc/pve/priv/ 2>/dev/null

# Automation SSH keys trusted by the cluster
wc -l /etc/pve/priv/authorized_keys 2>/dev/null
ls -la /root/.ssh/

Build one list. A rotation that covers API tokens and forgets the SSH key the backup script uses has rotated the credential that was audited and not the one that grants root.

Step 2: Decide the mode

A suspected compromise inverts this entire procedure

The gradual, overlapping rotation below is designed to avoid breaking things. It assumes the old credential continuing to work for a few hours is harmless.

If a credential may be in someone else’s hands, that assumption is false and the order reverses: revoke first, repair afterwards. Every hour of overlap is an hour of continued access.

In that mode:

Revoke immediately, before identifying consumers.
Assume everything that token could reach is compromised, not just what the logs show it touched.
Rotate everything the affected account could have read, including any secret stored in a guest it had access to.
Audit the task log and access log for what was done with it.
Treat the broken automation as expected work, not as a failure of the rotation.

If you are unsure which mode you are in, use this one. The cost of over-rotating is a busy afternoon.

Step 3: Find the real consumers

Documentation says what a token was created for. The access log says what is using it.

Read-only / Safewho is actually authenticating

TOKENID=backup-runner

grep -h "$TOKENID" /var/log/pveproxy/access.log* 2>/dev/null | tail -30
grep -h "$TOKENID" /var/log/pveproxy/access.log* 2>/dev/null | awk '{print $1}' | sort | uniq -c | sort -rn

That second command gives you the source IPs, which is the answer to “who uses this”. Match each IP against known systems.

Read-only / Safetokens with no recent use are their own finding

for T in $(pveum user token list automation@pve --output-format json 2>/dev/null \
  | grep -oE '"tokenid":"[^"]+"' | cut -d'"' -f4); do
N=$(grep -hc "$T" /var/log/pveproxy/access.log* 2>/dev/null | paste -sd+ | bc)
echo "$T used $N times in retained logs"
done

Step 4: Create the replacement alongside the old one

Configuration changenew token, minimum privileges

USER=automation@pve
NEWID=backup-runner-2026-08

pveum user token add "$USER" "$NEWID" --privsep 1 --comment 'rotation 2026-08, replaces backup-runner'
# The secret is displayed ONCE. Put it in the secret store now.

Configuration changegrant only what it needs

USER=automation@pve
NEWID=backup-runner-2026-08

# With --privsep 1 the token has NO permissions until granted explicitly.
pveum acl modify /vms --tokens "$USER!$NEWID" --roles PVEVMAdmin
pveum acl modify /storage/pbs-main --tokens "$USER!$NEWID" --roles PVEDatastoreUser

pveum acl list | grep "$NEWID"

Step 5: Deploy and prove

Read-only / Safedoes the new token work, and is it correctly scoped

HOST=192.0.2.11
USER=automation@pve
NEWID=backup-runner-2026-08
SECRET=REPLACE_ME_FROM_SECRET_STORE

# Should succeed
curl -sS -o /dev/null -w '%{http_code}\n' \
-H "Authorization: PVEAPIToken=$USER!$NEWID=$SECRET" \
"https://$HOST:8006/api2/json/nodes"

# Should FAIL with 403 if the token is correctly limited
curl -sS -o /dev/null -w '%{http_code}\n' \
-H "Authorization: PVEAPIToken=$USER!$NEWID=$SECRET" \
"https://$HOST:8006/api2/json/access/users"

A permissions test that only checks the success case proves the token works. Checking that it is refused where it should be is what proves it is scoped.

Then update each consumer, one at a time, and confirm each one works before moving to the next.

Step 6: Watch for stragglers

Read-only / Safeis anything still using the old token

OLDID=backup-runner
SINCE='2026-08-12'

grep -h "$OLDID" /var/log/pveproxy/access.log 2>/dev/null | tail -20
grep -hc "$OLDID" /var/log/pveproxy/access.log 2>/dev/null

Wait through at least one full cycle of the longest schedule that could use it. A daily job needs 24 hours; a monthly report needs a month, or an explicit confirmation from its owner.

Step 7: Revoke, and prove the revocation

Service impact possibleremove the old token

USER=automation@pve
OLDID=backup-runner

pveum user token remove "$USER" "$OLDID"
pveum user token list "$USER"

Read-only / Safethe old secret must now fail

HOST=192.0.2.11
USER=automation@pve
OLDID=backup-runner
OLDSECRET=REPLACE_ME_OLD_VALUE

curl -sS -o /dev/null -w '%{http_code}\n' \
-H "Authorization: PVEAPIToken=$USER!$OLDID=$OLDSECRET" \
"https://$HOST:8006/api2/json/nodes"
# Must print 401. Anything else means the revocation did not take effect.

That check can fail, which is the point. “I deleted the token” and “the token no longer authenticates” are different claims, and only one of them is testable.

Step 8: SSH keys

Same pattern: add, prove, then remove.

Configuration changeissue and distribute the new key

ssh-keygen -t ed25519 -f /root/.ssh/id_ed25519_2026 -C 'pve-automation-2026-08'
ssh-keygen -lf /root/.ssh/id_ed25519_2026.pub

for N in pve01 pve02 pve03; do
ssh-copy-id -i /root/.ssh/id_ed25519_2026.pub "root@$N"
done

Read-only / Safeprove the new key on every node before removing the old

for N in pve01 pve02 pve03; do
ssh -i /root/.ssh/id_ed25519_2026 -o IdentitiesOnly=yes -o BatchMode=yes \
  "root@$N" 'hostname -s' || echo "FAILED on $N"
done

Service impact possiblethen remove the old key, everywhere

OLDFP='SHA256:REPLACE_WITH_OLD_KEY_FINGERPRINT'
OLDCOMMENT='pve-automation-2023'

for N in pve01 pve02 pve03; do
ssh -i /root/.ssh/id_ed25519_2026 "root@$N" \
  "sed -i '/$OLDCOMMENT/d' /root/.ssh/authorized_keys; grep -c . /root/.ssh/authorized_keys"
done

Step 9: Storage credentials

Configuration changerotate a PBS credential used by a storage

STORE=pbs-main

grep -A8 "pbs: $STORE" /etc/pve/storage.cfg

# Set the new password or token secret non-interactively is not supported;
# use the GUI, or:
pvesm set "$STORE" --password
# then re-verify:
pvesm status | grep "$STORE"

The PBS datastore encryption key is not a credential and cannot be rotated this way

Access credentials to a PBS datastore can be rotated freely. The encryption key used for client-side encrypted backups cannot: every existing snapshot is encrypted under it, and changing it does not re-encrypt them.

If you change the key, old backups remain readable only with the old key - so the old key must be retained for as long as those backups must be restorable, which is usually the whole retention period. Deleting it makes every backup taken under it permanently unreadable.

Treat the encryption key as a long-lived secret with its own lifecycle, kept in the secret store and never rotated as part of a routine credential sweep. If it must be changed - because it was exposed - the correct sequence is: introduce the new key, re-take full backups under it, keep the old key until the old snapshots have aged out of retention, and only then retire it.

Verify the key is retrievable from somewhere that is not the cluster it protects. A key stored only on the node whose loss you are protecting against is not a backup strategy.

Step 10: Verify the whole estate

Read-only / Safenothing quietly failing

journalctl -u pveproxy --since '24 hours ago' | grep -iE 'authentication failure|401|permission denied' | tail -30
journalctl -u pvedaemon --since '24 hours ago' | grep -iE 'auth|denied' | tail -20

pvesm status
ha-manager status
grep -h 'ERROR' /var/log/pve/tasks/index 2>/dev/null | tail -20

Then wait for one full run of each scheduled job and confirm it succeeded. Backups are the ones that fail silently and matter most.

Rollback

Stage	Rollback
New credential created, old still active	Point the consumer back at the old one
Consumers migrated, old not yet revoked	Same. This is why the overlap exists
Old credential revoked	None. The secret is unrecoverable. Issue a new one to the broken consumer
SSH old key removed	Re-add the public key, if you still have it. Otherwise console access
Storage credential changed	Set it back, if the old value is still known
Compromise-mode revocation	No rollback by design

Common patterns

Symptom	Likely cause	Resolution
Backup fails days after rotation	A monthly job used the old token	Issue it a new one; extend log retention
New token returns 403 everywhere	`--privsep 1` with no ACL granted	`pveum acl modify` for the paths it needs
Old token still works after removal	Removed the wrong token, or a cached session	Re-check `token list`; test with `curl`
Automation locked out of one node	New key never landed there	Console; re-add; verify all nodes next time
Secret ended up in shell history	Pasted on a command line	Rotate again; use a secret store and a file
Storage inactive after rotation	Password set but fingerprint changed too	Re-check the PBS fingerprint in `storage.cfg`
Old backups unreadable	Encryption key rotated	Restore the old key. If it is gone, they are gone

Escalation

Escalate when:

The rotation is a response to a possible compromise.
Any credential’s consumer could not be identified.
A credential is found with privileges far beyond its purpose.
A secret is found stored in plaintext somewhere durable.
The PBS encryption key is in question.

Why this matters

Rotation is a two-step operation that people perform as one. Revoking is easy and instant; replacing means finding every consumer, and the list of consumers is never the list in the documentation. The overlap period is not caution - it is the only mechanism that turns “we think that is all of them” into evidence.

The access log is the tool that makes this real, with one important limitation: it only knows what happened inside its retention window. A quarterly job is invisible to a weekly log, and the failure it produces arrives long after anyone would connect it to the rotation.

The other thing worth separating clearly in your head: access credentials and encryption keys are different objects that both feel like secrets. You can rotate the first as often as you like. Rotating the second without re-encrypting what it protects does not improve security, it just decides which of your backups you can still read.