API security, hardening, and credential management

API tokens are the keys to your cluster. Lose one and an attacker has full automation power. This lesson covers the security practices that keep tokens safe.

Token security

A token is a credential. Treat it like a password.

Storage rules:

• Never in version control
• Never in plain-text config files
• Never in chat, tickets, or email
• Always in a secrets manager (HashiCorp Vault, AWS Secrets Manager, Mozilla SOPS, or sealed-secrets for Kubernetes)
• Always encrypted at rest

Token properties:

• Long enough to be cryptographically secure (the default 8-byte UUID is fine)
• Scoped to a user with minimum required privileges
• Tagged with a description that identifies the consumer
• Expiry set for human users (--expire 365); tokens for service accounts can be long-lived

Creating tokens with least privilege

# Create a user for backup reporting
pvesh create /access/users/reporting@pve --comment "PBS reporting"

# Create a custom role with only read access
pvesh create /access/roles/ReportingRole \
  --privs "VM.Audit,Datastore.Audit,Pool.Audit,User.Audit"

# Apply the role to the user at the cluster root
pvesh create /access/acl --path / --roles ReportingRole --users reporting@pve

# Create a token for that user with a 1-year expiry
pvesh create /access/users/reporting@pve/token/api --privsep 0 --expire 365

The user has only the privileges of ReportingRole, no more. Even if the token leaks, the attacker can only read cluster state, not modify it.

Rotating tokens

Tokens should be rotated on a schedule:

# Generate a new token
NEW_TOKEN=$(pvesh create /access/users/automation@pve/token/rotation \
  --privsep 0)

# Update the secret in your secrets manager
vault kv put secret/proxmox/token value="$NEW_TOKEN"

# Verify automation still works
./run-automation.sh

# Delete the old token once verified
pvesh delete /access/users/automation@pve/token/old-name

A good rotation cadence:

• Service accounts (automation): every 90 days
• Human admin tokens: every 30 days or after the human leaves
• Emergency tokens: created on demand, deleted immediately

Detecting API abuse

The PVE API logs every request. Detect abuse by:

# Top API users by request count
journalctl -u pveproxy --since '1 day ago' | \
  grep -oE 'user [^ ]+' | sort | uniq -c | sort -rn | head

# Failed authentication attempts
journalctl -u pveproxy --since '1 day ago' | \
  grep -i 'auth fail\|invalid' | wc -l

# Source IPs hitting the API
journalctl -u pveproxy --since '1 day ago' | \
  grep -oE 'from [0-9.]+' | sort | uniq -c | sort -rn

# Token usage (which tokens are active and from where)
pvesh get /access/users --output-format json | \
  jq '.[] | .tokens[]?.tokenid'

Set up Prometheus alerts:

# alert on unusual API activity
- alert: APIAuthFailuresSpike
  expr: |
    sum by(source_ip) (
      rate(pve_api_auth_failures_total[5m])
    ) > 5
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Possible brute force from {{ $labels.source_ip }}"

- alert: NewTokenCreated
  expr: |
    increase(pve_tokens_created_total[1h]) > 0
  labels:
    severity: info
  annotations:
    summary: "New API token created — review for legitimacy"

Secrets manager integration

For team-scale automation, integrate API tokens with a secrets manager. HashiCorp Vault example:

# Store the token in Vault
vault kv put secret/proxmox/automation \
  token="automation@pve!token=12345678-..." \
  url="https://pve-01:8006/api2/json"

# Read the token from Vault in automation
export PROXMOX_TOKEN=$(vault kv get -field=token secret/proxmox/automation)
export PROXMOX_URL=$(vault kv get -field=url secret/proxmox/automation)
./run-automation.sh

For Kubernetes:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: proxmox-automation
spec:
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: proxmox-automation
  data:
  - secretKey: token
    remoteRef:
      key: proxmox/automation
      property: token

Production considerations

• Token expiry enforcement. Tokens with --expire 365 expire after a year. Tokens without expire never expire. Always set expiry for human tokens; document why a service token is long-lived.
• Audit log review. Every token use is logged. Review the log weekly for unexpected tokens or unexpected sources.
• Network segmentation. API endpoints should only be reachable from the management network. Restrict via PVE firewall and network ACLs.
• Break-glass procedures. Document how to recover from a compromised token: how to revoke all tokens, create a new one, rotate secrets in dependent systems.

Common mistakes

• Tokens in shell history. Set tokens via read -s or via a secrets manager.
• Tokens shared between services. Each service should have its own token. Sharing means you can’t revoke one without breaking the other.
• Long-lived tokens without rotation. Tokens without expiry are tokens you’ll forget about. Set expiry and rotate.
• No audit log. Without logs, you can’t tell if a token was compromised.

Key takeaways

• Treat API tokens like passwords: secrets manager, rotation, audit.
• Use least-privilege roles, not the admin user.
• Detect abuse via audit logs and rate alerts.
• Have a break-glass procedure for token compromise.

Knowledge check