Certificate rotation, ACME renewal, and TLS hardening

The cluster certificate is the identity your cluster presents to every client: the GUI, the API, pvesh, qm remote, every SDK, every Terraform provider. If it expires without warning, the cluster goes dark for everyone using external integrations. If it expires with warning, you get a frantic Friday afternoon.

This lesson covers the full certificate lifecycle on PVE 9.x: choosing between ACME and manual, configuring HTTP-01 and DNS-01 challenges, planning rotation to avoid outage, and hardening the TLS configuration on the API and console proxy.

Mental model

PVE has three independent certificate surfaces:

1. Cluster certificate (/etc/pve/local/pveproxy-ssl.pem) — shared via pmxcfs to every node. Used by pveproxy for the GUI and API, by pvedaemon for the local socket, and by Spice / noVNC for the console proxy.
2. Node API certificate (/etc/pve/local/pveproxy-ssl.pem on the node you ran pvenode cert install on) — superseded by the cluster certificate in PVE 7+.
3. Per-VM display certificates (Spice / noVNC) — generated on demand, short-lived, no operational concern.

The cluster certificate is what you manage. The other surfaces inherit it.

How ACME works in PVE

pvenode acme is a thin wrapper around acme.sh that handles account registration, challenge setup, renewal scheduling, and service reloads. The flow:

1. Register an account with the CA (Let’s Encrypt by default).
2. Configure a plugin for the challenge type (HTTP or DNS).
3. pvenode acme cert order issues a new cert from the CA using the configured challenge.
4. pvenode acme cert install writes the cert to pmxcfs and reloads pveproxy / pvedaemon.
5. A systemd timer (pve-daily-update.timer or the ACME renewal timer) checks expiry daily and renews if fewer than 30 days remain.

PVE 9.x uses ACME v2 (RFC 8555). The default CA is Let’s Encrypt (https://acme-v02.api.letsencrypt.org/directory).

Choosing a challenge type

For a production cluster of more than one node, DNS-01 with a wildcard cert is the right choice — it gives every node the same cert without opening port 80.

Configuring DNS-01 with the PowerDNS plugin

The DNS-01 plugin list is the same as acme.sh’s. For PowerDNS:

# Install the plugin into acme.sh\'s plugin directory
curl -s https://raw.githubusercontent.com/acmesh-official/acme.sh/master/dnsapi/dns_pdns.sh \
  -o ~/.acme.sh/dnsapi/dns_pdns.sh

# Register an ACME account
pvenode acme account register default contact@example.com

# Configure the plugin with PowerDNS API URL and key
# (heredoc replaced)
echo "PDNS_Url="https://dns.example.com/api/v1"" >> /etc/pve/priv/acme/plugins/pdns.cfg
echo "PDNS_ServerId="localhost"" >> /etc/pve/priv/acme/plugins/pdns.cfg
echo "PDNS_Token="<api-token>"" >> /etc/pve/priv/acme/plugins/pdns.cfgchmod 600 /etc/pve/priv/acme/plugins/pdns.cfg

# Order a wildcard certificate
pvenode acme cert order --domain cluster.example.com --domain "*.cluster.example.com" --plugin pdns

# Install it cluster-wide
pvenode acme cert install cluster.example.com

Verify:

pvenode acme cert info
# Subject: CN = *.cluster.example.com
# Issuer: C = US, O = Let\'s Encrypt, CN = R10
# Not Before: ...
# Not After:  ...

# Confirm the running services see the new cert
systemctl reload pveproxy pvedaemon
curl -kvI https://pve-01.cluster.example.com:8006/api2/json | grep -i 'subject\|issuer\|expire'

Renewal timing and what can go wrong

Let’s Encrypt certs are valid for 90 days. pve-daily-update.timer (or the equivalent ACME renewal check) renews at 30 days remaining. That gives you 60 days of buffer if ACME itself is broken.

The failure modes to plan for:

• ACME account lost — the CA deletes the account if it’s unused for a long time. Solution: ensure renewal has happened at least once in the last 60 days.
• DNS plugin credentials expired — rotate the API token and update the plugin config. The next renewal will use the new credentials.
• Network egress blocked — PVE needs to reach the CA. If the CA is unreachable, the existing cert keeps working until it expires; you have 90 days to fix egress.
• Rate limit hit — Let’s Encrypt has a 50 certs/week limit per registered domain. Don’t run pvenode acme cert order from CI on every commit.

Manual rotation (offline / air-gapped)

If your cluster has no internet access at all, you need to use a private CA or maintain an offline ACME workflow:

# Generate a CSR
openssl req -new -newkey rsa:4096 -nodes \
  -keyout /etc/pve/priv/pveproxy.key \
  -out /tmp/pveproxy.csr \
  -subj "/CN=cluster.example.com"

# Sign the CSR with your private CA
# (then cat the signed cert + CA chain into a single PEM)
cat /tmp/pveproxy.crt /etc/pve/priv/ca-chain.pem \
  > /etc/pve/local/pveproxy-ssl.pem

# Reload services
systemctl reload pveproxy pvedaemon

Set a calendar reminder to repeat this at least 14 days before expiry. For air-gapped clusters this is the only safe pattern — never rely on long-lived certificates.

TLS hardening

The default PVE TLS configuration is reasonable but not strict. To harden:

# /etc/pve/local/pveproxy-ssl.pem should be at least 4096-bit RSA or
# EC P-384. Check the current key:
openssl x509 -in /etc/pve/local/pveproxy-ssl.pem -noout -text | \
  grep -E 'Public-Key Algorithm|NIS Curves'

# Disable TLS 1.0 and 1.1 by ensuring the node OpenSSL config rejects
# them. PVE 9.x already does this, but verify with:
openssl s_client -connect pve-01:8006 -tls1 </dev/null 2>&1 | grep -i 'protocol\|alert'
# Expected: "alert handshake failure" or similar

For the console proxy, pveproxy exposes its own TLS settings. To restrict cipher suites:

# (heredoc replaced)
echo "PVE_TLS_CIPHERS="ECDHE+AESGCM:ECDHE+CHACHA20:DHE+AESGCM:DHE+CHACHA20"" >> /etc/default/pveproxy
echo "PVE_TLS_PROTOCOLS="--tls-max 1.3"" >> /etc/default/pveproxysystemctl restart pveproxy

Verify with nmap --script ssl-enum-ciphers pve-01 -p 8006 from a test host.

Planning rotation during business hours

Two safe patterns:

1. Renew-then-swap — let ACME renew the cert (the existing cert is still served by pveproxy until reload), then bounce the service during a planned window. Pro: simple, predictable. Con: brief window where old cert is served.

2. Dual-cert serve — configure pveproxy to serve the new cert on one node, validate, then swap cluster-wide. This is more complex but lets you roll back instantly.

For most clusters, pattern 1 with ACME auto-renewal at 30 days remaining is fine. Bounce pveproxy after hours.

Common mistakes

• Forgetting to set up the ACME timer — the cert is ordered and installed, but the daily timer that renews it is missing. After 60 days, the cert expires.
• Mixed challenge types — having one node on HTTP-01 and another on DNS-01 means renewal failures look different on each node. Standardise.
• Cert in the wrong file — PVE expects pveproxy-ssl.pem to contain the leaf + chain concatenated. A cert without the chain produces client-side “untrusted issuer” warnings.
• Wildcard for a single-node cluster — DNS-01 with a wildcard is overkill if you only have one node and one FQDN. HTTP-01 is simpler.

Key takeaways

• ACME + DNS-01 + wildcard is the standard pattern for multi-node clusters.
• The cert lives in /etc/pve/local/pveproxy-ssl.pem and is replicated via pmxcfs. One cert, every node.
• PVE auto-renews at 30 days remaining. If renewal silently fails for 60 days, the cluster goes dark on day 90.
• Test TLS hardening on one node before rolling to the cluster.

Knowledge check