PKI and certificate rotation — x509, ACME, Let's Encrypt, the rotation schedule

A production VyOS router uses certificates for the HTTP API, IPsec, WireGuard, OpenVPN, and the Let’s Encrypt ACME client. Each certificate has a finite lifetime; when it expires, the service that uses it fails. The defensive pattern: every certificate has a documented rotation schedule, every expiry is monitored, and every renewal is automated.

This lesson covers the PKI model on VyOS 1.5 LTS, the three certificate sources (self-signed, internal CA, ACME / Let’s Encrypt), the certificate rotation schedule, the expiry failure mode, and the production discipline of pre-expiry rotation.

The PKI model

flowchart LR
  CA["Certificate Authority"] -->|signs| CERT["Certificate"]
  KEY["Private key<br/>stays on router"] --> CERT
  CERT --> API["HTTP API<br/>TLS"]
  CERT --> VPN["IPsec / WireGuard<br/>TLS"]
  CERT --> ACME["ACME client<br/>renews automatically"]
  ACME --> LE["Let's Encrypt<br/>public CA"]
  CA -.->|self-signed<br/>internal CA| CERT

The PKI model:

• Certificate — a public key signed by a CA. The certificate identifies the service (e.g., api.example.com).
• Private key — the secret half of the key pair. Stays on the router.
• Certificate Authority (CA) — the entity that signs the certificate. Self-signed (the router signs its own), internal CA (an organisation’s CA), or public CA (Let’s Encrypt).
• Service — uses the certificate for TLS. The HTTP API, IPsec, WireGuard, etc.

Where certificates live on VyOS 1.5

Before any command: on a modern VyOS the certificate itself is part of the configuration, not a file the configuration points at. The CLI stores the PEM body base64-encoded under pki:

set pki certificate router-api certificate <base64 body>
set pki certificate router-api private key <base64 body>
set pki ca internal-ca certificate <base64 body>
set pki ca internal-ca private key <base64 body>

Everything downstream then references the name, not a path. This is the single largest change from the older model in which, as the VyOS documentation puts it, “every service referenced a file”. Two consequences follow immediately and both matter operationally:

• A configuration backup now contains the private keys. Treat the config archive as a secret store, encrypt it at rest, and think twice before pasting a show configuration into a ticket.
• Rotating a certificate is a commit, which means it is also a rollback — the previous certificate is in the revision history alongside everything else. The 1.3-era “replace the file and restart the service” procedure has no equivalent here, and looking for one is how operators end up editing files under /config/auth that nothing reads any more.

Generating a self-signed certificate

For services that do not need to be trusted by external clients — the internal HTTP API is the usual case — a self-signed certificate is enough. Generation is an operational-mode command, not a set:

generate pki certificate self-signed install router-api

The CLI prompts for the subject fields and validity, then writes the result straight into the configuration as pki certificate router-api. Drop the trailing install <name> and it prints the certificate and key to the console instead, formatted as the set commands you would paste into configuration mode — which is how you move a key onto a second router without it touching a filesystem in between.

Attach it to the HTTP API by name:

configure
set service https certificates certificate router-api
set service https listen-address 192.0.2.1
set service https tls-version 1.3
commit
save

There is no cert-file or key-file under service https on 1.5 — the node is certificates certificate <name>, with certificates ca-certificate <name> for the chain and certificates dh-params <name> where DH parameters are needed. If no certificate is configured at all the service generates its own self-signed one, which is convenient for a first boot and unsuitable for anything after that, because nothing tracks its expiry.

Clients must add the certificate to their trust store, or skip verification with curl -k — and a router whose API is only ever reached with -k has a TLS configuration that is decoration rather than security.

Using an internal CA

For services that internal clients must trust — a monitoring system scraping the API, for instance — sign with an internal CA. VyOS can hold the CA itself, and the whole chain is built with operational commands:

generate pki ca install internal-ca
generate pki certificate sign internal-ca install router-api

The first creates the CA and installs it as pki ca internal-ca. The second creates a key pair, signs it with that CA, and installs the result as pki certificate router-api. Then reference both by name:

configure
set service https certificates certificate router-api
set service https certificates ca-certificate internal-ca
commit
save

If the CA lives off-box — which is the arrangement the next block argues for — you bring the signed material in with import, which reads a file once and stores the contents in the configuration:

import pki ca internal-ca file /tmp/internal-ca.crt
import pki certificate router-api file /tmp/router-api.crt
import pki certificate router-api key-file /tmp/router-api.key

Note what is not imported in that sequence: the CA’s private key. import pki ca <name> key-file <path> exists and will put it on the router, and on a production edge router you should be able to say out loud why you used it.

ACME / Let’s Encrypt

For public-facing services (e.g., a router’s VPN endpoint accessible from the Internet), Let’s Encrypt provides free, automated certificates via the ACME protocol:

ACME is a sub-tree of an ordinary pki certificate entry — the same node that holds a hand-installed certificate, with the issuance delegated:

configure
set pki certificate router-public acme domain-name 'router.example.com'
set pki certificate router-public acme email 'admin@example.com'
set pki certificate router-public acme listen-address '192.0.2.1'
set pki certificate router-public acme rsa-key-size '4096'
commit
save

domain-name and email are mandatory, and domain-name can be repeated for a multi-name certificate. url defaults to Let’s Encrypt’s production directory and only needs setting to point at a staging directory or a private ACME server — and pointing it at a staging endpoint while you get the challenge working is the difference between a failed test and a week-long rate limit.

listen-address is documented as “the address the server listens to during http-01 challenge”. That one option tells you what the router is going to do: bring up a listener on the named address and wait for the ACME server to reach it over HTTP.

Renewal runs automatically. The documented manual trigger is:

renew certbot

with the documentation noting that renewal “will be done twice a day” on its own. The operator’s job is not to run this; it is to know that when the automatic attempt fails, nothing on the router escalates.

Certificate rotation

Every certificate has a finite lifetime (typically 90 days for Let’s Encrypt, 365 days for internal CAs). The defensive pattern: every certificate has a documented rotation schedule, and the operator rotates the certificate before it expires.

flowchart LR
  A["Certificate issued<br/>validity 90 days"] --> B{"Days to expiry?"}
  B -- "30+" --> C["No action"]
  B -- "30 to 7" --> D["Renew certificate<br/>test the new cert"]
  B -- "less than 7" --> E["URGENT: service<br/>may fail soon"]
  B -- "expired" --> F["Service is down<br/>renew immediately"]

The rotation schedule:

• 30 days before expiry: renew the certificate (test in a staging environment if possible).
• 7 days before expiry: the renewal should already be deployed; if not, treat as urgent.
• At expiry: the service that uses the certificate fails (HTTP API returns 503, IPsec tunnel drops, WireGuard peer fails).
• Post-expiry: the service is down; the operator must renew and restart.

Monitoring for expiry

# Prometheus blackbox exporter: seconds remaining before the certificate expires
probe_ssl_earliest_cert_expiry - time() < 30 * 86400

The blackbox exporter’s probe_ssl_earliest_cert_expiry is a gauge holding the Unix timestamp of the earliest expiry in the presented chain, so subtracting time() gives seconds remaining. Note that this probes the endpoint from outside, which means it measures the certificate the service is actually presenting — the thing show pki certificate cannot tell you. An alert fires when the expiry is less than 30 days away:

# /etc/prometheus/rules/certificates.yml
groups:
  - name: certificates
    rules:
      - alert: CertificateExpiringSoon
        expr: probe_ssl_earliest_cert_expiry - time() < 30 * 86400
        for: 1h
        labels:
          severity: warning
        annotations:
          summary: "Certificate expires in less than 30 days"
          description: "Certificate for {{ $labels.instance }} expires in {{ $value | humanizeDuration }}."

      - alert: CertificateExpiringImminently
        expr: probe_ssl_earliest_cert_expiry - time() < 7 * 86400
        for: 1h
        labels:
          severity: critical
        annotations:
          summary: "Certificate expires in less than 7 days"
          description: "Certificate for {{ $labels.instance }} expires in {{ $value | humanizeDuration }}. Service may fail soon."

The alert fires when the certificate is within 30 days of expiry (warning) or 7 days (critical). The operator renews the certificate before it expires.

How the result is validated

The first stop is the router’s own view of its PKI store:

show pki ca
show pki ca internal-ca
show pki certificate
show pki certificate router-api

show pki certificate lists what is installed and show pki certificate router-api details one entry. Be clear about what you are reading: the router’s view of a certificate that is in its own configuration. It does not tell you what a client negotiating TLS actually receives, which is a different question and the one that matters while the API is failing.

For that, ask from outside, over the wire:

openssl s_client -connect 192.0.2.1:443 -servername router-api.example.com </dev/null 2>/dev/null | openssl x509 -noout -subject -issuer -dates
curl -v --cacert ./internal-ca.crt https://router-api.example.com/show/version

The openssl s_client pipeline shows the certificate the service is presenting right now. That is the check that catches the case where the new certificate was committed but the service was never repointed: the router holds two certificates, show pki certificate lists both, and the one on the wire is still the expired one.

Note what is absent: there is no openssl x509 -in /config/auth/... step, because on 1.5 there is no such file to read. The certificate is in the configuration, and the /config/auth path belongs to the pre-1.4 model.

A working certificate:

• Has a subject that matches the service (router-api.example.com)
• Has an issuer that is the configured CA (internal CA or Let’s Encrypt)
• Has a validity period that is at least 30 days from now
• Is trusted by the configured clients (the trust store includes the issuer)
• Is the certificate the service is actually presenting, confirmed from a client rather than from the router

vyos@R1:~$ show pki certificate

Certificates:

Name           Type   Subject                      CA            Issued      Expiry
-------------  -----  ---------------------------  ------------  ----------  ----------
router-api     Cert   CN=router-api.example.com    internal-ca   2026-08-19  2027-08-19
router-public  Cert   CN=router.example.com        N/A           2026-08-16  2026-11-14

The shape above is illustrative rather than a capture — read it for the columns worth looking at, not for the exact formatting, which varies between releases. The three that decide whether you have a problem are the subject, the issuing CA, and the expiry date.

The certificate rotation procedure

flowchart TD
  A["30 days before expiry"] --> B["Generate new CSR<br/>or request ACME renewal"]
  B --> C["Sign with CA<br/>or wait for ACME"]
  C --> D["Install new certificate<br/>on the router"]
  D --> E["Test the service<br/>HTTPS endpoint"]
  E --> F{"Service works?"}
  F -- "yes" --> G["Commit and save"]
  F -- "no" --> H["Debug<br/>rollback to old cert"]
  G --> I["Update monitoring<br/>document the rotation"]

The rotation procedure:

1. Generate the new certificate under a new name — generate pki certificate sign internal-ca install router-api-2027, or let ACME reissue. Never overwrite the entry the service is currently using; you want both in the configuration at once so the switch and the rollback are each a single edit.
2. Point the service at it. set service https certificates certificate router-api-2027, then commit-confirm 5. The confirm window matters: if the new certificate is malformed the API is the thing you would have used to fix it.
3. Test from a client, not from the router — openssl s_client against the service and check the dates and issuer you get back.
4. confirm, then save. Until you confirm, the old certificate is one timeout away from coming back.
5. Delete the retired entry once the new one has been serving for a full monitoring interval: delete pki certificate router-api. Leaving it installed is how a later operator repoints the service at an expired certificate that show pki certificate still lists.
6. Update monitoring and document the rotation. Record the date, the fingerprint, and the name you used, because the name is now the thing every service configuration references.

How it fails

The production failure modes a routing engineer must recognise:

• Certificate expired. The certificate was valid for 365 days; nobody renewed. The HTTP API returns 503; IPsec tunnels drop. The fix: renew immediately; the service is down.
• ACME renewal failure. The HTTP-01 challenge fails because port 80 is not reachable on the ACME listen-address. The renewal never happens and nothing alerts. The fix on VyOS is to make port 80 reachable at renewal time; DNS-01 is not an option the CLI offers, so the alternative is to issue off-box and import the result.
• Internal CA compromised. The CA private key is stolen. Every certificate it signed is compromised. The fix: revoke the CA, reissue every certificate, update every trust store.
• Wrong certificate installed. The operator installed the new certificate under a new name but never changed set service https certificates certificate <name>, so the service still presents the old one. show pki certificate looks healthy — it lists the new certificate — while the wire says otherwise. The fix: check which name the service references, not which certificates exist.
• Certificate chain incomplete. The certificate is signed by an intermediate CA, but the intermediate CA is not in the trust store. The fix: install the complete chain.
• Certificate rotation during maintenance window. The certificate is rotated during a maintenance window, but the new certificate has a typo in the subject. The service fails to start. The fix: test the certificate in a staging environment first.

Rollback

Because the certificate is configuration, the recovery from a bad rotation is a configuration recovery:

• The commit that repointed the service broke it. commit-confirm 5 reverts on its own if you never confirm. This is why the rotation procedure uses it.
• You already committed and saved. rollback 1 then commit restores the previous revision — including the previous certificate, because the certificate is in the revision.
• You want a surgical revert. Point the service back at the still-installed old entry: set service https certificates certificate router-api and commit. This is only possible because step 1 of the rotation created the new certificate under a new name instead of overwriting.
• The new certificate is fine but the chain is not. Add the issuer: set service https certificates ca-certificate <ca-name>. A client rejecting an otherwise valid certificate is usually a missing intermediate, not a bad leaf.

Production discipline

Cross-course references

• XLVII-VyOS-MgmtPlane (vyos-xlvii-02-api-auth) covers the HTTP API authentication that uses these certificates.
• XLI-VyOS-WireGuard covers the WireGuard public key cryptography that complements this lesson.
• XLII-VyOS-IPsec covers the IPsec IKEv2 certificate authentication.
• LXXI-Linux-TLS (Linux course) covers the underlying TLS certificate management.

Quiz