Certificate renewal and expiry - the lifecycle discipline

Certificate renewal is the most common operational TLS task. This lesson covers monitoring, automation, and the production discipline.

Monitor expiry

# Check a single cert
openssl x509 -enddate -noout -in server.crt
# notAfter=Dec 31 23:59:59 2026 GMT

# Check all certs in a directory
# /etc/ssl/certs holds certificates; /etc/ssl/private holds keys and is mode 0700
for cert in /etc/ssl/certs/*.crt; do
    [ -f "$cert" ] || continue
    if openssl x509 -checkend 2592000 -noout -in "$cert" >/dev/null; then
        echo "OK      $cert"
    else
        echo "EXPIRES $cert: $(openssl x509 -enddate -noout -in "$cert" | cut -d= -f2)"
    fi
done

# Exporter for Prometheus
node_exporter with textfile collector

-checkend 2592000 exits non-zero when the certificate expires within 30 days. It is a comparison the library does for you, so it handles the timezone and the date format correctly. Parsing the notAfter string yourself works until a locale or a format change makes the comparison quietly wrong, which is the kind of bug that surfaces on the day the certificate expires.

Quote "$cert". An unquoted path breaks on any directory name containing a space, and the loop then checks the wrong file or none at all.

Alert:

• 30 days before expiry: warning.
• 7 days before expiry: critical.
• Expired: emergency.

Automated renewal

Let’s Encrypt (public certificates)

# Install certbot
sudo apt install certbot

# Request a certificate
sudo certbot certonly --standalone -d example.com

# Auto-renew (cron or systemd timer)
sudo certbot renew --dry-run

Certbot sets up a cron job or systemd timer for renewal. The certificate is renewed automatically before expiry.

Internal CA

Copy the names from the certificate you are replacing first, so the renewal covers exactly what the old one covered:

openssl x509 -in server.crt -noout -ext subjectAltName

# Generate a new key and CSR, carrying the names forward
openssl genrsa -out server-new.key 2048
chmod 600 server-new.key
openssl req -new -key server-new.key -out server-new.csr \
    -subj "/CN=server.example.com" \
    -addext "subjectAltName=DNS:server.example.com"

# The CA supplies the extensions; x509 -req adds none
cat > server-ext.cnf <<'EOF'
subjectAltName = DNS:server.example.com
basicConstraints = critical,CA:FALSE
keyUsage = critical,digitalSignature,keyEncipherment
extendedKeyUsage = serverAuth
EOF

openssl x509 -req -in server-new.csr \
    -CA ca.crt -CAkey ca.key -CAcreateserial \
    -out server-new.crt -days 365 -sha256 \
    -extfile server-ext.cnf

# Validate BEFORE swapping anything in
openssl x509 -in server-new.crt -noout -ext subjectAltName
openssl verify -CAfile ca.crt \
    -verify_hostname server.example.com -purpose sslserver \
    server-new.crt

# Prove the key matches the certificate. The two hashes must be identical.
openssl x509 -noout -pubkey -in server-new.crt | openssl sha256
openssl pkey  -noout -pubout -in server-new.key | openssl sha256

Only now install it, and install it reversibly:

set -euo pipefail
TS=$(date +%FT%H%M%S)
CRT=/etc/ssl/certs/server.crt
KEY=/etc/ssl/private/server.key

# 1. Back up the working pair before anything overwrites it
sudo cp -a "$CRT" "$CRT.$TS.bak"
sudo cp -a "$KEY" "$KEY.$TS.bak"

# 2. Install with explicit owner and mode - not mv
sudo install -o root -g root     -m 0644 server-new.crt "$CRT"
sudo install -o root -g ssl-cert -m 0640 server-new.key "$KEY"
sudo restorecon -v "$CRT" "$KEY"      # RHEL family: restore cert_t

# 3. Test the config before the service reads it for real
sudo nginx -t                          # or: apachectl configtest, sshd -t

# 4. Reload rather than restart where the service supports it
sudo systemctl reload nginx

# 5. Validate from a client, not from the file
openssl s_client -connect localhost:443 -servername server.example.com \
    -verify_hostname server.example.com -verify_return_error </dev/null

Rollback, if step 3, 4 or 5 fails:

sudo cp -a "$CRT.$TS.bak" "$CRT"
sudo cp -a "$KEY.$TS.bak" "$KEY"
sudo restorecon -v "$CRT" "$KEY"
sudo nginx -t && sudo systemctl reload nginx

Note the directories. On Debian-family systems /etc/ssl/private is mode 0700, owned root:ssl-cert, and holds keys only; certificates belong in /etc/ssl/certs. Putting a certificate in the key directory hides it from anything that scans /etc/ssl/certs, including the expiry loop above.

Prefer reload over restart. nginx, Apache and HAProxy all pick up a new certificate on reload without dropping established connections. A restart drops them, which turns a routine renewal into a visible blip.

Renewal is where the missing-SAN failure bites hardest. openssl x509 -req adds no X.509 v3 extensions and drops the ones requested in the CSR, so a renewal signed without -extfile produces a CN-only certificate. It signs cleanly, openssl verify reports OK, and then every Go client (Prometheus, Consul, Vault, Docker, Kubernetes components) and every browser rejects the handshake the moment the service reloads. Run the validation commands while the new pair is still a pair of ordinary files in a working directory, not after it has replaced the live one.

Automate via configuration management (Ansible, Puppet).

Revoke compromised certificates

If a certificate’s private key is compromised:

1. Revoke: tell the CA to mark the certificate as invalid.
2. Publish CRL: the Certificate Revocation List.
3. OCSP: real-time check; clients can ask if a certificate is valid.

# Revoke with OpenSSL
openssl ca -config ca.conf -revoke server.crt -keyfile ca.key -cert ca.crt

# Generate CRL
openssl ca -config ca.conf -gencrl -keyfile ca.key -cert ca.crt -out crl.pem

Distribute the CRL to clients, or set up an OCSP responder.

Certificate lifecycle summary

• Issue: generate key, create CSR, sign.
• Deploy: install cert and key on server.
• Use: clients connect and verify.
• Monitor: alert 30 days before expiry.
• Renew: generate new cert, sign, validate the pair and the chain, back up the live pair, install with the right owner and label, config-test, reload, verify from a client. Keep the backup until the next renewal.
• Revoke: if compromised, revoke and publish CRL.

Knowledge check