Certificate rotation and renewal — the operational discipline

Certificate rotation and renewal is the operational discipline of certificate management. This lesson walks cert-manager’s automatic renewal, the failure modes, the rotation strategy, and the discipline.

cert-manager automatic renewal

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: myapp-cert
spec:
  secretName: myapp-tls
  duration: 2160h  # 90 days
  renewBefore: 720h  # renew 30 days before expiry
  issuerRef:
    name: letsencrypt-prod

cert-manager monitors each certificate and renews it before renewBefore. The renewal is automatic; no human intervention required.

The renewal flow:

1. cert-manager checks the certificate’s expiry against renewBefore.
2. If within the renewal window, cert-manager requests a new certificate from the Issuer.
3. The Issuer returns the new certificate.
4. cert-manager updates the Secret with the new material.
5. The kubelet syncs the Secret to the Pod’s volume.

The renewal flow

flowchart LR
    A[cert-manager checks expiry] --> B{Within renewBefore?}
    B -->|No| C[No action]
    B -->|Yes| D[Request new cert]
    D --> E{Success?}
    E -->|Yes| F[Update Secret]
    E -->|No| G[Retry with backoff]
    G --> D
    F -->|kubelet sync| H[Pod volume updated]

The renewal flow:

1. cert-manager periodically checks each certificate’s expiry.
2. Within the renewal window, cert-manager requests a new certificate.
3. If successful, the Secret is updated.
4. The kubelet syncs the Secret to the Pod’s volume (eventually consistent).
5. If the request fails, cert-manager retries with exponential backoff.

The failure modes

flowchart LR
    A[Failure modes] --> B[ACME rate limits]
    A --> C[CA key compromised]
    A --> D[DNS propagation]
    A --> E[Network egress blocked]
    A --> F[Issuer misconfigured]

The failure modes:

• ACME rate limits. Let’s Encrypt limits certificates per domain per week; renewal bursts can exhaust the quota.
• CA key compromised. The CA’s private key is compromised; every certificate signed by it must be revoked.
• DNS propagation. A dns-01 challenge requires DNS propagation; slow propagation delays the renewal.
• Network egress blocked. NetworkPolicy blocks ACME traffic; renewal fails.
• Issuer misconfigured. The Issuer’s configuration is wrong (expired account, mis-typed email).

The rotation strategy

flowchart LR
    A[New cert issued] --> B[Old cert still valid]
    B --> C[Overlap period]
    C --> D[Application uses new cert]
    D --> E[Old cert expires]

The rotation strategy:

1. Overlap period. The new certificate is available before the old expires. Default with Let’s Encrypt: 60 days overlap (90-day cert, renewBefore 30 days).
2. Application reload. The application hot-reloads the certificate without restart.
3. Canary. A canary deployment validates the new certificate before full rollout.

The overlap period is the safety margin; if the new certificate is broken, the old certificate is still valid for a while.

The monitoring

certmanager_certificate_expiration_timestamp_seconds - time() < 86400 * 14

Alert when a certificate expires within 14 days. This is a backup; cert-manager should renew automatically before the alert fires.

sum by (namespace, name) (certmanager_certificate_ready_status{condition="True"})

Alert when no certificate is ready (renewal failed).

Quiz

The operational discipline

Certificate rotation in production rests on five non-negotiable elements:

• renewBefore 30 days. Generous safety margin.
• Overlap period. New cert before old expires.
• Hot-reload support. Application reloads without restart.
• Monitor expiry. Alert on certs expiring soon.
• Test renewal quarterly. Verify cert-manager renews correctly.

Certificate rotation is production TLS management. The discipline is automatic renewal with monitoring and testing.