Internal ACME — automating issuance from a private certificate authority

Nothing in ACME is specific to any one certificate authority. It is an HTTPS protocol with four object types, and any CA that implements the resources can serve it, including one you run. That matters more inside an organisation than outside, because the reason internal certificates are long-lived, hand-carried and poorly inventoried is almost never a policy decision. It is that the internal CA never offered a way to get one without asking a person.

The same clients, a different directory

An ACME client is configured with a directory URL and an account key. Change the directory URL and the identical client, with the identical renewal logic and the identical deploy hooks, now talks to your CA. That is the whole migration path, and it is why internal ACME is unusually cheap: the client ecosystem was already written and hardened against a public service.

# The directory URL is a property of your CA. Keep it in
# configuration, not compiled into a client or a container image.
ACME_DIRECTORY="$(cat /etc/acme/directory-url)"
curl -sS --fail "$ACME_DIRECTORY"

Several private CA implementations expose an ACME endpoint. This course uses step-ca 0.30.x for the internal CA lab, and the reference open-source implementation of the server side is Boulder, which is what Let’s Encrypt runs; its published list of divergences from RFC 8555 is a useful reminder that server implementations differ in small ways and that a client should be tested against the server it will actually use.

Trust has to arrive before the first order

There is a bootstrap ordering problem that catches every first deployment. The ACME client talks to the directory over HTTPS, and it validates that connection against the host trust store. If the internal CA’s own endpoint is served with a certificate from the internal CA, the client cannot reach the directory until the root is already trusted. The root must therefore be distributed out of band.

flowchart LR
    A["Root distributed\nby image or config management"] --> B["Host trust store\nupdated"]
    B --> C["ACME client validates\nthe directory endpoint"]
    C --> D["Account and order"]
    D --> E["Certificate issued\nand installed"]
    E --> F["Service reload"]
    B -.->|"missing"| X["Client cannot fetch\nthe directory at all"]

The root reaches the host through the golden image or through configuration management, the host trust store is rebuilt, and only then can the client validate the directory endpoint and place an order. If that first step is skipped the client fails before any ACME object exists, which reads as a network problem and is not one.

# Install the internal trust anchor, then rebuild the store.
sudo cp internal-root.crt /usr/local/share/ca-certificates/internal-root.crt
sudo update-ca-certificates

Lifetime is a design variable again, and renewal margin is an outage budget

Publicly you take the lifetime the ecosystem gives you. Internally you choose it, and the choice is a trade between exposure after a key compromise and tolerance for your own CA being unavailable. The rule that makes the trade concrete is that the renewal margin is your outage budget.

If clients renew when a third of the lifetime remains, a 24-hour certificate gives you eight hours to restore the CA before anything expires, and a 30-day certificate gives you ten days. Pick the lifetime so that the resulting margin comfortably exceeds the worst CA recovery you can honestly promise, including the case where recovery means restoring the CA from backup at three in the morning. Teams that skip this arithmetic tend to choose very short lifetimes for good security reasons and then discover that their CA has become a service with a tighter availability requirement than anything it protects.

Internal ACME servers expose the lifetime as a profile setting. A small ACME server used for testing describes its own profiles like this:

{
  "profiles": {
    "default":    { "description": "The profile you know and love", "validityPeriod": 7776000 },
    "shortlived": { "description": "A short-lived cert profile, without actual enforcement", "validityPeriod": 518400 }
  }
}

Those periods are seconds: 7,776,000 is 90 days and 518,400 is six days. The point is not the numbers but that lifetime is a server-side knob, so shortening it is a CA configuration change rather than a fleet-wide client change.

Who is allowed to get a certificate for what

The trust boundary in ACME is simple and unforgiving: whoever can satisfy the challenge for a name receives a certificate for that name. Publicly, that is backed by CAA records, by corroboration from several network perspectives, and by the fact that hijacking a public DNS name is hard and noisy. Inside a flat corporate network, none of those hold. A host that can bind port 80 for an internal name, or poison an internal resolver, can obtain a valid internal certificate for a service it does not own.

Internal CAs answer this with something a public CA cannot offer: a per-provisioner name policy. Each enrolment path is bound to the set of names it may request, so the provisioner used by the payments fleet cannot obtain a certificate for an identity belonging to the directory service, whatever it manages to answer on the network. Three further controls are worth building in from the start:

• Prefer DNS-01 into a delegated zone for high-value names. It removes the ability to win a certificate by being fast on a port, and the credential can be scoped to one small zone.
• Segment the validation path. If only the CA can reach the validation port, and only from a known segment, spoofing requires a foothold in that segment rather than anywhere on the network.
• Log issuance as the audit record. The CA log replaces the ticket queue as the evidence of who obtained what and when, and it is complete in a way the ticket queue never was.

What breaks in the first week

An ACME-issued certificate is usually minimal. Here is one, as issued by an ACME server and read back with OpenSSL:

subject=
issuer=CN=Pebble Intermediate CA 03d090
serial=3DC74918D250C7E9
notBefore=Aug 26 21:31:04 2026 GMT
notAfter=Nov 24 21:31:03 2026 GMT

X509v3 Key Usage: critical
    Digital Signature
X509v3 Extended Key Usage:
    TLS Web Server Authentication
X509v3 Basic Constraints: critical
    CA:FALSE
X509v3 Subject Alternative Name: critical
    DNS:web.lab.example

The subject line is empty. There is no Common Name at all, and because the subject is an empty sequence the subjectAltName extension is marked critical, which RFC 5280 requires in exactly that case. Identity lives entirely in the SAN. Any internal consumer that still reads a Common Name to decide who it is talking to, and there are usually several, will fail the moment it meets one of these certificates. Find them before the migration, not during it.

Production discipline

1. Ship the trust anchor with the host, not with the app. Trust distribution belongs to provisioning, so that a host cannot exist in the estate without it.
2. Give the CA endpoint certificate a longer life than everything it issues. A CA whose own certificate expires during an outage cannot be recovered by the mechanism it provides.
3. Bind every provisioner to a name policy. Answering a challenge should not be sufficient authority for a name that belongs to another team.
4. Randomise renewal within the margin. A synchronised fleet turns a routine renewal window into a self-inflicted load test of the CA.
5. Audit consumers for Common Name matching before you migrate. An empty subject is normal for ACME issuance and fatal for a client that never learned to read the SAN.

Cross-course references

• Linux for Production Sysadmins - Part XXXIV (ConfigMgmt) covers desired state, idempotency and drift detection, which is the machinery that gets the internal trust anchor onto every host and keeps it there.
• Kubernetes for Production Sysadmins - Part CXIV (TLS) covers in-cluster ACME issuers, where the same private CA is consumed through cluster resources rather than a host client.
• Observability for Production Sysadmins - Part LXV (DNSMonitoring) covers DNS incident patterns, the failure class that lets an attacker win an internal challenge for a name they do not own.

Quiz