X.509 client certificates — the legacy default

X.509 client certificates were the first authentication mechanism Kubernetes supported. They are still in use for kubelet identity and for cluster bootstrap, but they are no longer appropriate for human users or for workloads because the credentials are long-lived, hard to rotate, and impossible to scope per request. This lesson covers how client certs work, where they fit in a modern production cluster, and how to operate them safely.

How the TLS handshake authenticates the client

When a client connects to the API server, the TLS handshake itself authenticates both parties. The API server presents its serving certificate (signed by the cluster CA); the client presents its client certificate (signed by a CA the API server trusts via --client-ca-file or the authentication-config file). The API server validates the signature and reads the certificate’s identity fields.

sequenceDiagram
    participant C as kubectl
    participant S as API server
    C->>S: ClientHello (no cert)
    S->>C: ServerHello + cert chain (cluster CA)
    C->>C: Validate cluster CA
    C->>S: Client cert + key exchange
    S->>S: Validate client cert via --client-ca-file
    S->>S: Read CN → username, O → groups
    S->>C: TLS established, request served

The validated certificate fields become the UserInfo that authorisation and admission see:

• Subject.CommonName → UserInfo.username
• Subject.Organization (each value) → UserInfo.groups
• Subject.OrganizationalUnit → not used directly, but sometimes added to groups

# Example client cert (decoded)
Subject: CN = alice
          O  = system:masters
          OU = prod-admins
# → UserInfo: {username: "alice", groups: ["system:masters"]}

Where client certs are still used

Three places in a modern cluster:

1. kubelet identity — each kubelet has a client certificate (/var/lib/kubelet/pki/kubelet-client.crt) with CN=system:node:<node-name> and O=system:nodes. The --rotate-server-certificates=true flag rotates the cert as it approaches expiry.
2. kube-apiserver ↔ etcd — mTLS with certs signed by the etcd CA. These are short-lived (90 days) and rotated by the control plane operator.
3. Bootstrap (kubeadm) — kubeadm init issues a short-lived admin cert via kubeadm certs. The operator then moves to OIDC.

Three places that are legacy and should be removed:

1. Long-lived human user certs — replaced by OIDC.
2. Long-lived ServiceAccount tokens (Secrets) — replaced by projected tokens (Part LVII lesson 3).
3. kube-apiserver --token-auth-file — replaced by TokenReview.

Generating a client cert

The traditional openssl workflow:

# Generate a private key
openssl genrsa -out alice.key 2048

# Create a CSR with the right subject
openssl req -new -key alice.key -subj "/CN=alice/O=team-a-dev" -out alice.csr

# Sign with the cluster CA
openssl x509 -req -in alice.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out alice.crt -days 365

The resulting alice.crt is valid for 365 days. The operator now has a long-lived credential. A leaked credential has 365 days of access; a fired employee has 365 days of access; the only revocation is to re-issue the CA.

# kubeconfig entry
users:
- name: alice
  user:
    client-certificate: /home/alice/.certs/alice.crt
    client-key: /home/alice/.certs/alice.key
clusters:
- name: prod
  cluster:
    certificate-authority: /home/alice/.certs/ca.crt
    server: https://api.prod.example.com:6443
contexts:
- name: alice-prod
  context:
    user: alice
    cluster: prod
    namespace: team-a

Rotation with cert-manager

For kube-apiserver ↔ etcd and for kubelet certs, rotation is handled by the control plane. For human users, the right answer is to switch to OIDC (Part LVII lesson 4). If OIDC is not feasible, cert-manager can rotate short-lived (24-hour) certificates issued by an internal CA. The cert-manager Certificate resource issues a cert via the internal-ca Issuer with a short duration; the kubeconfig is updated via a renewal hook.

# cert-manager issuance for a short-lived client cert
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: alice-prod
  namespace: certs
spec:
  secretName: alice-prod-tls
  duration: 24h
  renewBefore: 8h
  issuerRef:
    name: internal-ca
    kind: ClusterIssuer
  commonName: alice
  organizations:
  - team-a-dev

Why client certs are a poor choice for humans

The structural reasons:

• No revocation — leaked certs are valid for the full duration.
• No per-request scoping — the cert has one identity; the holder can use any RBAC verb the identity has.
• No audit log enrichment — the audit log records the CN, but the CN is a name, not a corporate identity. “alice” cannot be traced to a HR record.
• No short-lived issuance — issuing a 24-hour cert requires an internal CA and a workflow; an organisation that does not have one should not use client certs.

OIDC solves each of these: tokens are short-lived, revocable by the IdP, scoped per request, and audited by the IdP.

Production failure modes

1. Long-lived human certs in production. “Alice has a 1-year cert” is a credential-management failure. The right answer is OIDC; the second-best is 24-hour cert-manager certs with revocation tied to the HR system.
2. No rotation for kubelet certs. --rotate-server-certificates=false on kubelet means kubelet certs expire and nodes go NotReady. Always enable rotation.
3. Compromised CA. The CA private key is the root of trust. If it is compromised, every cert in the cluster is compromised. The CA should be kept offline (the cert is reused, but the key is on a vaulted host).
4. O=system:masters issued for non-bootstrap. The group is bound to cluster-admin. Any cert with this group is cluster-admin. It is acceptable for the day 1 bootstrap; it is not acceptable for any other purpose.

Cross-course references

• The Linux course covers OpenSSL and the CA hierarchy that the cluster relies on.
• The Observability course covers the audit log that records every client cert connection.

Quiz

Production discipline

X.509 client certificates are an excellent credential for machines (kubelet, etcd, control plane) and a poor credential for humans. A defensible production cluster uses client certs only where rotation is automatic (kubelet) or where the CA is short-lived and rotated (etcd). Human users authenticate via OIDC. A cluster that hands a long-lived client cert to a developer on day 1 has a credential-management failure that should be fixed before any other security work. The structural absence of CRL support makes client certs a permanent fixture in the kubelet identity story, but a temporary and dangerous fixture anywhere else.