kubelet credentials and rotation — TLS, tokens, and certificates

The kubelet has three sets of credentials: the client certificate (used to authenticate to the API server), the serving certificate (used to authenticate the kubelet’s HTTPS endpoint to the API server and to operators), and the bootstrap token (used to obtain the initial certificate). Each has a lifetime; each has a rotation mechanism. This lesson walks the credentials, the rotation flows, and the operational patterns for keeping the kubelet healthy.

The three credential sets

The kubelet’s credentials:

flowchart LR
    A[kubelet] -->|uses| B[Client cert<br/>kubelet-client.crt]
    A -->|serves| C[Serving cert<br/>kubelet-server.crt]
    A -->|can request| D[Bootstrap token]
    B -->|authenticates to| E[API server]
    C -->|authenticates to| F[kubectl exec, logs]
    D -->|obtains| B

• Client certificate (kubelet-client.crt): the kubelet’s identity when calling the API server. The certificate is signed by the cluster’s certificate authority (CA). The Common Name (CN) is system:node:<node-name>; the Organization (O) is system:nodes.
• Serving certificate (kubelet-server.crt): the kubelet’s identity when serving HTTPS. The certificate is signed by the cluster’s CA. The certificate’s Subject Alternative Name (SAN) includes the node’s IP addresses and DNS names.
• Bootstrap token: a short-lived token used to obtain the initial client certificate. The token is created by the cluster’s bootstrap process (kubeadm) and is invalidated after the certificate is issued.

The three credentials are stored in the kubelet’s --cert-dir (default /var/lib/kubelet/pki).

The client certificate

The client certificate authenticates the kubelet to the API server. The certificate’s lifetime is 1 year by default in kubeadm-managed clusters.

kubelet --cert-dir=/var/lib/kubelet/pki

The certificate is rotated by the kubelet’s --rotate-certificates flag. The kubelet creates a new CSR when the certificate is close to expiry. The cluster’s CSR signing controller approves the CSR and returns the new certificate.

The kubelet automatically rotates the client certificate when the rotation is enabled. The cluster’s kubelet service account must have the certificatesigningrequests sub-resource permission to create the CSR.

sequenceDiagram
    autonumber
    participant K as kubelet
    participant API as API server
    participant CSR as CSR controller

    K->>K: certificate expires in 30 days
    K->>API: POST CSR
    API->>CSR: pending CSR
    CSR->>CSR: approve (if valid)
    CSR-->>K: new certificate
    K->>K: write to cert-dir
    K->>API: use new certificate

The CSR signing is asynchronous. The kubelet continues to use the old certificate until the new one is issued. The rotation window is the cluster’s grace period.

The serving certificate

The serving certificate authenticates the kubelet’s HTTPS endpoint. The certificate is used by:

• kubectl exec — the API server proxies the exec request to the kubelet’s HTTPS endpoint.
• kubectl logs — the API server proxies the log request to the kubelet’s HTTPS endpoint.
• kubectl port-forward — the API server proxies the port-forward request to the kubelet’s HTTPS endpoint.
• kubectl run — the API server proxies the run request to the kubelet’s HTTPS endpoint.

The serving certificate’s lifetime is 1 year by default. The kubelet self-signs the certificate if the cluster does not provide a CA. The cluster’s control plane authenticates the kubelet by checking the certificate’s CA.

The serving certificate is rotated by the kubelet’s self-signed rotation or by the cluster’s CA-issued rotation. The kubelet opens the renewal between 70% and 90% of the certificate’s lifetime, so a CSR left unapproved still leaves weeks before the endpoint breaks.

The bootstrap token

The bootstrap token is a short-lived token used to obtain the initial client certificate. The token is created by kubeadm during the cluster bootstrap:

kubeadm token create --print-join-command

The token is written to the kubelet’s --bootstrap-kubeconfig flag. The kubelet uses the token to authenticate to the API server and request a CSR.

The token is invalidated after the CSR is approved. The kubelet writes the issued certificate to the cert-dir; the token is no longer needed.

The bootstrap token is a single-use token. After the kubelet has been bootstrapped, the token is invalidated. A new token is required for a new node.

The CSR signing

The cluster’s CSR signing is handled by the csr- signing controller in the kube-controller-manager. The controller:

1. Watches the certificatesigningrequests API.
2. Filters the CSRs by signer (kubernetes.io/kube-apiserver-client-kubelet, kubernetes.io/kubelet-serving).
3. Validates the CSR’s identity (the CN is system:node:<node-name>).
4. Approves the CSR (signs the certificate).
5. Returns the signed certificate to the kubelet.

The controller’s approval is automatic. The cluster’s bootstrapping must enable the csr-signing controller (kubeadm does this by default).

The kubelet’s identity

The kubelet’s identity is the client certificate’s CN and O. The CN is system:node:<node-name>; the O is system:nodes. The cluster’s RBAC uses the identity to grant the kubelet permissions:

• system:nodes group can create and update Node objects.
• system:nodes group can read Pods.
• system:nodes group can create and update Pod status.

The cluster’s RBAC is configured by the kubeadm bootstrap. The system:nodes group is bound to the system:node ClusterRole.

The kubelet’s authentication to the API server

The kubelet authenticates to the API server via TLS client authentication. The kubelet presents the client certificate; the API server validates the certificate against the cluster’s CA.

The kubelet’s --anonymous-auth flag controls whether the kubelet accepts anonymous requests. The default is true; the cluster’s RBAC may restrict the anonymous requests.

The kubelet’s --authorization-mode flag controls the kubelet’s authorization mode. The default is AlwaysAllow; the production setting is Webhook with the cluster’s RBAC.

The kubelet’s serving authentication

The kubelet’s serving endpoint is authenticated by the serving certificate. The API server validates the certificate against the cluster’s CA.

The kubelet’s --tls-cert-file and --tls-private-key-file flags specify the serving certificate. The default is the kubelet’s self-signed certificate.

The kubelet’s --rotate-server-certificates flag enables the cluster’s CA-issued rotation. The CSR is created with the signer kubernetes.io/kubelet-serving.

The credential lifecycle

The credential lifecycle:

The kubelet’s credentials are long-lived. The cluster operator should monitor the expiration and rotate at 75% of the TTL.

The diagnostic

A kubelet that is failing authentication is a kubelet whose credentials are expired or invalid. The diagnostic:

journalctl -u kubelet | grep -E "401|403|certificate"

The kubelet logs the error. The fix is to:

• Rotate the client certificate.
• Rotate the serving certificate.
• Verify the cluster’s CA is correct.
• Verify the cluster’s RBAC is correct.

kubectl get csr

The CSR list shows pending and approved CSRs. A pending CSR that is not approved is a kubelet that is waiting for the CSR to be signed.

The kubelet’s certificates in the kubelet’s filesystem

The kubelet’s certificates are stored in the --cert-dir (default /var/lib/kubelet/pki):

ls /var/lib/kubelet/pki

kubelet-client-2026-01-01-00-00-00.pem
kubelet-client-current.pem
kubelet-server-2026-01-01-00-00-00.pem
kubelet-server-current.pem

The current.pem symlink points to the latest certificate. The kubelet reads the current certificate on every authentication.

The kubelet’s bootstrap kubeconfig

The bootstrap kubeconfig is stored in /etc/kubernetes/bootstrap-kubelet.conf. The file contains the bootstrap token and the API server’s URL.

The kubelet reads the file on startup. The token is used to authenticate the CSR request. After the CSR is approved, the kubelet uses the issued certificate (kubelet.conf).

cat /etc/kubernetes/kubelet.conf

apiVersion: v1
kind: Config
clusters:
  - cluster:
      certificate-authority: /etc/kubernetes/ca.crt
      server: https://api.example.com:6443
    name: default
contexts:
  - context:
      cluster: default
      user: system:node:node-1
    name: default
current-context: default
users:
  - name: system:node:node-1
    user:
      client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem
      client-key: /var/lib/kubelet/pki/kubelet-client-current.pem

The file is the kubelet’s runtime identity. The kubelet reads it on every authentication.

Quiz

Production discipline

• Monitor the kubelet’s certificate expiry. A kubelet whose certificate has expired is a node that is failing. The kubelet’s metrics expose the certificate’s NotAfter date.
• Enable certificate rotation. The --rotate-certificates flag enables the rotation; the cluster’s CSR signing controller must approve the CSR.
• Audit the CSR signing controller. A controller that is not running is a controller that is not approving CSRs. The kubelet’s rotation will fail.
• Restrict the kubelet’s API. The kubelet’s API should not accept anonymous requests in production. The --anonymous-auth=false flag and the --authorization-mode=Webhook enforce the restriction.
• Use the cluster’s bootstrap automation. The kubelet’s credentials should be issued by the cluster’s CSR signing controller, not by a human. The production rule is to use kubeadm or the cluster’s bootstrap automation.
• Audit the kubelet’s credentials at every release. A kubelet whose credentials are stale is a node that is failing silently. The audit catches the staleness.