Node registration — how a node joins the cluster

A node joins the cluster through three stages: the kubelet’s TLS bootstrap, the API server’s admission of the Node object, and the node controller’s validation of the node’s status. The three stages are independent; each has a distinct failure mode. This lesson walks the registration flow and the diagnostics for each stage.

The TLS bootstrap

The kubelet starts with a bootstrap token or a static credential. The bootstrap token is a short-lived bearer token that allows the kubelet to authenticate to the API server for the initial certificate signing.

kubelet \
  --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf \
  --kubeconfig=/etc/kubernetes/kubelet.conf \
  --cert-dir=/var/lib/kubelet/pki

The bootstrap kubelet.conf contains the token. The kubelet uses the token to authenticate to the API server and request a client certificate. The API server’s certificatesigningrequests controller signs the certificate and returns it to the kubelet.

The kubelet writes the certificate to the --cert-dir. The kubelet then uses the certificate for subsequent authentication.

sequenceDiagram
    autonumber
    participant K as kubelet
    participant API as API server
    participant CSR as CSR controller
    participant CA as CA

    K->>API: POST /nodes (bootstrap token)
    API-->>K: 401 Unauthorized + token review
    K->>API: POST /nodes (token)
    API->>API: authenticate token
    API-->>K: 201 Created (Node object)
    K->>API: create CSR
    API->>CSR: pending CSR
    CSR->>CA: sign certificate
    CSR-->>K: certificate issued
    K->>API: PATCH /nodes/status (with new cert)

The CSR flow is async. The kubelet creates a CSR; the controller signs it. The kubelet then uses the signed certificate for subsequent API calls.

The Node object creation

The kubelet creates the Node object with the node’s address, capacity, and labels. The API server validates the request and creates the object.

# Substitute your own node name before running:
NODE=node-1

kubectl get node "$NODE" -o yaml | head -30

apiVersion: v1
kind: Node
metadata:
  name: node-1
  labels:
    kubernetes.io/hostname: node-1
    node.kubernetes.io/instance-type: m5.large
    topology.kubernetes.io/zone: us-east-1a

The labels are set by the kubelet from the node’s environment. The cloud provider’s node lifecycle controller adds the cloud-specific labels (node.kubernetes.io/instance-type, topology.kubernetes.io/zone).

The API server’s authentication layer checks the kubelet’s credentials. The kubelet’s client certificate is the identity; the system:masters group is the typical authorization for the bootstrap. After the bootstrap, the kubelet is authorized to create and update the Node object via the system:nodes group.

The cloud provider’s role

In a cloud-managed cluster, the cloud provider’s node lifecycle controller (running in the cloud-controller-manager) adds the node-specific labels and addresses. The controller:

• Reads the node’s metadata from the cloud’s metadata service (instance type, zone, region).
• Adds the node’s ExternalIP if the cluster needs it.
• Adds the node.kubernetes.io/instance-type label.
• Adds the topology.kubernetes.io/zone label.
• Removes the node.cloudprovider.kubernetes.io/uninitialized taint when the controller has finished.

The cloud provider’s controller runs on the control plane, not on the node. The kubelet does not talk to the cloud provider directly.

flowchart LR
    A[kubelet] -->|Node object| B[API server]
    B --> C[Cloud controller<br/>adds labels]
    C --> D[Node controller<br/>validates]
    D --> E[Node is Ready]

The cloud-provider integration is a separate component. A cluster that runs without a cloud provider (a bare-metal or on-prem cluster) skips this stage; the kubelet sets the labels directly.

The node controller’s validation

The node controller in the kube-controller-manager watches the Node objects. When a new Node object is created, the controller:

1. Assigns a Pod CIDR to the node (if the CNI uses per-node CIDRs).
2. Validates the node’s addresses.
3. Sets the NetworkUnavailable condition to False once the CNI has configured the node.
4. Removes the node.cloudprovider.kubernetes.io/uninitialized taint if the cloud provider has finished.

The node controller’s validation is the cluster’s endorsement of the node. Until the node controller has validated, the node is not a candidate for new Pods.

The Pod CIDR assignment

A cluster that uses per-node CIDR ranges (the default for most CNI plugins) requires the API server to assign a CIDR to each node. The assignment is performed by the node controller; the CIDR is stored in spec.podCIDR.

# Substitute your own node name before running:
NODE=node-1

kubectl get node "$NODE" -o jsonpath='{.spec.podCIDR}'

10.244.1.0/24

The CNI plugin uses the CIDR to assign Pod IPs. The CIDR is added to the node’s spec.podCIDR by the node controller; the kubelet uses the CIDR after the node controller has assigned it.

The heartbeat

Once the node is registered, the kubelet starts heart-beating. The heartbeat is a lease object in the kube-node-lease namespace:

kubectl get lease -n kube-node-lease

NAME         HOLDER      AGE
node-1       node-1      5m
node-2       node-2       5m

The lease is renewed every 10 seconds by the kubelet. The node controller watches the lease; if the lease is not renewed for --node-monitor-grace-period (default 40s), the node controller marks the node NotReady.

The lease is separate from the Node object’s status. The Node.Status is updated every 10 seconds; the lease is updated every 10 seconds. The two are independent.

The registration failure modes

A node that is registered but not Ready has one of three failure modes:

1. The kubelet cannot reach the API server. The kubelet logs connection errors. The node is not yet in the cluster’s view.
2. The credentials are invalid. The kubelet logs 401 or 403. The Node object is created if the token is valid but the subsequent calls fail.
3. The CNI plugin is not configured. The kubelet reports NetworkUnavailable: True. The node controller waits for the CNI to report ready.

The diagnostic:

journalctl -u kubelet | tail -50

kubelet.go:1234] Starting kubelet
...
kubelet_node_status.go:70] "Setting node to NotReady" condition="NetworkUnavailable"

The log line tells the operator which condition is blocking the node.

The kubelet’s failure to authenticate

The kubelet’s credentials have a 1-year TTL by default. A kubelet that has been running for a year will fail to authenticate when the certificate expires. The fix is to rotate the kubelet’s certificate before the expiry.

kubelet --rotate-certificates

The kubernetes-csr-approver controller (or the cluster’s external signer) approves the rotation request. The kubelet then uses the new certificate.

The node object’s UID

The Node object’s UID is a UUID generated by the API server. The UID is unique to the Node object; it is re-generated when the Node object is deleted and re-created. A replacement node that joins the cluster has a different UID.

The UID is used by the cluster’s controllers to track the node’s state. The kubelet’s identity is the certificates, not the UID.

Quiz

Production discipline

• The kubelet’s certificate must be rotated. A cluster that does not rotate kubelet certificates will fail at the certificate’s expiry. Monitor the expiry; rotate at 75% of the TTL.
• The cloud provider’s labels must be set. The cluster’s affinity rules depend on topology.kubernetes.io/zone and node.kubernetes.io/instance-type. A node that does not have these labels is a node that cannot be used by affinity rules.
• The Pod CIDR must be assigned. The CNI plugin needs the Pod CIDR to assign Pod IPs. A node that does not have a Pod CIDR is a node that cannot host Pods.
• The kubelet’s logs are the first place to look. A node that is not Ready has a kubelet log line that explains why. The operator should read the kubelet’s logs first, second, and third.
• Audit the node’s bootstrap. A node that joins the cluster via a manual process is a node that can be impersonated. The production rule is to use the cluster’s bootstrap automation; the kubelet’s credentials should be issued by the cluster, not by a human.