controller-manager loops — the cluster's reconciliation

When the controller-manager stops, the cluster does not return errors — it simply stops acting. kubectl apply still succeeds and the API server still accepts every write, and then nothing happens: Deployments do not roll, deleted Pods are not replaced, and EndpointSlices keep pointing at Pods that no longer exist. The failure is silent by construction, and the two causes that produce it most often are a lost leader election and API server rate limiting that throttles the loops without ever stopping the process.

The controller-manager’s role

The controller-manager runs the cluster’s core controllers: the ReplicaSet controller, the Deployment controller, the Node controller, the ServiceAccount controller, the EndpointSlice controller, and many more.

flowchart LR
    A[API server] --> B[controller-manager]
    B --> C[ReplicaSet controller]
    B --> D[Deployment controller]
    B --> E[Node controller]
    B --> F[ServiceAccount controller]
    B --> G[EndpointSlice controller]

The controller-manager is the cluster’s reconciliation.

The diagnostic

The canonical diagnostic:

# Substitute your own values before running:
CP_NODE=192.0.2.11                              # a control-plane node
KCM_POD=kube-controller-manager-cp-01           # its static Pod

# 1. Check the controller-manager's health
kubectl get --raw /api/v1/componentstatuses kube-controller-manager

# 2. Check the controller-manager's logs
kubectl logs -n kube-system -l component=kube-controller-manager --tail=200

# 3. Check the controller-manager's process
ssh "$CP_NODE" "ps aux | grep kube-controller-manager"

# 4. Check the controller-manager's metrics
kubectl port-forward -n kube-system "$KCM_POD" 10257:10257
# Browse to http://localhost:10257/metrics

The diagnostic is the controller-manager’s health, the logs, the process, and the metrics.

Common failures

• Controller-manager is down. The controller-manager container has crashed. The remediation is to restart the static Pod by moving its manifest aside and back.
• Leader election failing. The controller-manager cannot become the leader. The remediation is to fix the election.
• Controller loop is failing. A specific controller is failing. The remediation is to investigate the controller.
• Rate limiting. The controller-manager is being rate-limited by the API server. The remediation is to reduce the controller’s rate.

flowchart TD
    A[Controller-manager failing] --> B{Process running?}
    B -->|No| C[Restart the controller-manager]
    B -->|Yes| D{Leader election OK?}
    D---|No| E[Fix the election]
    D---|Yes| F{Controller loop OK?}
    F -->|No| G[Investigate the controller]
    F -->|Yes| H{Rate limited?}
    H -->|Yes| I[Reduce the rate]
    H -->|No| J[Unknown]

The rate limiting

The controller-manager’s rate limiting is a common production issue. The API server applies a default rate limit (50 QPS, 200 burst) to the controller-manager’s service account. If the controller-manager exceeds the limit, the API server returns 429 Too Many Requests.

# Check the rate limiting
kubectl logs -n kube-system -l component=kube-controller-manager | grep -i "rate"

The remediation is to increase the rate limit or reduce the controller’s rate.

The remediation

The remediation depends on the cause:

# Substitute the controller named in the logs before running:
CONTROLLER=endpointslice

# Option 1: Restart the controller-manager.
# kubeadm runs it as a static Pod, so move the manifest out of
# the kubelet's watch directory and back again.
mv /etc/kubernetes/manifests/kube-controller-manager.yaml /tmp/
# wait for the kubelet's fileCheckFrequency, 20s by default
mv /tmp/kube-controller-manager.yaml /etc/kubernetes/manifests/

# Option 2: Investigate the failing controller
kubectl logs -n kube-system -l component=kube-controller-manager | grep -i "$CONTROLLER"

# Option 3: Increase the rate limit
# Edit the controller-manager's flags
# --kube-api-qps=100 --kube-api-burst=200

# Option 4: Investigate the leader election
kubectl logs -n kube-system -l component=kube-controller-manager | grep -i "leader"

The remediation is the controller-manager recovery.

Production discipline

A controller-manager failure is the cluster’s hypothesis. The discipline is to walk the 11-step methodology applied to the control plane, identify the cause, apply the remediation. The controller-manager is the cluster’s reconciliation; the remediation is the controller-manager recovery.

• Check the controller-manager’s logs. The logs are the controller-manager’s voice.
• Check the rate limit. The rate limit is the hidden failure.

Quiz