CPU throttling and limits — the compute bottleneck

CPU limits are enforced by the CFS scheduler in 100ms slices rather than as a smooth average, so a container reporting 30% utilisation can still spend a third of every period frozen. That is why kubectl top is the wrong instrument here: it shows usage comfortably under the limit while the application’s latency doubles, and no Pod event mentions it. The number that tells the truth is nr_throttled in the container’s cpu.stat, and this lesson covers reading it, deciding whether the limit or the workload is at fault, and what removing the limit altogether actually costs.

The CPU throttling

The CPU throttling is the cgroup’s response to a container that has exceeded its CPU limit. The cgroup uses the CFS (Completely Fair Scheduler) quota to limit the CPU.

flowchart LR
    A[Container] --> B{CPU limit reached?}
    B -->|Yes| C[Throttle]
    B -->|No| D[Run]
    C --> E[Wait for next period]
    E --> D

The throttling is the cgroup’s bottleneck.

The diagnostic

The canonical diagnostic:

# Substitute your own values before running:
POD=checkout-api-5f9c7d8b6c-2xk9p
NS=production

# 1. Check the CPU throttling
kubectl exec -it "$POD" -n "$NS" -- cat /sys/fs/cgroup/cpu.stat

# 2. Check the CPU usage
kubectl top pod "$POD" -n "$NS"

# 3. Check the CPU limits
kubectl describe pod "$POD" -n "$NS" | grep -A5 Limits

# 4. Check the application's metrics
kubectl port-forward -n "$NS" "$POD" 8080:8080
# Browse to http://localhost:8080/metrics

# 5. Check the events
kubectl get events -n "$NS" --field-selector involvedObject.name="$POD"

The diagnostic is the CPU throttling, the CPU usage, the limits, and the application’s metrics.

Common failures

• CPU limit too low. The CPU limit is too low. The remediation is to increase the CPU limit.
• CPU request too low. The CPU request is too low; the application is scheduled on a node with insufficient CPU. The remediation is to increase the CPU request.
• CPU-intensive code. The application has a CPU-intensive loop. The remediation is to optimise the application.
• Too many replicas on one node. The node is over-saturated. The remediation is to scale the cluster.

flowchart TD
    A[CPU throttling] --> B{CPU limit low?}
    B -->|Yes| C[Increase the CPU limit]
    B -->|No| D{CPU request low?}
    D -->|Yes| E[Increase the CPU request]
    D---|No| F{CPU-intensive code?}
    F -->|Yes| G[Optimise the application]
    F -->|No| H{Node oversaturated?}
    H -->|Yes| I[Scale the cluster]
    H -->|No| J[Unknown]

The CPU limit best practice

The CPU limit best practice is to set the CPU request to the expected steady-state CPU and the CPU limit to the burst-allowance CPU. The default CPU limit is 4x the CPU request.

resources:
  requests:
    cpu: 100m
  limits:
    cpu: 400m

The CPU limit is the cgroup’s boundary.

The remediation

The remediation depends on the cause:

# Substitute your own values before running:
DEPLOY=checkout-api
NS=production
NODEPOOL=workers-a
NODEPOOL_REPLICAS=6

# Option 1: Increase the CPU limit
kubectl set resources deployment "$DEPLOY" -n "$NS" --limits=cpu=1

# Option 2: Remove the CPU limit
kubectl set resources deployment "$DEPLOY" -n "$NS" --limits=cpu=

# Option 3: Optimise the application
# (application-specific)

# Option 4: Scale the cluster
# `nodepool` is not a core Kubernetes resource: substitute your provider's
# node-group API (Karpenter NodePool, Cluster API MachineDeployment) or its
# cloud CLI.
kubectl scale nodepool "$NODEPOOL" --replicas="$NODEPOOL_REPLICAS"

The remediation is the compute recovery.

Production discipline

A CPU throttling issue is the cluster’s hypothesis. The discipline is to walk the 11-step methodology applied to the compute layer, identify the cause, apply the remediation. The compute is the cluster’s capacity; the remediation is the compute recovery.

• Check the CPU throttling. The throttling is the cgroup’s bottleneck; nr_throttled in the container’s cpu.stat is the number that tells the truth, not kubectl top.
• Check the CPU limits. The limits are the workload’s boundary.
• Check the CPU requests. The requests are the workload’s reservation.

Quiz