Scale-up triggers — when the cluster grows

The Cluster Autoscaler scales up when there are unschedulable pods. The scale-up is triggered by the API server’s pod status. This lesson walks the triggers, the scan interval, the scale-up delay, and the bin-packing algorithm.

The trigger

The Cluster Autoscaler watches the API server for unschedulable pods:

flowchart LR
    A[Pod created] --> B{Schedulable?}
    B -->|yes| C[Pod scheduled]
    B -->|no| D[Unschedulable]
    D --> E[Cluster Autoscaler detected]
    E --> F[Scale up node group]
    F --> G[New node joins]
    G --> H[Pod scheduled]

The trigger is the pod’s status. The Cluster Autoscaler queries the API server for pods with status.phase=Pending and reason=Unschedulable.

The scan interval

The scan interval is the reconciliation loop:

--scan-interval=10s

The default is 10 seconds. The Cluster Autoscaler reconciles every 10 seconds; it checks for unschedulable pods and adjusts the node count.

A shorter interval (e.g., 5s) makes the autoscaler more responsive but increases the API server load. A longer interval (e.g., 30s) is less responsive but reduces the load.

The scale-up delay

The scale-up delay is the time between an unschedulable pod being detected and the node group being scaled up:

--scale-up-from-zero=true
--max-node-provision-time=15m

For scaling from zero, the autoscaler waits for the node to be provisioned (which can take 5-15 minutes on cloud). The --max-node-provision-time is the maximum time the autoscaler will wait for a node to be ready.

The bin-packing

The Cluster Autoscaler uses bin-packing to determine the node type and the number of nodes:

Unschedulable pods:
  Pod A: 1 vCPU, 2 GB
  Pod B: 0.5 vCPU, 1 GB
  Pod C: 2 vCPU, 4 GB
  Pod D: 0.5 vCPU, 1 GB

Total: 4 vCPU, 8 GB

Option 1: 1 x t3.xlarge (4 vCPU, 16 GB) → 1 node, 50% wasted
Option 2: 2 x t3.large (2 vCPU, 8 GB each) → 2 nodes, 0% wasted
Option 3: 1 x t3.large (2 vCPU, 8 GB) + 1 x t3.medium (2 vCPU, 4 GB) → 2 nodes, some wasted

The Cluster Autoscaler uses the --expander strategy to choose between options:

• least-waste: minimizes the wasted resources.
• most-pods: maximizes the number of pods that fit.
• random: random selection.
• priority: node group priority.

flowchart LR
    A[Unschedulable pods: 4 vCPU, 8 GB] --> B{Expander?}
    B -->|least-waste| C[2 x t3.large]
    B -->|most-pods| D[1 x t3.xlarge]
    B -->|random| E[Choice may vary]

The scale-up flow

sequenceDiagram
    participant API as API server
    participant CA as Cluster Autoscaler
    participant ASG as AWS ASG
    participant EC2 as EC2
    loop every 10s
        CA->>API: list unschedulable pods
        API-->>CA: 4 unschedulable pods
        CA->>CA: bin-packing
        CA->>ASG: scale up by 2 nodes
        ASG->>EC2: launch 2 instances
        EC2-->>ASG: instances ready
        ASG-->>CA: 2 nodes joined
        CA->>API: pods scheduled
    end

The flow repeats every 10 seconds. The autoscaler is eventually consistent.

The unschedulable pod status

# A Pod stuck in Pending, from `kubectl get pods --field-selector status.phase=Pending`:
PENDING_POD=billing-7d8f-abcde

kubectl describe pod "$PENDING_POD"

Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  5m    default-scheduler  0/5 nodes are available: 5 Insufficient cpu.

The FailedScheduling event is the signal. The Cluster Autoscaler detects this and scales up.

The scale-up failure

A scale-up failure can be diagnosed from the autoscaler logs:

kubectl logs -n kube-system deploy/cluster-autoscaler

I0816 10:00:00.000000       1 scaleup.go:300] Pod nginx-1-abc is unschedulable, scaling up node group workers
I0816 10:00:00.000001       1 scaleup.go:400] Scaling up node group workers from 5 to 6
W0816 10:05:00.000000       1 scaleup.go:500] Failed to scale up node group workers: ASG scaling timeout

The ASG scaling timeout indicates the ASG did not scale within the timeout. Investigate the ASG, the IAM permissions, the cloud quotas.

The cross-course references

• The Observability course covers autoscaler metrics.
• The AWS course covers ASG scaling and IAM.
• The Terraform course covers IaC-managed node groups.

Quiz

Production discipline

• Set the scan interval appropriately. 10s is the default; tune for the workload.
• Configure the expander. least-waste is the production default.
• Monitor the autoscaler metrics. Prometheus exposes them.
• Investigate scale-up failures. Cloud quotas, IAM, ASG limits.
• Test the scale-up. Catch the bin-packing issue before production.
• Document the algorithm. The scan interval, the expander, the bin-packing.

The scale-up is the cluster’s reactive growth. Operating it well is configuring the scan interval, the expander, and the bin-packing.