Scale-down triggers — when the cluster shrinks

The Cluster Autoscaler scales down when there are underutilized nodes. The scale-down is a careful process: the node is drained, the pods are rescheduled, the node is terminated. This lesson walks the triggers, the delays, the PDB-bound scale-down, and the failure modes.

The trigger

The Cluster Autoscaler watches the cluster for underutilized nodes:

flowchart LR
    A[Node utilization] --> B{Underutilized?}
    B -->|yes| C[Mark as scale-down candidate]
    C --> D[Wait unneeded time]
    D --> E[Wait scale-down delay]
    E --> F[Drain node]
    F --> G[Terminate node]

The trigger is the node’s utilization. A node is considered underutilized when its CPU and memory utilization are below thresholds for a period of time.

The scale-down delay

The scale-down delay is the time between a node being unneeded and the scale-down being initiated:

--scale-down-delay-after-add=10m
--scale-down-delay-after-delete=10m
--scale-down-delay-after-failure=3m

The flags:

• --scale-down-delay-after-add: the time after a node is added before it can be scaled down.
• --scale-down-delay-after-delete: the time after a node is deleted before another scale-down can happen.
• --scale-down-delay-after-failure: the time after a failed scale-down before retrying.

The defaults are conservative. The delay prevents thrashing.

The unneeded time

The unneeded time is the time a node must be underutilized before it is a scale-down candidate:

--scale-down-unneeded-time=10m

The default is 10 minutes. A node that has been underutilized for 10 minutes is a candidate.

A shorter time (e.g., 5m) is more aggressive; a longer time (e.g., 30m) is more conservative.

The utilization threshold

The utilization threshold is the boundary for “underutilized”:

--scale-down-utilization-threshold=0.5

The default is 0.5. A node with CPU utilization < 50% and memory utilization < 50% is underutilized.

A higher threshold (e.g., 0.7) is more aggressive; a lower threshold (e.g., 0.3) is more conservative.

The scale-down flow

sequenceDiagram
    participant CA as Cluster Autoscaler
    participant API as API server
    participant ASG as AWS ASG
    participant EC2 as EC2
    loop every 10s
        CA->>API: list nodes
        API-->>CA: 5 nodes
        CA->>CA: worker-3 utilization < 0.5
        CA->>CA: mark worker-3 as scale-down candidate
        Note over CA: Wait unneeded time (10m)
        Note over CA: Wait scale-down delay (10m)
        CA->>API: cordon worker-3
        CA->>API: drain worker-3
        API->>API: pods rescheduled
        CA->>ASG: terminate worker-3
        ASG->>EC2: terminate instance
    end

The scale-down is a multi-step process. The cluster Autoscaler waits for the delays; drains the node; then terminates it.

The PDB-bound scale-down

The scale-down is bounded by the workload’s PDB:

Worker-3 has 5 pods from a Deployment with maxUnavailable: 1.
Cluster Autoscaler wants to drain worker-3.
The drain would cause 5 pods to be unavailable (5 - 0 = 5).
This violates maxUnavailable: 1.
The scale-down is blocked.

The autoscaler does not delete a node if the drain would violate a PDB.

flowchart LR
    A[Scale-down initiated] --> B[Check PDB]
    B -->|OK| C[Drain node]
    B -->|violated| D[Block scale-down]
    D --> E[Wait for next scale-down cycle]

The skip conditions

The Cluster Autoscaler skips a node from scale-down if:

• The node has local storage (e.g., emptyDir with data).
• The node has pods that cannot be rescheduled (e.g., a bare Pod without a controller).
• The node has annotations that exclude it (cluster-autoscaler.kubernetes.io/scale-down-disabled).

# Annotation on the node
metadata:
  annotations:
    cluster-autoscaler.kubernetes.io/scale-down-disabled: "true"

The annotation is per-node; the autoscaler does not scale-down the annotated node.

The aggressive scale-down

An aggressive scale-down (short delays, low utilization threshold) causes thrashing:

1. HPA scales pods down (no demand)
2. Cluster Autoscaler marks nodes as underutilized
3. Cluster Autoscaler drains nodes
4. Cluster Autoscaler terminates nodes
5. New pods arrive (request comes in)
6. New pods are unschedulable
7. Cluster Autoscaler launches new nodes
8. Cycle repeats

The thrashing causes availability issues. The mitigation is to use conservative delays.

flowchart LR
    A[Pods down] --> B[Nodes underutilized]
    B --> C[Scale down]
    C --> D[Pods up]
    D --> E[Pods unschedulable]
    E --> F[Scale up]
    F --> A

The monitor

The Cluster Autoscaler emits metrics:

cluster_autoscaler_nodes_count
cluster_autoscaler_unschedulable_pods_count
cluster_autoscaler_scaledown_operations_total
cluster_autoscaler_unneeded_nodes_count

The metrics are scraped by Prometheus. The unneeded_nodes_count metric is the leading indicator of a scale-down.

Cross-course references

• The Observability course covers autoscaler metrics.
• The Kubernetes events course covers pod reschedule events.
• The AWS course covers ASG scaling.

Quiz

Production discipline

• Configure the scale-down delay. Conservative defaults (10m) prevent thrashing.
• Configure the unneeded time. 10m is the default.
• Set the utilization threshold. 0.5 is the default.
• Skip nodes with local storage. The annotation excludes them.
• Monitor the scale-down metrics. Prometheus exposes them.
• Test the scale-down. Catch the thrashing before production.

The scale-down is the cluster’s cost optimization. Operating it well is using conservative delays and monitoring the thrashing.