Cooldown and balance — the multi-node-group strategy

The Cluster Autoscaler’s cooldown and balance strategies are the production tuning knobs. The cooldown prevents thrashing; the balance distributes nodes across similar groups; the expander priority selects the highest-priority group. This lesson walks the configurations and the production tuning.

The cooldown

The cooldown is the time after a node is added before the node can be scaled down:

--scale-down-delay-after-add=10m

The default is 10 minutes. The autoscaler does not scale-down a node that was added less than 10 minutes ago.

The balance-similar-node-groups

The balance strategy distributes nodes across similar node groups:

--balance-similar-node-groups=true

When multiple node groups have the same instance type and the same labels, the autoscaler balances the size of the groups.

flowchart LR
    A[Cluster Autoscaler] --> B[workers-a t3.large]
    A --> C[workers-b t3.large]
    B --> D[5 nodes]
    C --> E[5 nodes]
    A --> F[Balance: ensure even distribution]

The balance is automatic; the autoscaler detaches nodes from the larger group and lets the smaller group shrink.

The expander priority

The expander priority selects the highest-priority node group when multiple groups are candidates:

# ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-autoscaler-priority-expander
  namespace: kube-system
data:
  priorities: |-
    10:
      - .*t3.large.*
    50:
      - .*t3.xlarge.*
    90:
      - .*p3.2xlarge.*

The priority is a number; higher priority is preferred.

flowchart LR
    A[Unschedulable pods] --> B[Priority expander]
    B --> C{Group priority?}
    C -->|90| D[p3.2xlarge]
    C -->|50| E[t3.xlarge]
    C -->|10| F[t3.large]

The expander selects the highest-priority group that fits the unschedulable pods.

The expander config map

The expander priority is configured via a ConfigMap:

--expander=priority

The autoscaler reads the cluster-autoscaler-priority-expander ConfigMap.

apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-autoscaler-priority-expander
  namespace: kube-system
data:
  priorities: |-
    10:
      - .*t3.large.*
    50:
      - .*t3.xlarge.*
    90:
      - .*p3.2xlarge.*

The pattern is a regex. The priority is matched against the node group’s name or labels.

The production tuning

The production tuning of the Cluster Autoscaler:

# cluster-autoscaler Deployment
command:
- ./cluster-autoscaler
- --v=4
- --stderrthreshold=info
- --cloud-provider=aws
- --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/cluster-name=my-cluster
- --balance-similar-node-groups=true
- --skip-nodes-with-local-storage=false
- --expander=priority
- --scale-down-enabled=true
- --scale-down-delay-after-add=10m
- --scale-down-delay-after-delete=10m
- --scale-down-delay-after-failure=3m
- --scale-down-unneeded-time=10m
- --scale-down-utilization-threshold=0.5
- --max-node-provision-time=15m

The flags are the production tuning. The values are conservative defaults; the operator tunes them based on the workload.

The metrics for tuning

The Cluster Autoscaler emits metrics:

cluster_autoscaler_scaled_up_nodes_total
cluster_autoscaler_scaled_down_nodes_total
cluster_autoscaler_unschedulable_pods_count
cluster_autoscaler_unneeded_nodes_count
cluster_autoscaler_node_group_size
cluster_autoscaler_balance_evaluations_total

The metrics are scraped by Prometheus. The metrics are the input for the tuning.

The tuning workflow

flowchart LR
    A[Workload pattern] --> B[Configure cooldown]
    B --> C[Run cluster for 1 week]
    C --> D[Review metrics]
    D --> E[Adjust cooldown]
    E --> F[Validate]

The tuning is iterative. The operator observes the metrics and adjusts the cooldown based on the workload pattern.

The cross-AZ balancing

The Cluster Autoscaler balances across availability zones:

--balance-similar-node-groups=true

The autoscaler attempts to distribute nodes across AZs for high availability.

flowchart LR
    A[Cluster Autoscaler] --> B[AZ1 workers]
    A --> C[AZ2 workers]
    A --> D[AZ3 workers]
    B --> E[Distribution: AZ1=3, AZ2=3, AZ3=3]

The balance is per-AZ for similar node groups.

Cross-course references

• The Observability course covers autoscaler metrics.
• The AWS course covers ASG scaling and AZs.
• The Terraform course covers IaC-managed node groups.

Quiz

Production discipline

• Configure the cooldown. Conservative defaults prevent thrashing.
• Use balance-similar-node-groups. Even distribution across AZs.
• Configure the expander priority. Select the right group for the workload.
• Monitor the metrics. Prometheus exposes them.
• Tune the autoscaler iteratively. Based on the workload pattern.
• Document the tuning. The flags, the values, the rationale.

The cooldown and balance are the production tuning. Operating it well is configuring the cooldown, the balance, and the expander.