HPA — Horizontal Pod Autoscaler concepts

The Horizontal Pod Autoscaler (HPA) is the workload-level scaler. It adjusts the number of pod replicas based on metrics. The Cluster Autoscaler adjusts nodes; the HPA adjusts pods. This lesson walks the HPA, the metrics sources, the controller loop, and the production patterns.

What the HPA does

The HPA is a controller that runs in the cluster. It monitors the API server for HPA objects and adjusts the replica count of the target workload.

flowchart LR
    A[Metrics Server] --> B[metrics.k8s.io API]
    B --> C[HPA controller]
    C --> D[Target Deployment]
    D --> E[Scale up replicas]
    D --> F[Scale down replicas]

The HPA controller runs every 15 seconds (default). It queries the metrics API for the current metric value; compares against the target; and adjusts the replica count.

The HPA configuration

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: nginx-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: nginx
  minReplicas: 3
  maxReplicas: 30
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

The HPA targets a Deployment. The min/max replicas are bounded. The metric is CPU utilization at 70%.

The metrics sources

The HPA supports multiple metrics sources:

flowchart LR
    A[HPA] --> B[Resource metrics]
    A --> C[Custom metrics]
    A --> D[External metrics]
    B --> E[metrics.k8s.io API]
    C --> F[custom.metrics.k8s.io API]
    D --> G[external.metrics.k8s.io API]

• Resource metrics: CPU and memory from the metrics server.
• Custom metrics: application-specific metrics from the custom metrics API.
• External metrics: metrics from outside the cluster (e.g., SQS queue depth).

The HPA controller

The HPA is implemented as a controller in the kube-controller-manager:

HPA controller:
  - Periodically query metrics API
  - Compute desired replica count
  - Update Deployment's spec.replicas
  - Deployment controller updates Pods

The HPA controller does not directly create pods; it updates the Deployment’s replica count, and the Deployment controller does the rest.

The metrics.k8s.io API

The metrics.k8s.io API is provided by the metrics server:

kubectl get --raw=/apis/metrics.k8s.io/v1beta1/pods

{
  "kind": "PodMetricsList",
  "apiVersion": "metrics.k8s.io/v1beta1",
  "items": [
    {
      "metadata": {"name": "nginx-1-abc", "namespace": "default"},
      "containers": [
        {
          "name": "nginx",
          "usage": {"cpu": "100m", "memory": "128Mi"}
        }
      ]
    }
  ]
}

The metrics server provides CPU and memory usage per pod.

The custom metrics API

The custom metrics API is provided by an adapter (e.g., Prometheus Adapter):

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: nginx-hpa
spec:
  metrics:
  - type: Pods
    pods:
      metric:
        name: http_requests_per_second
      target:
        type: AverageValue
        averageValue: "1000"

The custom metric http_requests_per_second is queried from the custom metrics adapter.

The controller loop

The HPA controller runs every 15 seconds:

sequenceDiagram
    participant HPA as HPA controller
    participant M as Metrics API
    participant D as Deployment
    loop every 15s
        HPA->>M: get current metric
        M-->>HPA: current value
        HPA->>HPA: compute desired replicas
        HPA->>D: update spec.replicas
        D->>D: scale Deployment
    end

The controller loop is the HPA’s heartbeat.

The HPA example

# Create the HPA
kubectl apply -f nginx-hpa.yaml

# Check the HPA
kubectl get hpa

NAME        REFERENCE          TARGETS   MINPODS   MAXPODS   REPLICAS
nginx-hpa   Deployment/nginx   50%/70%   3         30        5

The output shows the current CPU utilization, the target, the min/max replicas, and the current replica count.

The kubectl autoscale

The kubectl autoscale command creates an HPA:

kubectl autoscale deployment nginx --cpu-percent=70 --min=3 --max=30

The command creates an HPA with CPU utilization target.

Cross-course references

• The Observability course covers metrics server setup.
• The Prometheus course covers custom metrics adapter.
• The VPA course (next part) covers the vertical alternative.

Quiz

Production discipline

• Configure the HPA with the right metrics. CPU, memory, or custom.
• Set the min/max replicas. Bound the scaling.
• Test the HPA. Use load generation to verify the scaling.
• Monitor the metrics. Prometheus exposes HPA metrics.
• Tune the target utilization. 70% is typical; tune for the workload.
• Document the HPA config. The target, the min/max, the rationale.

The HPA is the workload’s elastic scaling. Operating it well is configuring the metrics, the target, and the bounds.