HPA anti-patterns — the pitfalls to avoid

The HPA has a set of anti-patterns that production operators must avoid. Scaling on memory, custom metrics that are not pod-level, mixing HPA with VPA, missing resource requests, and scaling on inappropriate metrics are the common pitfalls. This lesson walks the anti-patterns and the production discipline.

The anti-patterns

flowchart LR
    A[HPA anti-patterns] --> B[Scaling on memory]
    A --> C[Custom metrics not pod-level]
    A --> D[Mixing HPA with VPA]
    A --> E[Missing resource requests]
    A --> F[Scaling on inappropriate metrics]

Each anti-pattern is a failure mode.

Anti-pattern: scaling on memory

metrics:
- type: Resource
  resource:
    name: memory
    target:
      type: Utilization
      averageUtilization: 80

Memory utilization is volatile. Memory is not freed deterministically; the OS may hold the memory for caching or future allocations. Scaling on memory causes unnecessary scale-up.

flowchart LR
    A[Memory utilization] --> B{Volatile?}
    B -->|yes| C[OS caches, heap]
    B -->|no| D[Application allocations]
    C --> E[Scale-up when not needed]
    D --> F[Scale-up correctly]

The mitigation is to scale on CPU or a custom metric.

Anti-pattern: custom metrics not pod-level

metrics:
- type: Object
  object:
    metric:
      name: queue_depth
    target:
      type: Value
      value: "100"

The metric is not pod-level. The HPA cannot compute the desired replicas per pod. The HPA may not scale correctly.

The mitigation is to use pod-level metrics:

metrics:
- type: Pods
  pods:
    metric:
      name: http_requests_per_second
    target:
      type: AverageValue
      averageValue: "1000"

Anti-pattern: mixing HPA with VPA

The HPA scales pods; the VPA (Part LXXXIII) scales resources. Mixing them causes conflicts:

# HPA on CPU
metrics:
- type: Resource
  resource:
    name: cpu
    target:
      type: Utilization
      averageUtilization: 70

# VPA on the same workload
spec:
  resourcePolicy:
    containerPolicies:
    - containerName: app
      maxAllowed:
        cpu: 2
        memory: 4Gi

The VPA may change the resource requests, which changes the CPU utilization, which triggers the HPA, which may trigger the VPA again. The feedback loop is unstable.

The mitigation is to use either HPA or VPA, not both.

Anti-pattern: missing resource requests

# Deployment without resource requests
spec:
  containers:
  - name: app
    image: app:1.0.0

The HPA cannot compute the CPU utilization as a percentage; it needs the resource request to calculate the utilization.

The mitigation is to always set the resource requests:

spec:
  containers:
  - name: app
    image: app:1.0.0
    resources:
      requests:
        cpu: 100m
        memory: 128Mi

Anti-pattern: scaling on inappropriate metrics

Some metrics are inappropriate for HPA:

# Bad: scaling on error rate
metrics:
- type: Pods
  pods:
    metric:
      name: http_error_rate
    target:
      type: AverageValue
      averageValue: "0.01"

Error rate is a signal, not a scaling trigger. Scaling on error rate causes HPA to scale up when the workload is failing (the opposite of what is wanted).

The mitigation is to scale on metrics that indicate load (e.g., CPU utilization, request rate, queue depth).

Anti-pattern: HPA on Deployments without PodDisruptionBudget

# HPA on a Deployment that has no PDB
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: nginx

The HPA may scale down to 0 replicas during a brief metric drop. Without a PDB, the application’s availability is not protected.

The mitigation is to set a minReplicas on the HPA:

spec:
  minReplicas: 3

Or to use a PodDisruptionBudget:

# PDB
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: nginx-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: nginx

The production discipline

The production discipline of HPA design:

1. Define the appropriate metric. CPU or pod-level custom metric.
2. Set the resource requests. The HPA needs the requests to compute the utilization.
3. Set the min/max replicas. Bound the scaling.
4. Configure the behavior block. Stabilization window and scaling policies.
5. Test the HPA. Use load generation to verify the scaling.
6. Monitor the HPA metrics. Prometheus exposes them.
7. Document the HPA config. The target, the min/max, the rationale.

flowchart LR
    A[Define metric] --> B[Set resource requests]
    B --> C[Set min/max replicas]
    C --> D[Configure behavior]
    D --> E[Test the HPA]
    E --> F[Monitor]
    F --> G[Document]

Cross-course references

• The VPA course (Part LXXXIII) covers the vertical alternative.
• The Prometheus course covers custom metrics.
• The Observability course covers HPA monitoring.

Quiz

Production discipline

• Scale on CPU or pod-level custom metrics. Avoid memory.
• Set resource requests. Always.
• Set minReplicas. Avoid scaling to zero.
• Don’t mix HPA and VPA. Choose one.
• Add a PDB. Protect availability.
• Test the HPA. Generate load; verify the scaling.
• Document the HPA config. The target, the min/max, the rationale.

The HPA anti-patterns are the everyday pitfalls. Operating it well is avoiding the anti-patterns and designing the HPA correctly.