Custom metrics API — application-specific scaling

The Custom Metrics API allows the HPA to scale based on application-specific metrics (e.g., HTTP requests per second, queue depth). The Metrics Server only provides CPU and memory; custom metrics require an adapter. This lesson walks the Custom Metrics API, the Prometheus Adapter, and the integration with HPA.

The Custom Metrics API

The Custom Metrics API is served by an adapter:

flowchart LR
    A[Prometheus] --> B[Prometheus Adapter]
    B --> C[custom.metrics.k8s.io API]
    C --> D[HPA]
    D --> E[Target Deployment]

The adapter reads from Prometheus and exposes the metrics via the custom.metrics.k8s.io API.

The Prometheus Adapter

The Prometheus Adapter is deployed as a Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus-adapter
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      name: prometheus-adapter
  template:
    metadata:
      labels:
        name: prometheus-adapter
    spec:
      containers:
      - name: prometheus-adapter
        image: registry.k8s.io/prometheus-adapter/prometheus-adapter:v0.11.x
        args:
        - --prometheus-url=http://prometheus.monitoring.svc:9090
        - --config=/etc/adapter/config.yaml
        volumeMounts:
        - name: config
          mountPath: /etc/adapter
      volumes:
      - name: config
        configMap:
          name: prometheus-adapter-config

The adapter queries Prometheus and exposes the metrics.

The adapter config

The adapter config maps Prometheus queries to custom metrics:

rules:
- seriesQuery: 'http_requests_total{namespace!="",pod!=""}'
  resources:
    overrides:
      namespace: {resource: "namespace"}
      pod: {resource: "pod"}
  name:
    matches: "^(.*)_total"
    as: "${1}_per_second"
  metricsQuery: 'sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)'

The rule:

• Matches the http_requests_total Prometheus metric.
• Groups by pod and namespace.
• Computes the rate per second.
• Exposes as http_requests_per_second custom metric.

The HPA configuration

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: nginx-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: nginx
  minReplicas: 3
  maxReplicas: 30
  metrics:
  - type: Pods
    pods:
      metric:
        name: http_requests_per_second
      target:
        type: AverageValue
        averageValue: "1000"

The HPA targets the http_requests_per_second custom metric. The target is 1000 req/s per pod.

The kubectl describe hpa

kubectl describe hpa nginx-hpa

Name:                                                  nginx-hpa
Namespace:                                             default
Reference:                                             Deployment/nginx
Metrics:                                               ( current / target )
  "http_requests_per_second" on Pods:                  850m / 1k
Min replicas:                                          3
Max replicas:                                          30

The output shows the current and target metric value.

The External Metrics API

The External Metrics API is similar to the Custom Metrics API but for metrics outside the cluster:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: nginx-hpa
spec:
  metrics:
  - type: External
    external:
      metric:
        name: sqs_queue_depth
        selector:
          matchLabels:
            queue: my-queue
      target:
        type: AverageValue
        averageValue: "100"

The External Metrics API is provided by an adapter (e.g., KEDA, Prometheus Adapter with external config).

The API surface

The three metrics APIs:

flowchart LR
    A[HPA] --> B[metrics.k8s.io]
    A --> C[custom.metrics.k8s.io]
    A --> D[external.metrics.k8s.io]
    B --> E[Metrics Server]
    C --> F[Custom Metrics Adapter]
    D --> G[External Metrics Adapter]

• metrics.k8s.io: CPU and memory per pod/node.
• custom.metrics.k8s.io: Custom metrics per pod.
• external.metrics.k8s.io: External metrics (e.g., SQS queue depth).

The failure modes

The common failure modes:

Adapter not deployed

error: unable to fetch custom metrics: no custom metrics API registered

The custom metrics adapter is not deployed. Verify the adapter pod is running.

Wrong metric name

error: metric http_requests_per_second not found

The HPA references a metric that does not exist. Verify the metric name in the Prometheus rules and the HPA.

Prometheus not connected

adapter: failed to query Prometheus: connection refused

The adapter cannot reach Prometheus. Verify the --prometheus-url flag.

flowchart LR
    A[Custom Metrics API] --> B{Issue?}
    B -->|Adapter not deployed| C[Deploy adapter]
    B -->|Wrong metric name| D[Verify metric name]
    B -->|Prometheus not connected| E[Verify Prometheus URL]

Cross-course references

• The Prometheus course covers adapter configuration.
• The Observability course covers metrics pipelines.
• The KEDA course covers external metrics.

Quiz

Production discipline

• Deploy the Prometheus Adapter. Required for custom metrics.
• Configure the rules carefully. The metric name, the aggregations.
• Verify the metric is exposed. Use kubectl get —raw to inspect.
• Tune the target value. 1000 req/s is typical; tune for the workload.
• Test the HPA. Use load generation to verify the scaling.
• Document the metrics. The name, the source, the target.

The Custom Metrics API is the HPA’s extension point. Operating it well is deploying the adapter, configuring the rules, and testing the scaling.