Aggregated API servers — kube-aggregator, extension points

The Kubernetes API is extensible. Beyond the built-in API groups (apps/v1, core/v1, etc.), the cluster can serve custom API paths through aggregated API servers. Production examples include the metrics server, custom-metrics adapter, and various operator-built servers. This lesson walks the mechanism, the production patterns, and the gotchas.

What aggregation means

flowchart LR
    C[kubectl / client] -->|https| AS[kube-apiserver]
    AS -->|/api/v1/* - built-in| AS
    AS -->|/apis/metrics.k8s.io/* - aggregated| A1[aggregated API server]
    AS -->|/apis/custom.metrics.k8s.io/* - aggregated| A2[custom-metrics adapter]
    A1 -->|TLS + auth delegation| AS[auth via kube-apiserver]
    A2 -->|TLS + auth delegation| AS

The kube-apiserver routes API requests based on path:

• Built-in paths (/api/v1, /apis/apps/v1) are served directly by the API server.
• Aggregated paths (registered via APIService objects) are proxied to the registered aggregated API server.
• The aggregated server handles its own storage but delegates authentication and authorisation to the kube-apiserver.

The kube-aggregator

The kube-aggregator is a built-in API server plugin that manages aggregation:

kubectl get apiservices

NAME                          SERVICE                      AVAILABLE   AGE
v1beta1.metrics.k8s.io        kube-system/metrics-server   True        30d
v1beta1.custom.metrics.k8s.io custom-metrics/svc           True        30d
v1beta1.external.metrics.k8s.io external-metrics/svc       True        30d

Each APIService registers a path prefix with the kube-apiserver. A request to /apis/metrics.k8s.io/v1beta1/nodes is proxy-routed to the metrics-server service.

The registered API server lifecycle

stateDiagram-v2
    [*] --> Pending: APIService created
    Pending --> Available: TLS handshake succeeds
    Available --> Pending: 5xx errors / heartbeat failure
    Available --> Removed: APIService deleted

The AVAILABLE column reflects the kube-aggregator’s last observation of the aggregated server:

• The aggregated server passes the health check (/healthz).
• The aggregated server’s client cert is trusted by the kube-apiserver’s CA bundle.

The certification flow

Each aggregated server must present a client cert signed by the kube-apiserver’s CA (/etc/kubernetes/pki/ca.crt):

1. Aggregated server presents client cert (from `--cert` flag)
2. kube-apiserver verifies against CA bundle
3. Aggregated server presents its server cert on the proxy
   connection back to clients
4. The server cert is trusted (because the cert chain or SAN
   matches the cluster's expected configuration)

The aggregated server typically does cert rotation through kubeadm/cert-manager; the cert SAN must include the service’s DNS name (e.g., metrics-server.kube-system.svc.cluster.local).

A trace through an aggregated request

sequenceDiagram
    autonumber
    participant Client as kubectl
    participant AS as kube-apiserver
    participant MS as metrics-server
    Client->>AS: GET /apis/metrics.k8s.io/v1beta1/nodes
    AS->>AS: authenticate (TLS, token)
    AS->>AS: route to APIService for metrics.k8s.io
    AS->>MS: proxy request (with auth headers)
    MS->>AS: SubjectAccessReview for the user
    AS-->>MS: allowed
    MS->>MS: query kubelet metrics
    MS-->>AS: response data
    AS-->>Client: 200 OK

The aggregated server runs SubjectAccessReview against the kube-apiserver to validate RBAC. The response is returned to the client through the proxy.

Production patterns

The metrics-server

The most common aggregated API server is the metrics server:

metrics.k8s.io/v1beta1/nodes
metrics.k8s.io/v1beta1/pods

The metrics-server reads kubelet’s /metrics/resource and presents aggregated node/pod metrics. The Horizontal Pod Autoscaler (HPA) uses these metrics.

kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes" | jq

Custom metrics adapters

The custom-metrics-apiserver is a deployment that provides custom metrics for the HPA. The implementation is open-source (Prometheus adapter); it queries Prometheus and serves the result via aggregated API.

Operator-built aggregation

Operators may extend the API surface for their custom resources. The simpler approach is CustomResourceDefinition (CRD); the aggregated approach is used when the API needs imperative semantics or high-throughput reads beyond CRD’s capabilities.

Failure modes

The “version skew” concern

Aggregated API servers typically have a Kubernetes version range they support. An aggregated server built for 1.32 may not work on 1.34; an aggregated server built for 1.34 may not work on 1.30. The version compatibility is the aggregated server’s responsibility — the operator must track.

Aggregated servers vs CRDs

For most use cases, CRDs are sufficient. Aggregated API servers are reserved for: imperative semantics, high- throughput reads, complex conversion logic, or integration with systems that need their own storage backend.

Production discipline

• Aggregated servers are part of the API surface. They appear in apiservice status; they fail visibly when down.
• Version skew is a real risk. A cluster upgrade that bumps the kube-apiserver version may break the aggregated server.
• RBAC is enforced in the aggregated server. Configure RBAC in the aggregated server too; it is not inherited.
• Monitor the APIService available status. A kubectl get apiservices check is a quick health check.

Aggregated API servers extend the cluster’s surface; understanding their lifecycle and discipline is part of operating Kubernetes at production scale.

Quiz