Service discovery fundamentals — the four patterns and the trade-offs

Service discovery is the mechanism by which an application finds the address of another service. In Kubernetes, four patterns are available: DNS, environment variables, labels, and the API. The production choice is DNS; the others are special- purpose. This lesson walks the four patterns, the trade-offs, and the operational discipline.

The four patterns

The four service discovery patterns in Kubernetes:

The standard pattern is DNS. The Kubernetes documentation recommends DNS for all new applications.

Pattern 1: DNS

The cluster’s DNS service (CoreDNS) serves DNS records for every Service. The client queries the Service’s name and gets the ClusterIP.

# Substitute your own value before running:
POD=web-5f9c7d8b6c-2xk9p

kubectl exec "$POD" -- nslookup billing.prod-app.svc.cluster.local

Name:    billing.prod-app.svc.cluster.local
Address: 10.96.0.10

The DNS records are automatically maintained by the DNS service. The client does not need to know the ClusterIP; the DNS resolver returns it.

sequenceDiagram
    autonumber
    participant C as Pod
    participant DNS as CoreDNS
    participant API as Kubernetes API
    DNS->>API: watch Services
    API-->>DNS: Service billing added
    DNS->>DNS: update zone file
    C->>DNS: billing.prod-app.svc.cluster.local
    DNS-->>C: 10.96.0.10

The DNS resolver is the discovery mechanism; the CoreDNS controller is the data source.

Pattern 2: Environment variables

The kubelet injects environment variables for every Service when the Pod is created:

# Substitute your own value before running:
POD=web-5f9c7d8b6c-2xk9p

kubectl exec "$POD" -- env | grep BILLING

BILLING_SERVICE_HOST=10.96.0.10
BILLING_SERVICE_PORT=80
BILLING_PORT=tcp://10.96.0.10:80
BILLING_PORT_80_TCP=tcp://10.96.0.10:80
BILLING_PORT_80_TCP_PORT=80
BILLING_PORT_80_TCP_PROTO=tcp

The environment variables are populated by the kubelet when the Pod is created. The variables are not updated when the Service changes; the Pod must be restarted to pick up the new values.

Pattern 3: Labels

Labels are not a discovery mechanism per se; they are a controller pattern. The Controller queries the API for Pods with matching labels and uses the Pod IPs as the backends.

// A controller that watches Pods with label app=billing
podList, err := clientset.CoreV1().Pods("prod-app").List(
    context.Background(),
    metav1.ListOptions{LabelSelector: "app=billing"},
)

The controller pattern is the basis of the Service controller in the controller-manager. The kube-proxy is the data plane; the controller-manager is the control plane.

Pattern 4: The API

The application can query the Kubernetes API directly for the Service’s ClusterIP. The API requires authentication and authorization; the application must have a ServiceAccount with the right RBAC.

service, err := clientset.CoreV1().Services("prod-app").Get(
    context.Background(),
    "billing",
    metav1.GetOptions{},
)

The API pattern is appropriate for control-plane applications that need to discover Services dynamically. The trade-off is the complexity of the API client and the RBAC.

The comparison

The four patterns in a comparison matrix:

The DNS pattern is the production choice. The env vars pattern is the legacy; the labels pattern is for controllers; the API pattern is for special cases.

The failure modes

The four patterns’ failure modes:

• DNS: CoreDNS is down, the Service’s DNS records are stale, the client does not handle DNS failures. The fix is to verify CoreDNS and the client’s DNS retries.
• Env vars: the Service is created after the Pod, the Service’s ClusterIP changes, the Pod is not restarted. The fix is to use DNS.
• Labels: the controller is down, the labels are wrong, the controller is not watching the right namespace. The fix is to verify the controller.
• API: the API is down, the ServiceAccount is wrong, the RBAC is misconfigured. The fix is to verify the API and the RBAC.

The operational discipline

The service discovery’s operational discipline:

• Use DNS for all new applications. DNS is the production choice.
• Audit the legacy env vars usage. The cluster operator must enumerate every Pod that uses env vars.
• Test the discovery in staging. The discovery must work for the workload.
• Monitor the discovery’s health. The DNS resolver’s latency is a leading indicator.
• Document the discovery pattern. The cluster operator must understand which pattern is used.
• Plan the discovery’s evolution. The legacy env vars pattern must be migrated to DNS.

Quiz

Production discipline

• Use DNS for all new applications. DNS is the production choice.
• Audit the legacy env vars usage. The cluster operator must enumerate every Pod that uses env vars.
• Test the discovery in staging. The discovery must work for the workload.
• Monitor the discovery’s health. The DNS resolver’s latency is a leading indicator.
• Document the discovery pattern. The cluster operator must understand which pattern is used.
• Plan the discovery’s evolution. The legacy env vars pattern must be migrated to DNS.
• Train the application’s team on the discovery patterns. The team’s understanding of the patterns is the basis of the migration.
• Document the discovery’s design. The discovery is the cluster’s networking primitive; the documentation is the reference.