Labels, selectors, and the publishNotReadyAddresses option

Labels and selectors are the basis of service discovery in Kubernetes. The Service selector matches Pods by labels; the controller-manager reconciles the EndpointSlices based on the selector. The publishNotReadyAddresses option is used for stateful workloads. The topologyKeys field controls local preference. This lesson walks the labels, the selector, and the operational discipline.

The label-driven discovery

The Service’s selector matches Pods by labels. The controller-manager maintains the EndpointSlices for every Service based on the selector.

apiVersion: v1
kind: Service
metadata:
  name: billing
spec:
  selector:
    app: billing
    tier: backend
  ports:
    - port: 80
      targetPort: 8080

The selector matches Pods with both labels app=billing and tier=backend. The EndpointSlices are populated with the matching Pods.

sequenceDiagram
    autonumber
    participant S as Service selector
    participant C as Controller
    participant P as Pods
    participant ES as EndpointSlice
    S->>C: selector app=billing, tier=backend
    C->>P: list Pods with labels
    P-->>C: Pod 1, Pod 2, Pod 3
    C->>ES: create/update slices
    ES->>ES: populate with Pod IPs

The controller is the source of truth for the Service-to-Pod mapping; the labels are the contract.

The selector’s power

The selector supports multiple keys and operators:

selector:
  app: billing
  tier: backend
  environment: production

The selector is an AND: a Pod must have all the labels to match. The selector also supports In, NotIn, Exists, and DoesNotExist operators via the selector.matchLabels and selector.matchExpressions fields.

selector:
  matchLabels:
    app: billing
  matchExpressions:
    - key: tier
      operator: In
      values: [backend, frontend]

The matchExpressions support set-based operators. The selector is a powerful tool for service discovery.

The publishNotReadyAddresses option

The Service has a publishNotReadyAddresses option that publishes Pod IPs in the EndpointSlice even when the Pod is not Ready:

apiVersion: v1
kind: Service
metadata:
  name: billing
spec:
  selector:
    app: billing
  publishNotReadyAddresses: true
  ports:
    - port: 80
      targetPort: 8080

The option is used for stateful workloads where the Pod needs to be reachable before it is Ready (e.g., a database that needs to be initialized before accepting traffic). The Pod is in the EndpointSlice but the client must handle the Pod’s not-Ready state.

The topologyKeys

The Service has a topologyKeys field that restricts the traffic to Pods in the same zone, region, or other topology:

apiVersion: v1
kind: Service
metadata:
  name: billing
spec:
  topologyKeys:
    - "topology.kubernetes.io/zone"
    - "*"
  selector:
    app: billing
  ports:
    - port: 80
      targetPort: 8080

The topologyKeys field is used for headless services to prefer local Pods. The first key in the list is the preferred topology; the fallback is "*" (any Pod).

The failure modes

The label-driven discovery’s failure modes:

• Selector mismatch: the Pod’s labels do not match the Service’s selector. The fix is to verify the labels.
• Pod not Ready: the Pod’s ready condition is false; the Pod is not in the EndpointSlice. The fix is to verify the readiness probe.
• publishNotReadyAddresses misconfigured: the Pod is published when not Ready; the client cannot handle the not-Ready state. The fix is to set the option correctly.
• topologyKeys wrong: the topology keys do not match the cluster’s topology. The fix is to verify the topology labels.
• Label changes: the Pod’s labels are changed and the EndpointSlice is updated. The fix is to verify the label changes.

The operational discipline

The label-driven discovery’s operational discipline:

• Document the Service’s selector. The cluster operator must understand which Pods the Service routes to.
• Audit the labels at every deploy. A label mismatch is the most common Service failure.
• Use consistent label conventions. The cluster operator should define a label schema.
• Verify the publishNotReadyAddresses for stateful workloads. The stateful workloads need the option.
• Test the topology keys for multi-zone clusters. The topology keys must match the cluster’s topology.
• Monitor the EndpointSlice for empty backends. An empty EndpointSlice is the leading indicator.
• Document the label strategy. The label strategy is the cluster’s discovery contract.

Quiz

Production discipline

• Document the Service’s selector. The cluster operator must understand which Pods the Service routes to.
• Audit the labels at every deploy. A label mismatch is the most common Service failure.
• Use consistent label conventions. The cluster operator should define a label schema.
• Verify the publishNotReadyAddresses for stateful workloads. The stateful workloads need the option.
• Test the topology keys for multi-zone clusters. The topology keys must match the cluster’s topology.
• Monitor the EndpointSlice for empty backends. An empty EndpointSlice is the leading indicator.
• Document the label strategy. The label strategy is the cluster’s discovery contract.
• Set up a CI check for the labels. The CI check can catch the typo at every deploy.
• Train the application’s team on the label strategy. The team’s understanding of the labels is the basis of the discovery.