Service discovery anti-patterns — the most common mistakes

Service discovery anti-patterns are the most common mistakes that Kubernetes operators make. The anti- patterns are: hard-coded IPs, environment variables for new apps, missing port names, ignoring the nodelocal DNS cache, and using ClusterIP for external services. This lesson walks the anti-patterns, the fixes, and the operational discipline.

Anti-pattern 1: Hard-coded IPs

The application has a hard-coded ClusterIP:

apiVersion: v1
kind: ConfigMap
metadata:
  name: my-app-config
data:
  database_url: "postgres://10.96.0.20:5432/db"

The ClusterIP is allocated by the API server and is stable for the Service’s lifetime, but the ClusterIP can change when the Service is deleted and recreated. A hard-coded ClusterIP breaks when the Service is recreated.

flowchart LR
    A[Application] -->|hard-coded 10.96.0.20| B[Service]
    B -->|Service deleted and recreated| C[New ClusterIP 10.96.0.30]
    A -->|connection refused| X[Failure]

The fix is to use the Service’s DNS name:

data:
  database_url: "postgres://db.prod-app.svc.cluster.local:5432/db"

The DNS name is stable for the Service’s lifetime; the DNS resolver returns the current ClusterIP.

Anti-pattern 2: Environment variables for new apps

The application uses environment variables for discovery:

apiVersion: v1
kind: Pod
metadata:
  name: my-app
spec:
  containers:
    - name: app
      image: my-app
      env:
        - name: DB_HOST
          value: "10.96.0.20"

The environment variables are not updated when the Service changes. The fix is to use DNS:

env:
  - name: DB_HOST
    value: "db.prod-app.svc.cluster.local"
  - name: DB_PORT
    value: "5432"

The DNS resolver returns the current ClusterIP. The Pod does not need to be restarted.

Anti-pattern 3: Missing port names

The Service’s ports are not named:

apiVersion: v1
kind: Service
metadata:
  name: billing
spec:
  selector:
    app: billing
  ports:
    - port: 80
      targetPort: 8080
    - port: 443
      targetPort: 8443

The Service does not have SRV records. The fix is to name the ports:

ports:
  - name: http
    port: 80
    targetPort: 8080
  - name: https
    port: 443
    targetPort: 8443

The SRV records are generated for each named port.

Anti-pattern 4: Ignoring the nodelocal DNS cache

The cluster does not have the nodelocal DNS cache. The Pod’s DNS queries go directly to the cluster DNS service, which adds latency and load.

flowchart LR
    A[Pod] -->|DNS query| B[CoreDNS]
    B -->|answer| A

The fix is to install the nodelocal DNS cache:

kubectl apply -f https://k8s.io/examples/admin/dns/dns-cache.yaml

The cache reduces latency and load on the cluster DNS service.

Anti-pattern 5: ClusterIP for external services

The application uses a ClusterIP to reach an external service:

apiVersion: v1
kind: Service
metadata:
  name: external-db
spec:
  selector:
    app: external-db  # no Pods match
  ports:
    - port: 5432

The ClusterIP has no backends. The fix is to use ExternalName:

apiVersion: v1
kind: Service
metadata:
  name: external-db
spec:
  type: ExternalName
  externalName: db.example.com

The DNS resolver returns the CNAME; the client connects to the external service directly.

Anti-pattern 6: Hard-coded namespace

The application hard-codes the namespace:

data:
  database_url: "postgres://db.prod-app.svc.cluster.local:5432/db"

The namespace is hard-coded; the application cannot be deployed to a different namespace without changing the config. The fix is to use the search path:

data:
  database_url: "postgres://db:5432/db"

The DNS resolver uses the search path; the application is portable.

The operational discipline

The anti-patterns’ operational discipline:

• Audit the application for hard-coded IPs. The cluster operator must scan the application’s config.
• Migrate environment variables to DNS. The cluster operator must update the application’s config.
• Name the ports in the Service manifest. The cluster operator must ensure the ports are named.
• Install the nodelocal DNS cache. The cache is a performance optimization.
• Use ExternalName for external services. The cluster operator must use the right Service type.
• Use the search path for portability. The application must be portable across namespaces.
• Document the discovery patterns. The cluster operator must understand which patterns are used.
• Set up a CI check for the anti-patterns. The CI check can catch the anti-patterns at every deploy.

Quiz

Production discipline

• Audit the application for hard-coded IPs. The cluster operator must scan the application’s config.
• Migrate environment variables to DNS. The cluster operator must update the application’s config.
• Name the ports in the Service manifest. The cluster operator must ensure the ports are named.
• Install the nodelocal DNS cache. The cache is a performance optimization.
• Use ExternalName for external services. The cluster operator must use the right Service type.
• Use the search path for portability. The application must be portable across namespaces.
• Document the discovery patterns. The cluster operator must understand which patterns are used.
• Set up a CI check for the anti-patterns. The CI check can catch the anti-patterns at every deploy.
• Train the application’s team on the discovery patterns. The team’s understanding of the patterns is the basis of the migration.