Labels and annotations — identifying and describing objects

Labels and annotations are both key-value metadata on objects. They differ in a fundamental way: labels drive selection; annotations hold non-identifying metadata. This lesson covers both, the rules for valid keys/values, and the production patterns that keep them useful at scale.

Labels: the selected kind of metadata

Labels are key-value pairs that identify and group objects. The defining property: labels are queryable. Services, Deployments, NetworkPolicies, and other controllers use label selectors to choose which objects they apply to.

metadata:
  labels:
    app.kubernetes.io/name: web
    app.kubernetes.io/component: frontend
    app.kubernetes.io/part-of: checkout
    app.kubernetes.io/managed-by: helm
    app.kubernetes.io/instance: web-prod
    environment: prod
    team: payments

Rules:

• Key: optional prefix (DNS subdomain, ≤ 253 chars) + / + name (≤ 63 chars). Without a prefix, the key is in the user’s domain.
• Value: ≤ 63 chars; alphanumeric, dashes, underscores, dots.
• Each object can have many labels.
• Labels can be added, modified, or removed at any time.

# Add a label
kubectl label pod web-abc env=prod

# Update a label
kubectl label pod web-abc env=staging --overwrite

# Remove a label
kubectl label pod web-abc env-

Production labels are typically prefixed with app.kubernetes.io/ (a Kubernetes-recommended convention) or the organisation’s domain (acme.example.com/team).

Annotations: the descriptive kind of metadata

Annotations are also key-value pairs but they are not queryable. They hold non-identifying information:

metadata:
  annotations:
    description: "Production web tier for checkout"
    contact: "team-payments@example.com"
    runbook: "https://wiki.example.com/runbooks/web-prod"
    prometheus.io/scrape: "true"
    prometheus.io/port: "9090"
    kubernetes.io/change-cause: "Image bump to 1.27.2"

Rules:

• Key: same as labels (prefix + name); but annotations can use a wider character set
• Value: any string (≤ 256 KB total per object)
• Annotations are not indexed in etcd
• Annotations are meant for tools, not for selection

Common annotation uses:

• Tool integration (Prometheus scrape config, Vault references, cert-manager)
• Documentation (runbook URLs, contact info)
• Build information (commit SHA, build timestamp)
• Lifecycle annotations (kubernetes.io/change-cause)

Label selectors

A label selector is a query that returns objects matching the labels. Two forms:

Equality-based

app = web
env != dev

Match labels with the exact value, or exclude them.

Set-based

app in (web, api)
env notin (dev)
!partition

Match labels in a set, or exclude a set. The ! prefix selects objects without the label.

Used in:

• Service.spec.selector — which Pods the Service routes to
• Deployment.spec.selector — which Pods the Deployment owns
• NetworkPolicy.spec.podSelector — which Pods the policy applies to
• kubectl get -l — query objects from the command line
• kubectl delete -l — delete by selector

# All Pods with app=web in any namespace
kubectl get pods -A -l app=web

# Pods in prod that are not part of the legacy stack
kubectl get pods -n prod -l environment=prod,stack!=legacy

# Pods labeled with the team annotation
kubectl get pods -A -l 'team in (payments,search)'

Production label conventions

A common pattern uses the Kubernetes-recommended app.kubernetes.io/* labels:

metadata:
  labels:
    app.kubernetes.io/name: web
    app.kubernetes.io/instance: web-prod
    app.kubernetes.io/version: "1.27.2"
    app.kubernetes.io/component: frontend
    app.kubernetes.io/part-of: checkout
    app.kubernetes.io/managed-by: argocd

Additional production conventions:

    environment: prod
    tier: frontend
    team: payments
    cost-center: "12345"
    compliance: pci-dss
    # for service mesh
    app: web
    # for tooling
    owner: team-payments

The convention varies by team; the discipline is to have one and apply it consistently.

How labels drive object discovery

# Find all objects belonging to the checkout application
kubectl get all -A -l app.kubernetes.io/part-of=checkout

# Find all objects managed by Helm
kubectl get all -A -l app.kubernetes.io/managed-by=helm

# Find all resources for the payments team's prod environment
kubectl get all -n payments -l environment=prod,team=payments

# Iterate over matching Pods in a script
for pod in $(kubectl get pods -l app=web -o name); do
  kubectl logs $pod --since=10m
done

A consistent label scheme is the difference between “I can find this object” and “I cannot find this object”.

Label selectors in controllers

Controllers use label selectors to choose which objects they own. The Deployment controller:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - name: nginx
        image: nginx:1.27.1

The Deployment’s selector.matchLabels must match the labels in template.metadata.labels. If they don’t, the API server rejects the manifest at creation.

A common operational mistake is changing a Deployment’s spec.selector after creation — the API server rejects it (field is immutable). To change the selector, delete the Deployment and recreate it (with the data being lost; this is why the selector is immutable).

The Service uses the same selector pattern:

apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  selector:
    app: web   # routes to Pods with this label
  ports:
  - port: 80
    targetPort: 8080

If the Service’s selector doesn’t match any Pods, the Service has no Endpoints; traffic to the Service is dropped.

Annotations vs labels: a decision framework

Labels and annotations are both visible to anyone with read access to the object. Do not put secrets in either; use a Secret or external secret manager.

Label cardinality and performance

The number of unique label values affects API server and etcd performance. A label with high cardinality (e.g., a unique value per request) can:

• Bloat etcd storage
• Increase watch event volume
• Slow down list operations

Production rules:

• Avoid labels with unbounded cardinality (timestamps, IDs)
• Prefer low-cardinality labels (env, tier, team)
• For high-cardinality identifiers, use annotations or external observability systems

Cross-course references

• The Linux course part XXVII-Linux-Auth covers identity/group concepts that map onto label-based grouping.
• The Observability course part IX-Observability-Exporters covers Prometheus label handling — similar trade-offs to Kubernetes labels.
• The Linux course part XXIX-Linux-Hardening covers classification schemes (security levels, compliance) that map onto label conventions.
• The Docker course part XXXVIII-Docker-Secrets covers secret handling; labels and annotations are not for secrets.

Quiz

Production discipline

• Adopt a label convention (e.g., app.kubernetes.io/*) and apply it to every object.
• Use labels for selection, annotations for documentation and tooling. Never confuse the two.
• Keep cardinality low; avoid labels with unbounded unique values.
• Treat metadata.labels as the Service/Deployment selector’s contract — they must match exactly.
• Document the label convention and enforce it with admission policies (e.g., ValidatingAdmissionPolicy that requires certain labels).