KSM labels and annotations — the metadata for queries

The KSM labels and annotations are the metadata for the queries. The labels are the input for filtering and aggregation; the annotations are the enrichments. This lesson walks the labels, the annotations, the queries, and the production patterns.

The KSM labels

The KSM metrics include the labels:

kube_pod_info{
  namespace="default",
  pod="nginx-1-abc",
  node="worker-1",
  host_network="false",
  created_by_kind="Deployment",
  created_by_name="nginx",
  owner_kind="ReplicaSet",
  owner_name="nginx-7b9f8c5f6",
  pod_ip="10.0.1.20"
}

The labels are the metadata for the queries.

The standard labels

The standard labels:

The labels are stable across versions.

The label joins

The label joins combine the metrics:

# Join KSM and cAdvisor metrics
rate(container_cpu_usage_seconds_total[5m])
on (namespace, pod)
kube_pod_info

The join is the power of Prometheus.

flowchart LR
    A[KSM metrics] --> C[Join]
    B[cAdvisor metrics] --> C
    C --> D[Combined metrics]

The aggregation by label

The aggregation by label:

# Pod count by namespace
count(kube_pod_info) by (namespace)

# Pod count by owner
count(kube_pod_info) by (created_by_kind, created_by_name)

# Pod count by node
count(kube_pod_info) by (node)

The aggregation is the input for the dashboards.

The annotation allowlist

The annotation allowlist:

spec:
  template:
    spec:
      containers:
      - name: kube-state-metrics
        args:
        - --metric-annotations-allowlist=namespaces=[team,environment]
        - --metric-labels-allowlist=namespaces=[team,environment]

The --metric-annotations-allowlist flag includes the specified annotations in the metrics. The annotations are not included by default to avoid the cardinality explosion.

The label allowlist

The label allowlist:

spec:
  template:
    spec:
      containers:
      - name: kube-state-metrics
        args:
        - --metric-labels-allowlist=namespaces=[team,environment]

The --metric-labels-allowlist flag includes the specified labels in the metrics. The labels are not included by default.

The label-based alerts

The label-based alerts:

- alert: KubePodCrashLooping
  expr: |
    rate(kube_pod_container_status_restarts_total[10m]) * 60 * 5 > 0
  for: 5m
  alerts:
    annotations:
      summary: "Pod {{ $labels.namespace }}/{{ $labels.pod }} is crash looping"
      runbook_url: "https://runbook.example.com/k8s/pod-crash"

The alerts use the labels for filtering.

The label-based dashboards

The label-based dashboards:

# Pods by namespace
count(kube_pod_info) by (namespace)

# Pods by owner
count(kube_pod_info) by (created_by_name)

# Pods by team (via label allowlist)
count(kube_pod_info{team="team-a"}) by (namespace)

The dashboards use the labels for grouping.

The cross-cutting labels

The cross-cutting labels:

# Labels for tenancy
team: team-a
environment: production

# Labels for cost
cost-center: engineering
project: payment

# Labels for compliance
data-classification: confidential

The cross-cutting labels are added to the namespaces.

kubectl label namespace default team=team-a environment=production

The labels are the input for the queries.

The label selectors

The label selectors:

# All pods in production
kube_pod_info{environment="production"}

# All pods in production by team
kube_pod_info{environment="production",team="team-a"}

# All pods in production by team and node
kube_pod_info{environment="production",team="team-a",node="worker-1"}

The selectors are the input for the dashboards.

Cross-course references

• The Prometheus course (Part LXXXVIII) covers the labels.
• The Prometheus Operator course covers the allowlists.
• The Multi-tenancy course covers the labels.

Quiz

Production discipline

• Use the KSM labels for filtering. The standard labels.
• Use the allowlists for annotations. Avoid cardinality explosion.
• Use the allowlists for labels. Avoid cardinality explosion.
• Build the dashboards using the labels. The cross-cutting.
• Document the labels. The standard labels, the allowlists.
• Test the labels. Verify the queries work.

The KSM labels and annotations are the metadata for queries. Operating it well is using the standard labels, allowing annotations carefully, and building the dashboards using the labels.