Service discovery — the Kubernetes integration

Prometheus service discovery in Kubernetes uses the Kubernetes API. The raw kubernetes_sd_configs, the ServiceMonitor, and the PodMonitor are the three mechanisms. This lesson walks the discovery, the mechanisms, the relabeling, and the production patterns.

The discovery mechanisms

flowchart LR
    A[Prometheus] --> B[Kubernetes API]
    B --> C[raw kubernetes_sd_configs]
    B --> D[ServiceMonitor]
    B --> E[PodMonitor]

The three mechanisms are different.

The raw kubernetes_sd_configs

The raw kubernetes_sd_configs:

scrape_configs:
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: (.+)
        target_label: __address__
        replacement: ${1}

The kubernetes_sd_configs discovers the pods via the Kubernetes API.

The discovery roles

The discovery roles:

The roles are the discovery targets.

The annotations

The annotations:

metadata:
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "8080"
    prometheus.io/path: "/metrics"
    prometheus.io/scheme: "http"

The annotations tell the Prometheus to scrape the pod.

The relabeling

The relabeling:

flowchart LR
    A[__meta_kubernetes_pod_annotation_prometheus_io_scrape] --> B{keep?}
    B -->|true| C[Keep]
    B -->|false| D[Drop]

The relabeling filters the targets based on the annotations.

The ServiceMonitor

The ServiceMonitor is the Operator-style discovery:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-app
  labels:
    release: my-prometheus
spec:
  selector:
    matchLabels:
      app: my-app
  endpoints:
  - port: http
    path: /metrics
    interval: 30s
  namespaceSelector:
    matchNames:
    - default
    - production

The ServiceMonitor selects the Services with the matching labels.

The ServiceMonitor features

The ServiceMonitor features:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-app
spec:
  selector:
    matchLabels:
      app: my-app
  endpoints:
  - port: http
    path: /metrics
    interval: 30s
    scrapeTimeout: 10s
    honorLabels: true
    relabelings:
    - source_labels: [__meta_kubernetes_pod_label_team]
      target_label: team
    metricRelabelings:
    - source_labels: [__name__]
      regex: 'go_.*'
      action: drop

The ServiceMonitor supports:

• Filter: the selector and the namespaceSelector.
• Endpoint: the port, the path, the interval.
• Relabeling: the target labels.
• Metric relabeling: the metric labels.

The PodMonitor

The PodMonitor is the pod-level scrape:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: my-app
  labels:
    release: my-prometheus
spec:
  selector:
    matchLabels:
      app: my-app
  podMetricsEndpoints:
  - port: http
    path: /metrics
    interval: 30s

The PodMonitor is used when the Service is not the right target (e.g., the pod is not behind a Service).

The discoverer

The discoverer:

flowchart LR
    A[Prometheus] --> B[Discoverer]
    B --> C[Kubernetes API]
    C --> D[Endpoints]
    D --> A
    A --> E[Scrape targets]

The discoverer queries the Kubernetes API every discovery_interval (default 30s).

The production patterns

The production patterns:

flowchart LR
    A[kube-prometheus-stack] --> B[ServiceMonitors for cluster components]
    A --> C[ServiceMonitors for applications]
    B --> D[Prometheus]
    C --> D

The ServiceMonitors are the discovery.

The troubleshooting

The troubleshooting:

# Check the discovery
kubectl get servicemonitor -n monitoring

# Check the targets
kubectl port-forward -n monitoring prometheus-xxx 9090
# Open http://localhost:9090/targets

# Check the logs
kubectl logs -n monitoring prometheus-xxx

The troubleshooting is direct.

Cross-course references

• The Prometheus Operator course covers the CRDs.
• The Grafana course covers the dashboards.
• The Helm course covers the chart installation.

Quiz

Production discipline

• Use the ServiceMonitor. The canonical discovery.
• Use the PodMonitor when appropriate. The pod-level scrape.
• Use the relabeling for filtering. The label-based filter.
• Use the metric relabeling for cardinality. The high-cardinality drop.
• Document the discovery. The ServiceMonitors, the relabeling.
• Test the discovery. Verify the targets.

The Prometheus service discovery is the cluster’s metrics flow. Operating it well is via the ServiceMonitor, with the relabeling for filtering, and the metric relabeling for cardinality control.