Lab 22: Monitor the cluster with Prometheus

Objective

By the end of this lab you will have a working Prometheus on a disposable kubeadm cluster, scraping both the cluster’s own components and one application you deployed yourself — and, more importantly, you will have spent part of the lab staring at a correctly-written ServiceMonitor that Prometheus was ignoring, with no error anywhere, until you learned to follow the three label hops that connect a Prometheus to a pod.

That silence is the point of the lab. Nothing in Kubernetes tells you that a ServiceMonitor is unmatched. kubectl get servicemonitor shows it. kubectl describe shows no events. The Operator logs nothing about it. The only place the absence is visible is in the scrape target list, and only if you know what should be there.

Architecture

Prometheus does not scrape pods. It scrapes targets that the Prometheus Operator wrote into a configuration file, and the Operator only writes a target when three separate label relationships all line up:

flowchart TB
    P["Prometheus CR<br/>spec.serviceMonitorSelector"] -->|"hop 1 — selector matches ServiceMonitor labels"| SM["ServiceMonitor<br/>metadata.labels"]
    SM -->|"hop 2 — spec.selector matches Service labels"| SVC["Service<br/>metadata.labels"]
    SVC -->|"hop 3 — endpoint port NAME matches a Service port name"| EP["EndpointSlice → Pod :8080/metrics"]
    EP --> PROM["Prometheus scrape"]

Each hop fails silently and independently. Hop 1 is the one people lose hours to, because the object they wrote looks correct in isolation — it is only wrong relative to a selector living in a different object, in a different namespace, that they never read.

The lab builds all three hops, breaks hop 1, then breaks hop 3, so you can tell the two failure signatures apart.

Requirements

A disposable kubeadm cluster: one control-plane node and two workers, Kubernetes 1.34.x, containerd as the CRI, a working CNI. The cluster built in Lab 01 is exactly right. Do not use a cluster you care about: Task 2 installs cluster-scoped CRDs and Cleanup deletes them.
At least 4 GiB of free memory on one worker. The stack runs a Prometheus, a kube-state-metrics, an Operator, and a node-exporter per node. On a 2 GiB worker the Prometheus pod is the first thing the kubelet evicts, and you will spend the lab debugging eviction instead of discovery.
kubectl 1.34.x, helm 3.x, curl and jq on your workstation.
Cluster-admin on the cluster. No out-of-band access requirement: this lab touches no node networking, no firewall and no SSH configuration, so nothing here can lock you out of the hosts.
Egress from the cluster’s nodes to a container registry, and from your workstation to the Helm chart repository.
The manifests below pin nginx:1.27-alpine. Check the tag still resolves before you start. An image tag is the part of any lab that ages first, and a lab that fails at ImagePullBackOff teaches nothing about Prometheus.

Scenario

A team installed kube-prometheus-stack a fortnight ago. The bundled dashboards are green, node CPU and memory graph correctly, kubectl top works, and everyone considers monitoring “done”.

Then an incident: their reporting service backs up and nobody notices for forty minutes, because the queue-depth metric the service has exposed on /metrics since the day it was written has never once been scraped. The ServiceMonitor is in Git. It was applied. It is in the cluster right now. It has simply never matched anything.

Your job is to reproduce that state deliberately, and to build the diagnostic habit that finds it in two minutes instead of two hours.

Tasks

Task 1: Record the starting state

Cleanup deletes cluster-scoped objects. It must delete only what this lab created, and the only way to know that is to write down what was there first.

WORKDIR="$HOME/k8s-prometheus-lab"
mkdir -p "$WORKDIR"
cd "$WORKDIR"

kubectl get namespaces -o name | sort > namespaces-before.txt
kubectl get crd -o name | sort > crds-before.txt
kubectl get nodes -o wide > nodes-before.txt

grep -c . crds-before.txt
grep 'monitoring.coreos.com' crds-before.txt || echo "no monitoring CRDs present - good"

If that last grep finds anything, an earlier install is still on the cluster. Finish removing it (the Cleanup section works standalone) before continuing, or hop 1 will be broken for a reason this lab did not create.

Task 2: Install the stack, and record what you installed

The chart moves. The Prometheus it bundles moves. A lab that says “install kube-prometheus-stack” and does not record which one you got is a lab you cannot compare notes on later.

prometheus-lab-values.yaml:

# Deliberately small. This is a lab cluster, not a monitoring platform.
alertmanager:
  enabled: false
grafana:
  enabled: false

prometheus:
  prometheusSpec:
    # No storageSpec, so Prometheus stores its TSDB in an emptyDir: the
    # data dies with the pod. Correct for a lab (Cleanup is total), wrong
    # for production (a restart loses every series).
    retention: 6h
    scrapeInterval: 15s
    resources:
      requests:
        cpu: 200m
        memory: 512Mi
      limits:
        memory: 1Gi

Grafana and Alertmanager are switched off because this lab is about whether a scrape happens, not about what is drawn or who is paged. Turn them on if you have the headroom; nothing below depends on them.

cd "$HOME/k8s-prometheus-lab"

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

# Write down what you are about to install, before you install it.
helm search repo prometheus-community/kube-prometheus-stack --versions \
  | head -3 | tee chart-version.txt

Configuration changeworkstation— Creates the monitoring namespace, the Operator, the CRDs, Prometheus, kube-state-metrics and a node-exporter DaemonSet

$ helm install prometheus prometheus-community/kube-prometheus-stack --namespace monitoring --create-namespace --values prometheus-lab-values.yaml --wait --timeout 10m

cd "$HOME/k8s-prometheus-lab"

kubectl get pods -n monitoring -o wide
kubectl get crd -o name | sort > crds-after-install.txt
comm -13 crds-before.txt crds-after-install.txt | tee crds-added.txt

crds-added.txt is the list of cluster-scoped API types this one Helm release added. Read it. This is the moment to internalise that a Helm release in one namespace changed the shape of the whole cluster’s API, and that helm uninstall will not change it back.

Task 3: Prove the platform works before you blame your application

When your application’s metrics are missing, the first question is not “what is wrong with my ServiceMonitor”. It is “is Prometheus scraping anything”. That check costs fifteen seconds and eliminates half the possible causes, which is the only reason to run it first.

Open a port-forward and leave it running in its own terminal:

kubectl port-forward -n monitoring svc/prometheus-operated 9090:9090

prometheus-operated is the headless Service the Operator creates for every Prometheus it manages. Use it rather than the chart’s own Service: its name does not change when the Helm release name changes, so the command survives being copied into a cluster where somebody named the release something else.

In a second terminal:

cd "$HOME/k8s-prometheus-lab"

curl -s http://127.0.0.1:9090/api/v1/targets \
  | jq -r '.data.activeTargets[] | "\(.health)\t\(.labels.job)"' \
  | sort | uniq -c | sort -rn | tee targets-before.txt

Read-only / Safeworkstation— Health and job label of every active scrape target, counted

$ curl -s http://127.0.0.1:9090/api/v1/targets | jq -r '.data.activeTargets[] | "\(.health)\t\(.labels.job)"' | sort | uniq -c | sort -rn

      3 up	node-exporter
    1 up	kube-state-metrics
    1 up	prometheus
    1 up	prometheus-operator

Illustrative output

The exact job names depend on the chart version, which is why you counted them from the cluster instead of reading them from here. What matters is the shape: several jobs, all up. If some cluster-component jobs are down on a kubeadm cluster, that is normal and not your problem today — several control-plane components bind their metrics endpoints to localhost by default, so the stack’s ServiceMonitors for them find nothing to scrape. Note which ones and move on.

The API, not the web UI, is the right tool here. targets-before.txt is a file you can diff in Task 6. A screenshot is not.

Task 4: Give Prometheus something of your own to scrape

A scrape target is not a special kind of object. It is an HTTP endpoint that returns text in Prometheus exposition format. This deployment proves that by serving a static file with nginx.

demo-exporter.yaml:

apiVersion: v1
kind: Namespace
metadata:
  name: demo
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: demo-exporter
  namespace: demo
data:
  nginx.conf: |
    server {
        listen 8080;
        location = /metrics {
            default_type text/plain;
            alias /usr/share/nginx/exposition/metrics;
        }
        location = /healthz {
            default_type text/plain;
            return 200 "ok\n";
        }
    }
  metrics: |
    # HELP demo_orders_total Orders the demo service has accepted since start.
    # TYPE demo_orders_total counter
    demo_orders_total{status="accepted"} 4211
    demo_orders_total{status="rejected"} 37
    # HELP demo_queue_depth Work items waiting in the demo service queue.
    # TYPE demo_queue_depth gauge
    demo_queue_depth 12
    # HELP demo_build_info Build metadata for the demo service.
    # TYPE demo_build_info gauge
    demo_build_info{version="0.1.0"} 1
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-exporter
  namespace: demo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: demo-exporter
  template:
    metadata:
      labels:
        app: demo-exporter
    spec:
      containers:
        - name: nginx
          image: nginx:1.27-alpine
          ports:
            - name: http-metrics
              containerPort: 8080
          readinessProbe:
            httpGet:
              path: /healthz
              port: http-metrics
          volumeMounts:
            - name: conf
              mountPath: /etc/nginx/conf.d
            - name: exposition
              mountPath: /usr/share/nginx/exposition
      volumes:
        - name: conf
          configMap:
            name: demo-exporter
            items:
              - key: nginx.conf
                path: default.conf
        - name: exposition
          configMap:
            name: demo-exporter
            items:
              - key: metrics
                path: metrics
---
apiVersion: v1
kind: Service
metadata:
  name: demo-exporter
  namespace: demo
  labels:
    app: demo-exporter
spec:
  selector:
    app: demo-exporter
  ports:
    - name: http-metrics
      port: 8080
      targetPort: http-metrics

cd "$HOME/k8s-prometheus-lab"

kubectl apply -f demo-exporter.yaml
kubectl rollout status -n demo deploy/demo-exporter --timeout=120s

Confirm the endpoint before you ask Prometheus to scrape it. If the endpoint is broken, every later step is debugging the wrong layer:

kubectl port-forward -n demo svc/demo-exporter 8080:8080 &
sleep 3
curl -s http://127.0.0.1:8080/metrics | head -6

The values in that file never change. demo_queue_depth will be a flat line at 12 forever, and that is deliberate: this lab is about whether the scrape happens, not about what it returns. A flat line is unambiguous evidence of a working scrape in a way that a noisy one is not.

Task 5: Wire it up — and watch nothing happen

servicemonitor-broken.yaml:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: demo-exporter
  namespace: demo
spec:
  selector:
    matchLabels:
      app: demo-exporter
  endpoints:
    - port: http-metrics
      path: /metrics
      interval: 15s

cd "$HOME/k8s-prometheus-lab"

kubectl apply -f servicemonitor-broken.yaml
kubectl get servicemonitor -n demo
kubectl describe servicemonitor -n demo demo-exporter | tail -12

The object exists. It has no status, no events, no warning. Wait a minute — longer than the Operator’s reconcile and Prometheus’s reload — then look for the target:

curl -s http://127.0.0.1:9090/api/v1/targets \
  | jq -r '.data.activeTargets[].labels.job' | sort -u

Your job is not there. Confirm it from the other direction too, because “the query returned nothing” and “the metric does not exist” are different statements only until you check:

curl -sG http://127.0.0.1:9090/api/v1/query \
  --data-urlencode 'query=demo_queue_depth' | jq '.data.result'

An empty array. Not an error — Prometheus answered your question correctly. It has no series by that name because nothing ever told it to go and fetch one.

Task 6: Find the broken hop, with evidence

Work the hops in order, cheapest first.

Hop 1 — does the Prometheus want this ServiceMonitor? Read the selector from the Prometheus object itself. Do not assume its name; derive it:

PROM_CR=$(kubectl get prometheus -n monitoring -o jsonpath='{.items[0].metadata.name}')
echo "Prometheus CR: $PROM_CR"

kubectl get prometheus -n monitoring "$PROM_CR" \
  -o jsonpath='{.spec.serviceMonitorSelector}{"\n"}'
kubectl get prometheus -n monitoring "$PROM_CR" \
  -o jsonpath='{.spec.serviceMonitorNamespaceSelector}{"\n"}'

The chart sets a selector that matches on the Helm release name. Your ServiceMonitor carries no labels at all, so it cannot match. That is hop 1, and it is broken.

Confirm it against the generated configuration, which is the only authoritative answer. The Operator renders every matched ServiceMonitor into one Prometheus config file and stores it, gzipped, in a Secret:

cd "$HOME/k8s-prometheus-lab"

kubectl get secret -n monitoring "prometheus-$PROM_CR" \
  -o jsonpath='{.data.prometheus\.yaml\.gz}' \
  | base64 -d | gunzip > prometheus-generated-before.yaml

grep -c 'job_name' prometheus-generated-before.yaml
grep 'job_name' prometheus-generated-before.yaml | grep demo || echo "no demo job in the generated config"

There is the evidence, and it is unambiguous: the scrape job does not exist. The problem is upstream of Prometheus entirely. Prometheus is doing exactly what its configuration says.

Now fix hop 1. Label the ServiceMonitor with the release name:

kubectl label servicemonitor -n demo demo-exporter release=prometheus
kubectl get servicemonitor -n demo --show-labels

Wait about a minute, then re-read both the config and the targets:

cd "$HOME/k8s-prometheus-lab"

kubectl get secret -n monitoring "prometheus-$PROM_CR" \
  -o jsonpath='{.data.prometheus\.yaml\.gz}' \
  | base64 -d | gunzip > prometheus-generated-after.yaml

diff <(grep 'job_name' prometheus-generated-before.yaml) \
     <(grep 'job_name' prometheus-generated-after.yaml)

curl -s http://127.0.0.1:9090/api/v1/targets \
  | jq -r '.data.activeTargets[] | "\(.health)\t\(.labels.job)\t\(.scrapeUrl)"' \
  | sort | tee targets-after.txt | grep demo

Find the job label your target actually carries in that output, then query the metric:

curl -sG http://127.0.0.1:9090/api/v1/query \
  --data-urlencode 'query=demo_queue_depth' | jq '.data.result'

A single result with value 12.

Task 7: Break hop 3, so you can tell the signatures apart

Hop 1 produced no scrape job at all. Hop 3 produces something different, and confusing the two costs time.

The endpoints[].port field is a Service port name, not a container port and not a number. Point it at the number instead:

kubectl patch servicemonitor -n demo demo-exporter --type=json \
  -p='[{"op": "replace", "path": "/spec/endpoints/0/port", "value": "8080"}]'

Wait a minute, then look:

curl -s http://127.0.0.1:9090/api/v1/targets \
  | jq -r '.data.activeTargets[] | "\(.health)\t\(.labels.job)"' | grep demo \
  || echo "demo target gone"

grep -c 'job_name' prometheus-generated-after.yaml
kubectl get secret -n monitoring "prometheus-$PROM_CR" \
  -o jsonpath='{.data.prometheus\.yaml\.gz}' | base64 -d | gunzip \
  | grep -c 'job_name'

The scrape job is still generated — the ServiceMonitor still matches the Prometheus, and still matches the Service — but it selects no endpoint, so no target appears. That is the signature: a job in the config with zero targets means the failure is downstream of hop 1. No job in the config at all means hop 1.

Put it back:

kubectl patch servicemonitor -n demo demo-exporter --type=json \
  -p='[{"op": "replace", "path": "/spec/endpoints/0/port", "value": "http-metrics"}]'

Validation

Run these against the finished state. Each one proves an outcome rather than restating a step.

cd "$HOME/k8s-prometheus-lab"

# 1. The target exists and is healthy.
curl -s http://127.0.0.1:9090/api/v1/targets \
  | jq -r '.data.activeTargets[] | select(.scrapeUrl | contains("8080")) | .health'

# 2. The metric has a value, and the value is the one the file serves.
curl -sG http://127.0.0.1:9090/api/v1/query \
  --data-urlencode 'query=demo_queue_depth' | jq -r '.data.result[0].value[1]'

# 3. The series carries Kubernetes identity, not just a scrape URL.
curl -sG http://127.0.0.1:9090/api/v1/query \
  --data-urlencode 'query=demo_queue_depth' | jq '.data.result[0].metric'

# 4. Cluster object state is present too, from kube-state-metrics.
curl -sG http://127.0.0.1:9090/api/v1/query \
  --data-urlencode 'query=count(kube_pod_info) by (namespace)' | jq -r '.data.result[]'

# 5. The evidence trail exists and shows the change.
diff prometheus-generated-before.yaml prometheus-generated-after.yaml | head -20

Check 2 must return 12. Check 3 must include namespace="demo" and a pod label — if it does not, the target was discovered but the Kubernetes service-discovery labels were dropped, which is a different defect. Check 4 must return one row per namespace: that is the taxonomy from Part LXXXVI, and it proves kube-state-metrics is being scraped as well as your own endpoint.

Expected Outcome

k8s-prometheus-lab/
├── chart-version.txt
├── crds-added.txt
├── crds-before.txt
├── demo-exporter.yaml
├── namespaces-before.txt
├── nodes-before.txt
├── prometheus-generated-before.yaml
├── prometheus-generated-after.yaml
├── prometheus-lab-values.yaml
├── servicemonitor-broken.yaml
├── targets-before.txt
└── targets-after.txt

On the cluster: a monitoring namespace running the Operator, one Prometheus, kube-state-metrics and a node-exporter per node; a demo namespace with one nginx pod serving a static exposition file; and one ServiceMonitor carrying release=prometheus whose target is up.

Production notes

This exercise maps onto a real change in two places, and they are not the same change.

Installing the stack is a cluster-scoped change: it adds CRDs, a cluster-wide DaemonSet with host mounts, and RBAC that lets the Operator read across namespaces. It belongs in a change window, with the CRD list recorded in the change record, because CRD removal is the part that cannot be rolled back cleanly later.

Adding a ServiceMonitor is not that. It is a namespaced object owned by an application team, and it should be reviewable and rollback-able on its own. The failure mode to watch is not the apply — it is the silence afterwards. Make “the target is up and the metric returns a value” an explicit acceptance criterion of the change, checked from the Prometheus API, before the change is closed. A ServiceMonitor merged into Git and never verified is the exact object the Scenario team had.

Holding is a legitimate outcome. If Task 3 shows the platform itself is degraded — Prometheus restarting, targets flapping — do not proceed to add targets. Adding scrape load to an unhealthy Prometheus makes diagnosis harder and can push it over its memory limit. Record the state, stop, and hand it on with a named owner and a time you will pick it up again.

Troubleshooting

The Prometheus pod is Pending. kubectl describe pod will name the reason. On a small lab cluster it is almost always insufficient memory against the 512Mi request. Reduce the request in the values file and helm upgrade, or give the worker more RAM.

helm install times out on a webhook. The chart installs an admission webhook and a Job that patches its certificate. If that Job cannot run — no schedulable node, image pull failure — the install hangs. Check kubectl get jobs -n monitoring and the Job’s pod logs.

The port-forward dies partway through. It is a single TCP session to a single pod; if the pod restarts, the forward drops. Restart it. If it drops repeatedly, Prometheus is being OOM-killed — check kubectl get pod -n monitoring -o wide for restart counts before assuming the network is at fault.

The target appears but is down with a connection-refused error. Hop 3 worked and the scrape itself failed. Check that the container is listening on 8080 and that the readiness probe passes; an unready pod is not in the EndpointSlice, so it is not a target at all.

The metric is present but has no pod or namespace label. Something dropped the discovery labels — usually a metricRelabelings block copied from elsewhere. Read the generated config for your job and look at the relabel rules.

kubectl get prometheus returns “no resources found”. The CRDs installed but the Prometheus CR did not, which means the Helm release failed partway. helm status prometheus -n monitoring.

Cleanup

Order matters here. Delete the namespaced objects first, then the release, then — only if Task 1 said they were not there before — the CRDs. Reverse that order and you orphan custom resources whose CRD is gone, which leaves objects that kubectl can no longer name.

Step 1. Remove what you deployed:

cd "$HOME/k8s-prometheus-lab"

kubectl delete -f demo-exporter.yaml --ignore-not-found
helm uninstall prometheus --namespace monitoring
kubectl delete namespace monitoring --ignore-not-found

Step 2. Look at what is still there:

cd "$HOME/k8s-prometheus-lab"

kubectl get crd -o name | sort > crds-after-uninstall.txt
comm -13 crds-before.txt crds-after-uninstall.txt

Every CRD the install added is still present. helm uninstall does not remove CRDs, by design — Helm cannot know whether another release, or a hand-written object, depends on them.

Deleting a CRD deletes every object of that kind, cluster-wide, with no confirmation

kubectl delete crd servicemonitors.monitoring.coreos.com does not delete “the lab’s ServiceMonitors”. It deletes every ServiceMonitor in the cluster, in every namespace, including any that another team applied and any that a second monitoring stack depends on. The objects are gone immediately and are not recoverable from the API.

On a shared cluster this is the step that turns a tidy-up into an incident. crds-added.txt from Task 2 is what makes it safe: delete only the names on that list, and only after confirming with comm that they were not present before you started.

Print what you are about to delete, and read it, before deleting it:

cd "$HOME/k8s-prometheus-lab"
cat crds-added.txt

Step 3. Remove them, on a disposable cluster only:

Destructiveworkstation— Deletes every custom resource of these kinds across the whole cluster

$ xargs -r kubectl delete --ignore-not-found < crds-added.txt

Step 4. Prove the cluster is back where it started:

cd "$HOME/k8s-prometheus-lab"

kubectl get crd -o name | sort > crds-after.txt
diff crds-before.txt crds-after.txt && echo "CRDs restored to the starting set"

kubectl get namespaces -o name | sort | diff namespaces-before.txt - \
  && echo "namespaces restored to the starting set"

Both diffs must be empty. Keep targets-before.txt, targets-after.txt and the two generated configs — they are the deliverables — and delete the rest of the working directory by hand if you want the space back.

Under the hood

There is no code path in Prometheus that knows what a ServiceMonitor is. Prometheus reads a YAML configuration file containing scrape_configs, exactly as it does outside Kubernetes. Everything the Operator adds happens before that file exists.

The Operator watches Prometheus, ServiceMonitor, PodMonitor, Probe and PrometheusRule objects. When any of them changes it re-renders the whole configuration: for each ServiceMonitor whose labels match the Prometheus’s serviceMonitorSelector, and whose namespace matches serviceMonitorNamespaceSelector, it emits one scrape job per entry in spec.endpoints. Each job uses kubernetes_sd_configs with role: endpointslice, plus a stack of relabel_configs that keep only the endpoints belonging to Services matching spec.selector, filter to the named port, and copy the namespace, service, pod and container names onto the series as labels. That relabel stack is why your metric arrives with Kubernetes identity attached rather than just an IP address.

The rendered file is gzipped and written into a Secret named prometheus- plus the Prometheus object’s name. The Prometheus StatefulSet mounts that Secret, and a config-reloader sidecar container watches the mounted file; when the contents change it asks Prometheus to reload. This is why a ServiceMonitor change takes effect without a restart, and why the delay between kubectl apply and a new target appearing is the sum of the Operator’s reconcile, the kubelet’s Secret-projection refresh, and the reload — comfortably under a minute, but never instant.

It also explains the diagnostic order used in Task 6. The Secret is the boundary between “the Operator’s opinion” and “Prometheus’s behaviour”. If your job is in that file, the Operator did its part and the fault is in discovery or the scrape. If it is not, nothing downstream can help you.

What You Learned

A scrape target is an HTTP endpoint returning text. You proved it with nginx and a static file. No agent, no SDK, no registration.
Three label hops connect a Prometheus to a pod, they fail independently, and each failure is silent. Selector to ServiceMonitor, ServiceMonitor to Service, endpoint port name to Service port name.
The two failure signatures differ. No scrape job in the generated config means hop 1. A scrape job with zero targets means hop 2 or hop 3. Knowing which you are looking at removes half the search space.
The Operator-generated Secret is the authoritative evidence, and you can read it in one command. Dashboards and the targets page are downstream of it.
Check the platform before you check your object. Fifteen seconds of “is Prometheus scraping anything at all” eliminates half the causes.
helm uninstall is not an uninstall. CRDs outlive the release, and removing them is a cluster-wide destructive act that needs the list you captured before you started.

Objectives

Prerequisites