Lab 17: Prometheus + cAdvisor + Grafana for container hosts

Objective

Stand up the canonical observability stack for a Docker host. Verify metrics flow end-to-end.

Tasks

Task 1: Compose file

# compose.yml
services:
  prometheus:
    image: prom/prometheus:v2.55.0
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
    ports:
      - "9090:9090"
    networks: [obs]

  cadvisor:
    image: gcr.io/cadvisor/cadvisor:v0.49.1
    privileged: true
    devices:
      - /dev/kmsg
    ports:
      - "8080:8080"
    networks: [obs]
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro

  grafana:
    image: grafana/grafana:11.3.0
    environment:
      GF_SECURITY_ADMIN_PASSWORD: admin
    ports:
      - "3000:3000"
    networks: [obs]

networks:
  obs:

Task 2: Prometheus config

# prometheus.yml
global:
  scrape_interval: 15s
scrape_configs:
  - job_name: cadvisor
    static_configs:
      - targets: ['cadvisor:8080']

Task 3: Bring up

docker compose up -d
sleep 30
docker compose ps

Task 4: Verify Prometheus is scraping

curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | {job: .labels.job, health: .health}'

You should see cadvisor with health: up.

Task 5: Query a metric

curl -s 'http://localhost:9090/api/v1/query?query=container_cpu_usage_seconds_total' | jq '.data.result[0]'

A non-empty result means container metrics are flowing.

Task 6: Grafana

Open http://localhost:3000 (admin/admin), add Prometheus as a data source at http://prometheus:9090, and import dashboard ID 893 (Docker / cAdvisor overview).

Cleanup

docker compose down -v