Prometheus + Grafana monitoring lab

This lab produces a working monitoring stack for a PVE cluster: metrics collection, storage, dashboards, and alerts.

Steps

1. Deploy node-exporter on each PVE node

# On every PVE node
apt install -y prometheus-node-exporter
systemctl enable --now prometheus-node-exporter

# Verify
curl http://localhost:9100/metrics | head

2. Install prometheus-pve-exporter

For PVE-specific metrics (VMs, storage, replication):

# On the monitoring host (LXC/VM)
apt install -y python3-pip
pip install prometheus-pve-exporter

# Create a config
cat > /etc/pve-exporter/pve.yml << 'EOF'
default:
  user: monitoring@pve
  password: <password>
  verify_ssl: false
EOF

# Create the user on PVE
pveum useradd monitoring@pve --comment "Monitoring exporter"
pveum passwd monitoring@pve
pveum aclmod / --roles PVEAuditor --user monitoring@pve

Run the exporter:

prometheus-pve-exporter --config.file=/etc/pve-exporter/pve.yml --addr=0.0.0.0:9221

3. Deploy Prometheus (Docker)

# On the monitoring host
mkdir -p /opt/monitoring
cat > /opt/monitoring/docker-compose.yml << 'EOF'
services:
  prometheus:
    image: prom/prometheus:latest
    ports: ["9090:9090"]
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus-data:/prometheus
  grafana:
    image: grafana/grafana:latest
    ports: ["3000:3000"]
    volumes:
      - grafana-data:/var/lib/grafana

volumes:
  prometheus-data:
  grafana-data:
EOF

cat > /opt/monitoring/prometheus.yml << 'EOF'
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'node'
    static_configs:
      - targets: ['pve-01:9100', 'pve-02:9100', 'pve-03:9100']

  - job_name: 'pve'
    static_configs:
      - targets: ['localhost:9221']

  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
EOF

docker compose -f /opt/monitoring/docker-compose.yml up -d

4. Verify Prometheus is scraping

Open http://<monitoring-host>:9090/targets in a browser. All targets should be UP.

5. Configure Grafana

Open http://<monitoring-host>:3000, log in (admin / admin), and:

1. Add Prometheus as a data source: URL = http://prometheus:9090
2. Import dashboard 1860 (Node Exporter Full)
3. Save

6. Create an alert rule

In Grafana → Alerting → Alert rules:

- alert: NodeDown
  expr: up{job="node"} == 0
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: "PVE node {{ $labels.instance }} is down"

- alert: HighCPU
  expr: 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
  for: 10m
  labels:
    severity: warning

7. Trigger the alert (test)

# Stop node-exporter on pve-02 to test NodeDown
ssh root@pve-02 systemctl stop prometheus-node-exporter
# Wait 2 minutes; alert should fire
ssh root@pve-02 systemctl start prometheus-node-exporter

Verification

• Prometheus targets all show “UP”
• Grafana dashboard shows real-time metrics from all nodes
• Alert fires when a node goes down
• PVE-specific metrics (VMs, storage) appear

Cleanup

docker compose -f /opt/monitoring/docker-compose.yml down -v
apt remove -y prometheus-node-exporter
# On each PVE node
pveum userdel monitoring@pve

Notes

For production, add:

• Long-term storage (Thanos or Cortex) so you have months of metrics
• Slack / email alert routing (Grafana contact points)
• TLS and authentication on the Prometheus and Grafana endpoints
• A separate alerting path (Alertmanager) so alerts work even if Grafana is down