Securing the monitoring stack

Why this matters in production

Metrics endpoints are unauthenticated by default and disclose the shape and state of the cluster in detail.

What each component exposes

ceph mgr services
curl -s "http://$(hostname -s):9283/metrics" | head -20

Metric names disclose pool names, host names, OSD counts, and capacity.
That is not a credential, but it is reconnaissance.

Restricting the metrics endpoints

# what the module is bound to
ceph config get mgr mgr/prometheus/server_addr
ceph config set mgr mgr/prometheus/server_addr 10.10.5.11

ceph config get mgr mgr/prometheus/server_port

The same reasoning as the dashboard: binding to a management interface
removes the endpoint from networks that should not see it.

# node-exporter, deployed by cephadm
ceph orch ls node-exporter --export

service_type: node-exporter
placement:
  host_pattern: '*'
networks:
  - 10.10.5.0/24

ceph orch apply -i node-exporter.yaml

Alertmanager and Grafana

Alertmanager without authentication permits anyone reachable to create
silences — which suppresses alerts cluster-wide.

ceph orch ls grafana --export
ceph dashboard get-grafana-api-url

# confirm anonymous access is not enabled
ceph orch ps --daemon-type grafana --format json | python3 -c '
import sys,json
for d in json.load(sys.stdin): print(d["daemon_name"], d.get("hostname"), d.get("ports"))'

Silence creation is the one that matters operationally: an attacker who
can silence alerts can act without the alerting path noticing.

Balancing access

Monitoring exists to be looked at. Restricting it so far that nobody
consults it during an incident defeats it.

# CEPH_HOST is the manager host serving the metrics endpoint; substitute your own:
CEPH_HOST=192.0.2.11

# verify from an unintended position that it is not reachable
curl -s -m 3 "http://$CEPH_HOST:9283/metrics" >/dev/null \
  && echo "REACHABLE — restrict this" || echo "not reachable"

Quiz

Production discipline

Put authentication in front of Alertmanager or restrict it to the management network — silence creation needs no authentication by default, and silencing alerts is the exposure with real operational consequence. Restrict metric endpoints via the service spec’s networks: field.

Cross-course references

• Kubernetes: unauthenticated metrics endpoints are the same reconnaissance exposure
• Linux: the ability to suppress alerting outranks the ability to read metrics