Prometheus Production Readiness

The configuration passes the offline schema check, run as a gate rather than as a habit. It calls the same load path the daemon uses at reload, and exits non-zero, so it is the cheapest place to catch an unknown key or a malformed job. The alternative is discovering it at reload time, when a single bad file rejects the whole configuration and holds every other pending change behind it.

promtool check config /etc/prometheus/prometheus.yml

Every rule file parses and the rule unit tests pass. Parsing proves the PromQL is syntactically valid; only a test that spans the alert's pending duration proves the rule means what its author intended. A `for` written as seconds instead of minutes passes every static check and flaps at every evaluation, and an alert nobody trusts is an alert nobody reads.

promtool check rules /etc/prometheus/rules/*.yml && promtool test rules /etc/prometheus/tests/*.yml

The running process reports a successful last reload, and the timestamp matches the change you expect. A failed reload does not stop Prometheus: it keeps serving the previous configuration and logs the failure once. Everything then looks healthy while the change you believe is live is not, which is a state that can persist for weeks with no other symptom.

curl -s http://localhost:9090/metrics | grep -E '^prometheus_config_last_reload'

The external labels give this server a unique identity — a cluster or region label plus a per-instance replica value. They are stamped on everything leaving the process, so without them an HA pair looks like one source to Alertmanager and pages twice for one incident, and looks like one uploader to a long-term store, which then rejects the second replica's blocks. This is identity, not a default.

curl -fsS http://localhost:9090/api/v1/status/config | jq -r '.data.yaml' | grep -A5 external_labels

The effective scrape timeout is inside the effective scrape interval on every job, and the evaluation interval is a deliberate number. The global block sets defaults that per-job settings override, so the effective value for a job is not always the one written in the global section. A timeout longer than the interval is rejected outright; an evaluation interval nobody chose quietly decides how fast every alert can fire.

curl -fsS http://localhost:9090/api/v1/status/config | jq -r '.data.yaml' | grep -E "scrape_interval|scrape_timeout|evaluation_interval"

Both retention limits are configured, and you know which one will bind first. Time expresses the intent — how far back somebody needs to look — and size expresses the physics of the volume; whichever is reached first wins. Setting only time means the disk is the real limit and it enforces itself by filling up, which on this host is a monitoring outage rather than a capacity warning.

curl -fsS http://localhost:9090/api/v1/status/flags | jq -r '.data["storage.tsdb.retention.time"], .data["storage.tsdb.retention.size"]'

The oldest sample actually held is roughly as old as the retention policy promises. If it is much newer, the size limit or the disk is silently overriding the intent and nobody has been told that the platform holds a fortnight rather than a quarter. This one query is the difference between a retention policy and a retention aspiration, and it is the number an incident review will ask for.

curl -fsS -G http://localhost:9090/api/v1/query --data-urlencode 'query=prometheus_tsdb_lowest_timestamp' | jq -r '.data.result[0].value[1]'

The volume has meaningful free space above the configured block budget. The size limit counts compacted blocks only: the write-ahead log, the in-memory head's chunks and the scratch space compaction needs while rewriting a block all sit outside it. Twenty to thirty percent headroom is the usual figure, and it exists because compaction is also the mechanism that would free space — so a full disk deadlocks its own recovery.

df -h /var/lib/prometheus; du -sh /var/lib/prometheus

Disk usage on this volume is alerted at staged thresholds by a system that is not this Prometheus. When the volume fills, appends fail, ingestion stalls and scrapes time out — and the component that would normally page you about it is the one that has stopped. This is the clearest example on the list of why a monitoring server cannot be its own monitor.

curl -fsS -G http://meta-prom:9090/api/v1/query --data-urlencode 'query=node_filesystem_avail_bytes{mountpoint="/var/lib/prometheus"}' | jq -r '.data.result[] | "\(.metric.instance) \(.value[1])"'

Where remote write is configured, the queue settings come from the measured sample rate and something alerts on the failure and drop counters. The queue is per shard and in memory, which makes it the buffer that decides whether a brief destination outage is invisible or is a permanent gap. Failures here are silent locally: queries against this server keep working perfectly while the long-term store develops a hole.

curl -s http://localhost:9090/metrics | grep -E '^prometheus_remote_'

The server reports the expected number of Alertmanagers as reachable, and none as dropped. Alert delivery has two halves that fail independently, and this half fails without any symptom at all: rules keep firing, alerts have nowhere to go, and the observable result is an absence of pages, which is indistinguishable from things being fine.

curl -fsS http://localhost:9090/api/v1/alertmanagers | jq '{active: (.data.activeAlertmanagers|length), dropped: (.data.droppedAlertmanagers|length)}'

More than one Alertmanager is running and clustered. A single instance is a single point of failure for the one function the platform exists to perform, and it fails in the direction that is hardest to notice — nothing arrives, and nothing reports that nothing arrived. Send alerts to every peer and let the cluster deduplicate; that is what the gossip layer is for.

curl -fsS http://alertmanager.internal:9093/api/v2/status | jq '.cluster'

Somebody has pushed a test alert through this delivery path and confirmed it arrived at the receiver a real person watches. Every other alerting item here checks a component boundary; this is the only one that crosses all of them at once, including the templates and the integration credentials. Until it has been done, the alerting path is a configuration that has never been executed.

amtool --alertmanager.url=http://alertmanager.internal:9093 alert add readiness_smoke severity=info

One alert is always firing by design, and something outside this platform notices when it stops arriving. Every other alert proves something is wrong; this is the only one that proves the pipeline still works. Without it, a Prometheus that stopped evaluating, an Alertmanager that lost its route, and a genuinely quiet night all produce exactly the same evidence: silence.

curl -fsS -G http://meta-prom:9090/api/v1/query --data-urlencode 'query=ALERTS{alertname="Watchdog",alertstate="firing"}' | jq '.data.result | length'

No rule group is missing evaluations. A group whose evaluation takes longer than its interval falls behind, and everything downstream of it — alerts, recorded series, SLO burn rates — is being decided on data that is progressively staler. The rules stay healthy throughout, so the only signal is the missed-iteration counter, and it needs an alert rather than an occasional look.

curl -fsS -G http://localhost:9090/api/v1/query --data-urlencode 'query=rate(prometheus_rule_group_iterations_missed_total[10m]) > 0' | jq -r '.data.result[] | "\(.metric.rule_group) \(.value[1])"'

The heaviest scrape targets have been looked at, and a sample ceiling exists for jobs whose output you do not control. One exporter that starts emitting a label with unbounded values can multiply this server's series count in a single scrape interval, and the failure is memory exhaustion on the monitoring host rather than a gradual slope. A per-scrape limit turns that into a failed target, which is a far better Tuesday.

curl -fsS -G http://localhost:9090/api/v1/query --data-urlencode 'query=topk(10, scrape_samples_scraped)' | jq -r '.data.result[] | "\(.metric.job) \(.metric.instance) \(.value[1])"'

The listener is bound to loopback or a private interface, not to every address. The default binds everywhere, and everything else in this section — authentication, TLS, which endpoints are enabled — only decides what a caller who can already reach the port is allowed to do. The bind address decides who the audience is, which makes it the first control rather than an additional one.

ss -tlnp | grep 9090

The HTTP API and UI require credentials and are served over TLS, configured through the web configuration file with bcrypt-hashed users. Without it, anyone who reaches the port can read every metric this server holds and enumerate the entire scrape configuration. Confirm by making an unauthenticated request and getting a refusal — a configuration file that looks right is not the same as a listener that rejects.

curl -sk -o /dev/null -w "%{http_code}\n" https://prometheus.internal:9090/api/v1/status/config

The admin API and the remote lifecycle endpoints are disabled unless a specific workflow needs them, in which case they are behind authentication and a restricted network path. With the admin API on, a caller who can reach the port can snapshot the whole database and take a copy of everything this server knows; with the lifecycle endpoints on, the same caller can stop the process over HTTP. Reload from the local supervisor instead.

curl -fsS http://localhost:9090/api/v1/status/flags | jq -r '.data["web.enable-admin-api"], .data["web.enable-lifecycle"]'

Scrape and remote-write credentials are file references, never literal values in the configuration. The configuration is version-controlled and readable by everyone with repository access, and it is also reachable through the server's own API. The same argument rules out passing a token as a process flag, where it is visible in the process arguments to any local account.

grep -nE "^[[:space:]]*(password|bearer_token|credentials):" /etc/prometheus/prometheus.yml /etc/prometheus/*.yml

The process runs as its own system account with no interactive shell, no capabilities, and no ability to gain any. Prometheus needs to read its config, write its data directory and open outbound connections; nothing on that list needs privilege. The account and the file modes are also what keep the configuration — which is a map of credentials — unreadable by every other service on the host.

getent passwd prometheus; ls -l /etc/prometheus/prometheus.yml; ls -ld /var/lib/prometheus

The service unit makes the filesystem read-only apart from the data directory, drops capabilities, and raises the open-file limit. The default descriptor limit is comfortably too small past a few hundred targets, and the failure it produces is scrapes intermittently failing under load rather than an obvious error — which reads as a network problem and is diagnosed as one, at length.

systemctl show prometheus -p ProtectSystem -p ReadWritePaths -p NoNewPrivileges -p CapabilityBoundingSet -p LimitNOFILE

A different Prometheus, on separate infrastructure, scrapes this one and alerts on its absence. Self-monitoring cannot report the failure that matters most, because a server that is down produces no data about being down and no alert about the missing data. The meta-monitor needs to share as little as possible with this platform — separate host, separate notification path — or it fails in the same event.

curl -fsS -G http://meta-prom:9090/api/v1/query --data-urlencode 'query=up{job="prometheus"}' | jq -r '.data.result[] | "\(.metric.instance) \(.value[1])"'

If this runs as an HA pair, both members scrape every target independently and sit on separate failure domains. Prometheus HA is not a cluster: it is two servers doing the same work, deduplicated downstream, so it costs double the scrape load and double the storage and buys nothing if both are placed on the same hypervisor, rack or availability zone. Confirm both are running and both are being seen.

curl -fsS -G http://meta-prom:9090/api/v1/query --data-urlencode 'query=count by (replica) (up{job="prometheus"} == 1)' | jq -r '.data.result[] | "\(.metric.replica) \(.value[1])"'

Ask the process, not the file

The two failure modes worth internalising

Sign-off

Critical24 items

Warning5 items