Global Settings

The pager fires twice for one incident. Two Prometheus servers run as an HA pair, scraping the same targets and evaluating the same rules, and Alertmanager is supposed to deduplicate their alerts. It does not: one server tags its alerts cluster: eu-1, the other tags them prometheus: eu-1b, and to Alertmanager those are two different alerts. The fix takes one line of YAML on one host — but only once you understand what the global: block actually controls.

That block is the topic of this lesson. It is the smallest section of prometheus.yml and the one most often cargo-culted from an old example without anyone being able to say which values are deliberate.

What the global block is

The global: block sets defaults and identity for one Prometheus server. In Prometheus 2.55 it carries four keys that matter day to day:

There is also scrape_protocols, which sets the content negotiation preference (OpenMetrics first, Prometheus text format as fallback). The default is right for almost every deployment; leave it alone.

Two properties make the global block different from everything else in the file:

1. These are defaults, not commands. A per-job scrape_interval wins over the global one. The global value only applies where the job stays silent.
2. external_labels is not a default at all. It is identity. It never touches the local TSDB; it is stamped onto data as it leaves the server.

How the overrides resolve

 global:                          scrape_configs:
   scrape_interval: 30s  -------->  - job_name: node
   scrape_timeout: 10s       |        (no interval set)
   evaluation_interval: 30s  |      - job_name: app
   external_labels: {...}    |        scrape_interval: 15s
                             |
        effective config:    |
          node -> 30s interval / 10s timeout   (inherits globals)
          app  -> 15s interval / 10s timeout   (interval overridden,
                                                 timeout still global)

The constraint that bites people: the effective scrape_timeout must not exceed the effective scrape_interval. Prometheus checks this at load time for every job after the global and per-job values are merged. Lower the global interval to 5s while a job inherits the default 10s timeout, and the entire configuration is rejected:

Checking /etc/prometheus/prometheus.yml
  FAILED: parsing YAML file /etc/prometheus/prometheus.yml:
  scrape timeout greater than scrape interval ("10s" > "5s")

Note the failure shape: if this happens during a reload, the old configuration keeps running. Nothing crashes. The only signals are a log line and a metric — which is why the validation lesson makes prometheus_config_last_reload_successful a first-class alert.

evaluation_interval sets the cadence for every rule group that does not declare its own interval. It is also the resolution of your alert detection: a condition that becomes true one second after an evaluation waits, on average, half an interval before any rule sees it. Set it to 5m to save CPU and you have silently added up to five minutes of detection latency to every alert that relies on the default.

external_labels: the identity of this Prometheus

external_labels answers one question: when data from this server arrives somewhere else, what labels say where it came from?

The labels are attached on the way out, to three kinds of traffic:

• Alerts sent to Alertmanager.
• Samples and metadata sent via remote_write.
• Series served by federation.

They are not added to series stored in the local TSDB. A PromQL query against the local server never sees them. And they follow a fill-if-absent rule: if a series or alert already carries a label with the same name, the external label is not applied. The data’s own label wins.

Those three behaviours explain the two canonical uses:

HA pairs. Give both members of a pair the same external labels. Their alerts then arrive at Alertmanager with identical label sets and are deduplicated naturally. Give them different labels and every alert pages twice.

Remote-write identity. A central store (Mimir, Cortex, Thanos, Grafana Cloud) receiving from fifty Prometheus servers needs cluster/environment labels to keep the streams apart. If the store deduplicates HA pairs (Thanos replica labels, Mimir HA tracker), the pair typically shares a cluster label and carries a distinct replica label the store knows how to handle.

The labeldrop decision comes at the destination boundary. External labels ride on every remote-written sample — bytes on the wire and cardinality at the store. If the destination only needed replica for deduplication, strip it in that destination’s write_relabel_configs (covered in the remote-write lesson) rather than carrying it forever. Keep the set small: environment, cluster, and replica only when the store needs it. Per-host identity belongs in target labels, not here.

Configuring the global block

A deliberate production global block, annotated:

global:
  # Fleet default: 30s is enough for most infrastructure exporters.
  # Jobs with tighter SLAs override it locally.
  scrape_interval: 30s

  # Set explicitly, never inherited blindly. If a job later overrides
  # its interval below 10s, the load-time check fails loudly instead
  # of silently truncating scrapes.
  scrape_timeout: 10s

  # Rule cadence. 30s halves the default detection latency for groups
  # without their own interval at a modest query cost.
  evaluation_interval: 30s

  external_labels:
    environment: production
    cluster: prometheus-eu-1
    # Only if the remote store dedupes on it:
    # replica: a

The same file on the HA partner differs by nothing at all when Alertmanager does the deduplication, or by exactly the replica value when the remote store does it. That is the whole decision.

Validating what is actually running

The file on disk is not the configuration Prometheus is running. Check both:

# 1. Offline validation before anything is applied (exit 0 = valid)
promtool check config /etc/prometheus/prometheus.yml
# Checking /etc/prometheus/prometheus.yml
#   SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config file syntax

# 2. The effective configuration the server loaded (defaults merged,
#    secrets redacted)
curl -s http://localhost:9090/api/v1/status/config | jq -r .data.yaml | head -12

# 3. Apply and confirm the reload actually landed
curl -sf -X POST http://localhost:9090/-/reload
curl -s http://localhost:9090/metrics | grep '^prometheus_config_last_reload'
# prometheus_config_last_reload_success_timestamp_seconds 1.7866e+09
# prometheus_config_last_reload_successful 1

If prometheus_config_last_reload_successful is 0, the file you just edited is not what is running — the previous valid configuration is.

How global settings fail

1. Timeout exceeds interval after a change. Reload rejected, old config keeps running. Symptom: prometheus_config_last_reload_successful is 0 and the log shows scrape timeout greater than scrape interval. The editor believes the change applied; it did not.
2. Global interval too aggressive. Someone sets 5s globally for one dashboard and every job inherits it. Symptom: ingestion rate and head-block churn multiply, remote-write lag appears, CPU climbs on every target and on Prometheus itself.
3. HA pair with mismatched external labels. Symptom: duplicate pages for single incidents; at a remote store without dedup, sporadic “duplicate sample for timestamp” rejections as the pair scrapes the same targets at slightly different offsets.
4. External labels changed on a running fleet. The destination sees a brand-new label set, which is a brand-new series. Symptom: graphs and alerts at the long-term store split at the moment of the change; history appears to end.
5. External label shadowed by target data. A job or exporter already sets cluster, so the external cluster never applies (fill-if-absent). Symptom: inconsistent identity at the destination — some series carry the target’s value, some carry none of yours.
6. evaluation_interval stretched to save CPU. Symptom: mean time to detect grows by minutes and nobody connects it to the config change; recording-rule-backed dashboards look stale.

Troubleshooting, in order

1. What is the server actually running? curl ... /api/v1/status/config and compare with the file on disk. Drift here means a failed reload.
2. Did the last reload succeed? The metric above, plus journalctl -u prometheus | grep -i 'configuration' for the load error text.
3. Which interval is effective for the job? Read the merged config from step 1 — the status endpoint shows the job as Prometheus sees it, not as the YAML file implies.
4. What identity does the destination see? Query the long-term store for a known series and inspect its labels. Missing or stale external labels there, with a correct local config, means the change never reloaded or the destination caches identity.
5. What does Alertmanager receive? Compare the label sets of “duplicate” alerts from pair members. One differing label explains the double page.

Security implications

External labels are content, and content leaves the perimeter. They name your clusters, environments and topology to every system that receives alerts or remote-written samples — useful to an attacker reading a leaked notification. Keep them coarse. Never place anything secret in a label, external or otherwise; labels are data, not configuration hygiene. The file itself holds no credentials in this block, but it sits beside blocks that do — keep prometheus.yml at 0640 root:prometheus as a habit.

Performance implications

The arithmetic is unforgiving: samples per second equals active series divided by effective scrape interval, summed over jobs. Halving the global interval doubles ingestion, head memory pressure, WAL write rate and remote-write egress. evaluation_interval divides total rule query cost by its value for the CPU bill. External labels add cardinality at the destination — one extra label value pair per series is trivial, but a label with a unique value per send (a build hash, a pod UID) recreates every series at the store. The levers are small and they are all multipliers.

Production guidance

• Pick two or three intervals for the whole fleet — for example 15s for core infrastructure jobs, 30s as the global default — and make any other value justify itself in the job’s comments.
• Set scrape_timeout explicitly in the global block. The inherited default is how silent timeout-interval collisions happen.
• Set evaluation_interval from your detection-latency budget, not from a CPU graph. If rules are too expensive, fix the rules.
• Keep external labels to environment, cluster, and replica when the store needs it. Record the choice in the team runbook; both HA members and every future migration depend on it.
• Manage the file with configuration management, reload rather than restart, and alert on prometheus_config_last_reload_successful == 0.

Verification

You should now be able to answer:

• What are the four keys of the global block and their 2.55 defaults?
• Which value wins when global and per-job intervals disagree, and what invariant does Prometheus enforce between timeout and interval?
• What three kinds of outbound traffic receive external labels, and what never receives them?
• Why do identical external labels on an HA pair stop duplicate pages, and what happens at a remote store when they differ?
• How do you prove which configuration the server is actually running?

Quiz