The Prometheus Pull Model

At 04:10 the pager fires: up{job="node", instance="db-01"} = 0. Nobody told Prometheus the exporter had died. Prometheus found out itself, because it was Prometheus that made the call — and the call failed. That property, liveness as a side effect of collection, is the core of the pull model, and it shapes everything about how you run Prometheus in production.

What it is

Prometheus collects metrics by pulling them: on a fixed schedule it initiates an HTTP GET against each target’s metrics endpoint — conventionally /metrics — parses the response, and stores the samples. The target is passive. It never opens a connection to Prometheus; it may not even know Prometheus exists.

This is a deliberate inversion of the older push tradition (Graphite, StatsD, the InfluxDB write API, the OpenTelemetry Collector forwarding OTLP), where the producer opens a socket and delivers telemetry to the platform.

Why Prometheus chose pull

The choice was argued about for years. It survives because of four concrete operational properties:

1. Liveness insight. Every scrape produces the synthetic up metric: 1 if the scrape succeeded, 0 if it failed. In a push system, a silent producer is ambiguous — dead host, dead telemetry path, or simply nothing to say? In a pull system the failure is the signal.
2. Central control. The scrape schedule, timeouts, credentials and TLS settings live in one configuration file. You change the collection rate of a thousand-node fleet by editing one value.
3. Trivial local testing. Run the exporter, then curl localhost:9100/metrics. What you see is byte-for-byte what Prometheus will ingest. There is no client library between you and the truth.
4. Independent collectors. Two Prometheus servers can scrape the same targets without any coordination. That is how the standard HA pair works: not replication, just two servers pulling the same data.

A bonus property: the direction of load pressure favours the server. Prometheus decides how much work it does per interval; targets cannot flood it by chattering. (A target can still return an enormous body — the scrape limits in lesson 02 exist for that.)

Where push is still correct: the Pushgateway

Pull has one genuine blind spot: a job that starts and finishes between scrapes. A cron job that runs for four seconds, a CI pipeline step, a database backup — the exporter pattern does not work because there is no long-lived process to scrape.

The answer is the Pushgateway: a small daemon that accepts pushed metrics over HTTP and holds the last values. The batch job pushes when it finishes; Prometheus scrapes the Pushgateway like any other target.

Be honest about the costs, because they bite:

• up{job="pushgateway"} reports the health of the gateway, not of your batch job. A job that has not run for a month leaves the gateway perfectly green.
• Pushed metrics never expire. A job that ran once in March is still exporting its last values until somebody deletes them via the Pushgateway API.
• The gateway is a single point of failure and an aggregation point for everything pushed through it.

The upstream documentation is blunt: the Pushgateway is for service-level batch jobs, not for daemons. Using it for long-running services is the single most common Prometheus anti-pattern in the wild.

The operational consequences of pull

Choosing pull buys you three ongoing obligations:

1. Network reachability. Prometheus must be able to open a TCP connection to every target. Firewalls are configured inbound to the exporter ports (9100 for node_exporter, and so on), from the Prometheus server. In segmented networks this often means one Prometheus per segment, with federation or remote write (lessons 05 and 06) carrying data upward.
2. Coupling to service discovery. What Prometheus monitors is exactly what discovery returns: static_configs for small estates, file_sd_configs, consul_sd_configs, kubernetes_sd_configs, ec2_sd_configs and friends for real ones. The blind spot to remember: a host absent from discovery produces no up metric at all — not even 0. Pull gives you liveness for everything Prometheus knows about, and silence for everything it does not.
3. Credentials in the scrape config. Bearer tokens, basic auth and TLS client certificates live per-job in prometheus.yml (or in files it references). File permissions on that config matter.

How to configure it

A minimal but honest prometheus.yml:

global:
  scrape_interval: 30s       # how often every target is pulled
  scrape_timeout: 10s        # a slower scrape is failed; must not exceed the interval
  evaluation_interval: 30s
  external_labels:
    dc: eu-west-1            # attached to everything; vital for HA pairs

scrape_configs:
  - job_name: node
    file_sd_configs:
      # targets live in a JSON/YAML file your config management owns;
      # Prometheus picks up file changes WITHOUT a reload
      - files: ['/etc/prometheus/targets/node.yml']

  - job_name: pushgateway
    # keep the job/instance labels the batch job pushed, instead of
    # replacing them with the gateway's own labels
    honor_labels: true
    static_configs:
      - targets: ['pushgateway.internal:9091']

How to validate it

# 1. Config parses and is internally consistent
promtool check config /etc/prometheus/prometheus.yml

# 2. Server is up and has finished loading
curl -s localhost:9090/-/ready
# Prometheus Server is Ready.

# 3. What does Prometheus itself see? (the ground truth)
curl -s 'localhost:9090/api/v1/targets?state=active' \
  | jq '.data.activeTargets[] | {job: .labels.job,
        instance: .labels.instance, health: .health,
        lastError: .lastError}'

# 4. The platform-level view: which targets are down right now
curl -s 'localhost:9090/api/v1/query?query=up%20%3D%3D%200' \
  | jq '.data.result[].metric'

# 5. Reproduce the pull by hand, from the Prometheus host
curl -s http://db-01:9100/metrics | head -5

A healthy target entry shows "health": "up" and an empty lastError. Step 5 is the one people skip: if curl from the Prometheus host fails, nothing Prometheus-side will fix it.

How it can fail

1. Firewall silently drops the exporter port. Symptom: a whole subnet goes dark at once; up is 0 with lastError of “context deadline exceeded” and scrape_duration_seconds pinned at the scrape timeout.
2. Target down, port refused. Symptom: up is 0 within one interval, with lastError of “connect: connection refused”. Fast and loud — the model working as designed.
3. Stale or missing discovery. A decommissioned host left in a static file produces a permanent up of 0 page-farm; a brand-new host missing from discovery produces nothing — no data, no up, no alert. The second shape is worse.
4. Pushgateway used for daemons. Symptom: ghost series. A service deleted months ago still appears on dashboards and in alert evaluations, because nobody deleted its metrics from the gateway.
5. The same exporter scraped twice under two jobs (a static config plus an overlapping discovery rule). Symptom: doubled series, doubled disk growth, and sum() results that are mysteriously twice reality.
6. A slow exporter — classically node_exporter with a broken smartctl or a hung NFS mount. Symptom: intermittent timeouts, up flapping, graphs with gaps that correlate with nothing in the application.

How to troubleshoot it

In order:

1. What does the target list say? The /targets page (or the API above) shows health, lastError, and both the discovered and the final labels. Most pull problems are visible here at a glance.
2. Is it down or absent? An up of 0 means “known target, failing scrape”. A missing up series means discovery or relabeling — a completely different fix.
3. Reproduce the pull. Curl the endpoint from the Prometheus host. If curl fails, the problem is network, DNS or the exporter — not Prometheus.
4. Check the discovery source itself. The file_sd file’s contents, the Consul catalogue, the Kubernetes API. Prometheus faithfully mirrors whatever they say.
5. Confirm the config in effect. Compare /api/v1/status/config with the file on disk, and check prometheus_config_last_reload_successful. A failed reload means Prometheus is running yesterday’s configuration.
6. Read the logs. journalctl -u prometheus shows reload errors and scrape-level panics that the API does not surface.

Security implications

• Every exporter is an unauthenticated HTTP server publishing host internals: process names, kernel version, network configuration, sometimes file paths. Treat exporter ports like management interfaces; restrict them to the Prometheus server.
• Scrape credentials in prometheus.yml (or files it references) are readable by anyone who can read the file. Use 0640 with the prometheus user and group, and keep secrets in separate files referenced via password_file or bearer_token_file.
• The Pushgateway accepts writes from anything that can reach it. Anyone on the network can overwrite your batch-job metrics — or inject series that your alerts will happily evaluate.
• The Prometheus API itself is unauthenticated by default. Bind it to localhost or put it behind an authenticating reverse proxy.

Performance implications

Ingestion rate is roughly (series per target × number of targets) divided by the scrape interval. The pull model caps server-side load at the configured schedule — the trade-off is granularity against CPU and disk. Exporters pay a small CPU cost per scrape, which doubles when you run an HA pair; negligible for node_exporter, worth measuring for blackbox or SNMP-style exporters where a single scrape can take seconds.

Production guidance

• Run exporters as daemons on every node; reserve the Pushgateway for true batch jobs, with a named owner and a deletion policy.
• Always set honor_labels: true on the Pushgateway job, or pushed series will collide with the gateway’s own labels.
• Prefer file_sd_configs over hand-edited static_configs once the estate grows past a rack: your configuration management can add and remove targets without touching or reloading Prometheus.
• Set external_labels from day one. The first time you stand up an HA pair or a second datacentre, you will need them.

Verification

You should now be able to answer:

• What signal does the pull model give you that a push pipeline cannot, and which metric carries it?
• Why is the Pushgateway wrong for a long-running daemon, and what happens to pushed metrics over time?
• What does Prometheus report for a host that is missing from service discovery entirely?
• Which labels does up carry, and who decides them?
• A colleague says “just switch to push and the firewall problem goes away”. What do you lose?

Quiz