Remote Write Section

A 03:42 page: “Prometheus is dropping samples”. The on-call engineer reconnects to the Prometheus host, runs prometheus_remote_write_samples_dropped_total, and sees the counter has been climbing for nine hours. The local TSDB has plenty of disk; the WAL is small; the queue shows zero pending. The remote is fine, the queue is fine, the disk is fine. Yet samples are being dropped. The answer is in a write_relabel_ configs block that drops every series whose job label is not in a hard-coded list, and a new exporter was added last week that does not match. The remote write itself is doing exactly what it was told. The lesson of this section is that the remote_write block is load-bearing configuration, and every key has a production consequence.

What it is

The remote_write block is Prometheus’s write-ahead replication to a remote receiver. The block is a YAML list; each entry configures one destination. The protocol is gRPC, framed as snappy-compressed protobuf over HTTP/2 (the Prometheus prometheus.proto, not OpenTelemetry). Canonical receivers are Thanos Receiver, Mimir Distributor, Cortex Distributor, and managed services such as Grafana Cloud, Datadog, or VictoriaMetrics.

The block is the only configuration in Prometheus that turns a single-host TSDB into a contributor to a horizontally scalable metrics platform. A single Prometheus instance can answer PromQL only about what it scraped; a remote_write pipeline turns many Prometheus hosts into one logical database, with deduplication, long-term retention, and HA query layers living on the remote side.

Why a sysadmin cares

Production teams run remote_write for four reasons, and each one places a different demand on the block:

1. Durability beyond the host. A dead Prometheus host should not mean lost metrics. The remote is the off-host copy.
2. Multi-host query. A single Prometheus cannot answer “what does the SLO error budget look like across all three regions”; a remote store holds samples from many hosts, deduplicated, and answers the cross-host question.
3. Long-term retention. Weeks or months of history that the local TSDB cannot hold becomes queryable on the remote.
4. Workload isolation. Recording rules and ad-hoc queries against the local TSDB steal CPU from scrape work. A remote_write-backed architecture lets queries hit the remote and keeps the local Prometheus focused on scraping.

The reason a sysadmin must understand the block itself (not just the concept) is that every key is a knob, every default is a choice, and every choice has a failure mode. The default queue_config.capacity is 500 samples per shard — a value chosen to keep memory predictable, not to survive a multi-minute network outage. The default metadata_config.send is true, which means the daemon ships HELP and TYPE lines to the remote on a separate stream. The default retry_on_http_429 is false, which means the receiver’s first “slow down” signal turns into a sample drop. The defaults are sensible for a freshly-installed test instance; they are usually wrong for production.

How it works

The data path was introduced in the previous lesson. The block adds the per-destination configuration and the destinations themselves:

Scrape -> WAL append -> Shard by tenant + series
                              |
                              v
                    per-shard in-memory queue
                              |
                              v
                    HTTP/2 stream per shard
                              |
                              v
                    Remote receiver #1   (remote_write[0])
                              |
                              v
                    Remote receiver #2   (remote_write[1])
                              |
                              v
                    Metadata stream      (remote_write[0].metadata_config)

Each entry in remote_write is a separate logical pipeline. A sample written to two destinations is appended to two WAL segments, sent on two HTTP/2 streams, and acknowledged by two receivers. The two pipelines do not share queues, do not share retries, and do not share drops. A drop on destination #1 says nothing about destination #2.

How to configure it

A production-tuned remote_write block, with every key annotated:

remote_write:
  # ---- Per-destination identity and transport --------------------
  - name: mimir-primary             # optional; surfaced in metrics
    url: https://mimir-distributor.metrics.svc:8080/api/v1/push
    remote_timeout: 30s             # default 30s; per-request timeout
    follow_redirects: true          # default; receiver may 301
    enable_http2: true              # default; protocol requires it
    proxy_url: http://proxy.ops:3128 # optional; rarely used

    # ---- Headers (custom HTTP headers) ---------------------------
    # Headers are sent on every request. Useful for tenant routing
    # on Mimir and Cortex, and for receiver-side authorisation.
    headers:
      X-Scope-OrgID: tenant-prod
      X-Prometheus-Region: eu-west-1

    # ---- Authentication (one of these blocks) --------------------
    basic_auth:
      username: ${REMOTE_WRITE_USER}
      password: ${REMOTE_WRITE_PASSWORD}
    tls_config:
      ca_file: /etc/prometheus/ca.pem
      cert_file: /etc/prometheus/client.pem
      key_file: /etc/prometheus/client.key
      server_name: mimir-distributor.metrics.svc

    # ---- Write-time relabeling -----------------------------------
    # Applied AFTER the scrapes have produced samples, BEFORE the
    # samples are queued. The same syntax as scrape relabeling.
    write_relabel_configs:
      # Drop a high-cardinality series we never query on the remote.
      - source_labels: [__name__]
        regex: 'go_gc_duration_seconds_bucket'
        action: drop
      # Add a label so the remote knows which Prometheus sent it.
      - target_label: source_cluster
        replacement: prod-eu-west-1
      # Drop exemplars on series that the remote does not need.
      - source_labels: [__name__]
        regex: 'http_request_duration_seconds_bucket'
        action: drop

    # ---- Queue, retries, and sharding -----------------------------
    queue_config:
      capacity: 10 000                 # samples per shard; in-memory
      min_shards: 2                    # baseline parallelism
      max_shards: 50                   # ceiling; auto-scales with load
      max_samples_per_send: 2 000      # batch ceiling
      batch_send_deadline: 5s          # max wait before flushing
      min_backoff: 30ms                # first retry delay
      max_backoff: 30s                 # last retry delay
      max_retries: 10                  # then drop the batch
      retry_on_http_429: true          # honour receiver back-pressure
      sample_age_limit: 0s             # 0 = no age limit; drop after N

    # ---- Metadata (HELP and TYPE) --------------------------------
    # Sent on a separate HTTP/2 stream. Most receivers want this.
    metadata_config:
      send: true
      send_interval: 1m               # re-send every minute
      max_samples_per_send: 2 000

    # ---- HTTP client tuning --------------------------------------
    http_config:
      keepalive: { time: 30s, timeout: 5s }

  # ---- Second destination: same block, different receiver ----------
  - name: mimir-secondary
    url: https://mimir-distributor-eu-west-2.metrics.svc:8080/api/v1/push
    remote_timeout: 30s
    headers:
      X-Scope-OrgID: tenant-prod
    basic_auth:
      username: ${REMOTE_WRITE_USER_EU_W2}
      password: ${REMOTE_WRITE_PASSWORD_EU_W2}
    queue_config:
      capacity: 10 000
      min_shards: 2
      max_shards: 50
      max_samples_per_send: 2 000
      batch_send_deadline: 5s
      max_retries: 10
      retry_on_http_429: true
      min_backoff: 30ms
      max_backoff: 30s
    metadata_config:
      send: true
      send_interval: 1m

A few of the keys deserve a sentence each:

• remote_timeout. Per-request timeout. Default 30s. Set this lower than the receiver’s own timeout; a Prometheus sender that holds an HTTP/2 stream open for 90s while the receiver has already timed out is wasting a shard.
• headers. Custom HTTP headers. The X-Scope-OrgID header is the tenant boundary on Mimir and Cortex; getting it wrong writes to the wrong tenant. The User-Agent is always Prometheus/X.Y.Z and cannot be overridden via headers.
• write_relabel_configs. Same syntax as metric_relabel_ configs in scrape jobs. The source_labels are the sample’s labels after the scrape’s labels have been applied. A write_relabel_configs entry with action: drop is the canonical way to strip high-cardinality series before they reach the remote.
• queue_config.capacity. Samples per shard. The total in-flight samples is capacity * max_shards. A 10 000-sample ceiling with 50 shards gives 500 000 in-flight samples, which is roughly half a million ~32-byte slots, or ~16 MB of queue memory per destination.
• queue_config.retry_on_http_429. Default false. The reason is that “true” requires the receiver to honour the protocol’s Retry-After semantics; many receivers do not. On Mimir and Cortex, set it to true. On a custom receiver, verify the receiver honours Retry-After before enabling.
• queue_config.sample_age_limit. Default 0 (no limit). Set to a positive value (5m, 15m) to drop samples that have been queued longer than the limit. Use this only when the remote is intentionally lossy and staleness is preferable to recency.
• metadata_config.send. Default true. Some receivers (notably older Cortex) ignore metadata; sending it then is harmless. On a receiver that rejects unknown metadata, set this to false.

How to validate it

Four commands, in order. Each proves a different property.

# 1. The block is syntactically valid against the daemon's schema.
promtool check config /etc/prometheus/prometheus.yml

# 2. The block is loaded with the expected destinations.
curl -s localhost:9090/api/v1/status/config \
  | jq '.data.yaml' | grep -B1 -A4 'remote_write'

# 3. Each destination is shipping samples.
for remote in mimir-primary mimir-secondary; do
  echo "=== $remote ==="
  curl -s localhost:9090/metrics \
    | grep "remote_name=\"$remote\"" \
    | grep -E 'remote_write_(samples_sent_total|samples_dropped_total|samples_pending|write_errors_total)'
done

# 4. The receivers are accepting. Check the matching ingest metric
#    on the receiver side (Mimir example).
curl -s mimir-distributor.metrics.svc:8080/metrics \
  | grep -E 'cortex_distributor_samples_in_total|cortex_distributor_samples_rejected_total'

Realistic output for a healthy pipeline:

=== mimir-primary ===
prometheus_remote_write_samples_sent_total{remote_name="mimir-primary"} 1.78e+07
prometheus_remote_write_samples_dropped_total{remote_name="mimir-primary"} 0
prometheus_remote_write_samples_pending{remote_name="mimir-primary"} 412
prometheus_remote_write_write_errors_total{remote_name="mimir-primary"} 0
=== mimir-secondary ===
prometheus_remote_write_samples_sent_total{remote_name="mimir-secondary"} 1.78e+07
prometheus_remote_write_samples_dropped_total{remote_name="mimir-secondary"} 0
prometheus_remote_write_samples_pending{remote_name="mimir-secondary"} 287
prometheus_remote_write_write_errors_total{remote_name="mimir-secondary"} 0

The remote_name label is added by the daemon from the optional name: key. Destinations without a name are labelled by the URL host. Always set name on multi- destination blocks so the metrics are distinguishable.

Three PromQL queries for the same question:

# Sent samples per second per destination, last 5 minutes
sum by (remote_name) (
  rate(prometheus_remote_write_samples_sent_total[5m])
)

# Pending samples per destination; sustained high values mean
# the receiver is the bottleneck.
sum by (remote_name) (
  prometheus_remote_write_samples_pending
)

# Dropped samples in the last hour, per destination.
sum by (remote_name) (
  increase(prometheus_remote_write_samples_dropped_total[1h])
)

How it can fail

The six failure modes that arrive most often in production:

1. write_relabel_configs drops a series the team expects to see. Symptom: the remote is healthy, the queue is shipping, but a specific series is missing from the remote store. The prometheus_remote_write_samples_dropped_total counter does not increment for relabel drops — only for queue-drops after max_retries. The diagnostic is a count query on the local TSDB versus the remote; the difference is the relabel drop. Fix the regex, or move the drop to a metric_relabel_configs block on the scrape job so the drop is visible in the local TSDB’s metrics.

2. X-Scope-OrgID is wrong or missing on Mimir/Cortex. Symptom: HTTP 401 from the receiver, or a 200 with samples silently written to the default tenant. The latter is the dangerous one — the silent writing of metrics to a tenant no one is watching. The fix is to fail fast: add a headers: block with the tenant ID and validate the result by checking the receiver’s cortex_distributor_samples_ in_total\{tenant=...\} metric.

3. Queue sized for the happy path. Symptom: a 30-second receiver restart fills the queue, samples spill to the WAL, and on recovery the WAL replay competes with scrapes for CPU. The producer is the receiver, not the sender. The fix is to size capacity and max_shards for a multi-minute outage, not a 30-second one.

4. retry_on_http_429: false on a back-pressured receiver. Symptom: the receiver sends 429s during a spike, the sender drops the batch on first 429, and the remote store has a visible gap. The sender dropped because the receiver asked it to slow down. The fix is to set retry_on_http_429: true and verify the receiver honours Retry-After.

5. Metadata stream is rejected by the receiver. Symptom: metrics are accepted but HELP text in Grafana is empty; the receiver logs metadata: unknown metric type for every sample. The metadata stream is on a separate HTTP/2 channel from the samples stream, so a metadata error does not stop sample ingestion. Set metadata_config.send: false if the receiver is known to reject metadata, or fix the receiver.

6. Two destinations, identical URLs, no external_labels distinction. Symptom: one destination silently wins on query-time dedup; the other is paying for storage it does not contribute to. The fix is to ensure each destination has a unique identifying label (a name: key, an external_labels set, or a header) so dedup can distinguish the writers.

How to troubleshoot it

Diagnostic steps, in order:

1. Is the block loading? curl -s localhost:9090/api/v1/ status/config | jq '.data.yaml' | grep -B1 -A4 'remote_write'. The daemon reports the loaded configuration. If the block is absent, the reload did not take.
2. Is each destination shipping? prometheus_remote_write_ samples_sent_total\{remote_name=...\} rising steadily. Stable is the failure shape.
3. Is each destination queue healthy? prometheus_remote_ write_samples_pending near zero. A sustained value above ~30% of capacity * max_shards is a warning; near the limit is a drop is imminent.
4. Are samples dropping? A non-zero increase( prometheus_remote_write_samples_dropped_total[15m]) is a page. The first question is why: check the receiver HTTP error code, then the network, then the queue sizing.
5. What does the receiver say? On Mimir or Cortex, the Distributor’s cortex_distributor_samples_in_total and cortex_distributor_samples_rejected_total are the matching pair. A mismatch between Prometheus’s “sent” and the Distributor’s “received” is a network or auth issue; a “received vs accepted” mismatch is a tenant or schema issue.
6. What does the daemon log? Grep for remote write, dropped samples, 5xx, auth. The sender logs the failure mode explicitly when one is available.

Security implications

• Credentials. The remote_write URL is a write-only endpoint; a leaked credential is a credential to write arbitrary metrics into the platform. Treat the password file or secret as a database root password: 0600, rotated, sourced from a secrets manager. SigV4 with instance roles or workload identity is preferable to long-lived bearer tokens.
• mTLS. On-prem Thanos Receiver, Mimir, and Cortex all support mTLS. The trade-off is operational: rotation cadence, CA trust, and cert expiry alerting. Without mTLS, the credential is the only line of defence against a network attacker.
• Tenant scoping. On Mimir and Cortex, the X-Scope-OrgID header (or an equivalent auth claim) is the tenant boundary. A misconfigured header that emits the wrong tenant ID writes another tenant’s data — silent, hard to detect, expensive to reconcile. The Prometheus that is supposed to be tenant A must always carry tenant A’s ID, and only tenant A’s ID.
• Network path. The remote_write endpoint should not be internet-reachable. The TLS layer protects the bytes; the network layer protects the attack surface.

Performance implications

remote_write is the most CPU-intensive feature of a production Prometheus server. A rough budget:

1 M samples/sec shipped   ~ 4-8 cores on the sender
                          ~ 4-8 GB RAM for queues
                          ~ 100-300 Mbit/s outbound
                          ~ 50-100 GB/day disk for WAL spill buffer

The dominant variables are the shard count (more shards, more parallel HTTP/2 streams, more memory) and the batch size (larger batches, lower per-sample CPU, higher per-batch latency). The trade-off between CPU and memory is capacity * max_shards: each pending sample costs ~32 bytes of queue memory.

Multi-destination blocks multiply the cost. Two destinations with the same shard configuration use roughly twice the CPU, twice the RAM, and twice the outbound bandwidth. The operational rule is: configure destinations deliberately. A mimir-primary and mimir-secondary is fine; a mimir-primary, mimir-secondary, datadog-backup, and debug-tap is rarely worth the cost.

Verification

You should now be able to answer:

• What does each key in the remote_write block do, and what is its production default?
• What does queue_config.capacity actually bound, and what happens when it is exceeded?
• Why is retry_on_http_429: true the right setting on Mimir and Cortex?
• How does the metadata stream differ from the samples stream, and which metrics expose each?
• What is the cost of a multi-destination remote_write block?

Quiz