Remote Write

A 03:00 page: “Prometheus is down”. Restarting it brings the server back; 40 minutes of metrics are missing for the busiest targets. The remote_write pipeline had been silently failing for three weeks, but the local retention was 30 days, so the gap was invisible until the disk filled and the head crashed. A separate remote_write configuration had been intended as a safety net; nobody had checked whether it was actually shipping.

This is what remote_write is for: replicating samples to a destination that outlives the local Prometheus. The implementation detail is the WAL -> shards -> queue -> HTTP/gRPC pipeline. The operational detail is that configured is not the same as shipping.

What it is

remote_write is Prometheus’s write-ahead replication to a remote receiver. The protocol is gRPC-based (Prometheus’s own prometheus.proto, not OpenTelemetry), and the canonical receivers are Thanos Receiver, Cortex / Mimir Distributors, and vendor services (Grafana Cloud, Datadog, VictoriaMetrics). It is the mechanism by which a single-host Prometheus contributes to a multi-host, multi-replica, horizontally scalable metrics platform.

The protocol is one-way: Prometheus writes to the remote; the remote never writes back to Prometheus. The local TSDB remains the authoritative store for the host’s own retention window. Queries against the remote store are independent of Prometheus’s own PromQL endpoints; they are issued against Thanos Querier, Mimir Querier, or Cortex / Mimir Query-frontend, usually via Grafana.

Why a sysadmin cares

remote_write is the difference between a metrics platform that survives the loss of a host and one that does not. The local TSDB on a Prometheus host has a single-writer invariant; if that host dies, the data dies with it. A correctly configured remote_write turns each Prometheus host into a contributor to a durable platform, not the sole owner of its own data.

The reasons a sysadmin runs remote_write:

1. Durability beyond the host. A dead Prometheus host should not mean lost metrics. The remote is the off-host copy.
2. Multi-host query. A single Prometheus can only answer questions about what it scraped. A remote store holds samples from many Prometheus hosts, deduplicated, so a query can ask questions no single host can answer.
3. Long-term retention. This is the use case the previous lesson framed; remote_write is the mechanism.
4. Workload isolation. Recording rules and ad-hoc queries against the local TSDB steal CPU from scrape work. A remote_write-backed architecture lets queries hit the remote and keeps the local Prometheus focused on scraping.

How it works

The remote_write data path:

Scrape        WAL append        Shard by tenant + series
  |               |                        |
  v               v                        v
TSDB head  ->  queue manager  ->  per-shard in-memory queue
                                          |
                                          v
                                    HTTP/2 stream
                                    (snappy-framed
                                     protobuf batches)
                                          |
                                          v
                                  Remote receiver
                                  (Thanos Receiver /
                                   Mimir Distributor /
                                   Cortex Distributor)

Every scrape append lands in the WAL. A shard manager hashes the series labels into one of N shards, where N is min_shards to max_shards configured on the queue. Each shard holds an in-memory queue of pending batches. A sender per shard packs the queue into 1-1000-sample batches (controlled by batch_send_deadline and max_samples_per_send) and streams them over a single long-lived HTTP/2 connection.

The remote receiver acknowledges successful ingestion; the sender removes the acknowledged range from the queue. On a non-2xx response, the sender retries with exponential back-off, until the batch either succeeds or max_retries is reached, at which point the batch is either dropped (default), or — if retry_on_http_429 is set — held until the receiver says it can keep up.

How to configure it

A production-tuned remote_write block:

remote_write:
  - url: https://mimir-distributor.metrics.svc:8080/api/v1/push
    # Order of authentication:
    #  1. basic_auth     (HTTP basic)
    #  2. authorization  (raw bearer token)
    #  3. sigv4          (AWS SigV4 for managed Prometheus / Mimir)
    #  4. oauth2         (client credentials grant)
    #  5. tls_config     (mTLS for Thanos Receiver on-prem)
    basic_auth:
      username: "${REMOTE_WRITE_USER}"
      password: "${REMOTE_WRITE_PASSWORD}"
    tls_config:
      ca_file: /etc/prometheus/ca.pem
      cert_file: /etc/prometheus/client.pem
      key_file: /etc/prometheus/client.key
      server_name: mimir-distributor.metrics.svc

    write_relabel_configs:
      # Drop exemplars on high-cardinality series that the remote
      # does not need. Drop _other_ noise that should never reach
      # long-term storage.
      - source_labels: [__name__]
        regex: 'go_gc_duration_seconds_bucket'
        action: drop

    # Queue: most incidents come from these knobs being wrong.
    queue_config:
      capacity: 10 000                    # samples per shard, in queue
      min_shards: 2                       # baseline parallelism
      max_shards: 50                      # ceiling; auto-scales with load
      max_samples_per_send: 2 000         # batch ceiling
      batch_send_deadline: 5s             # max wait before flushing
      max_retries: 10                     # exponential, then drop
      retry_on_http_429: true             # honour receiver back-pressure
      min_backoff: 30ms
      max_backoff: 30s

    # Metadata: keep default unless the remote is a known
    # non-metadata sink. Sent on a separate stream.
    metadata_config:
      send: true
      send_interval: 1m

    # HTTP/2 keepalive; helps detect dead receivers faster.
    http_config:
      keepalive: { time: 30s, timeout: 5s }

The single most-misconfigured knob is queue_config.capacity. Too small and the queue fills on every blip; samples spill to the WAL and recovery takes minutes. Too large and a remote outage holds gigabytes of samples in memory.

How to validate it

Three commands, in this order.

# 1. The pipeline is configured.
curl -s localhost:9090/api/v1/status/config \
  | jq '.data.yaml' | grep -A2 'remote_write'

# 2. The pipeline is shipping. Rate of sent samples must be
#    non-zero and match the scrape rate.
curl -s localhost:9090/metrics \
  | grep -E 'prometheus_remote_write_(samples_pending|samples_sent_total|bytes_transmitted_total)'

# 3. The remote is accepting. Drop and error counters must be
#    zero (or very near it).
curl -s localhost:9090/metrics \
  | grep -E 'prometheus_remote_write_(samples_dropped_total|samples_failed_total|write_errors_total)'

Realistic output for a healthy pipeline:

prometheus_remote_write_samples_pending              412
prometheus_remote_write_samples_sent_total           1.78e+07
prometheus_remote_write_bytes_transmitted_total      4.21e+10
prometheus_remote_write_samples_dropped_total        0
prometheus_remote_write_write_errors_total          {remote_name="mimir"} 0

Three PromQL queries for the same answer:

# Sent samples per second, last 5 minutes
rate(prometheus_remote_write_samples_sent_total[5m])

# Queue depth by remote; sustained non-zero means the receiver is slow
prometheus_remote_write_samples_pending

# Dropped samples in the last hour
increase(prometheus_remote_write_samples_dropped_total[1h])

How it can fail

1. Wrong URL or DNS. Symptom: write_errors_total rises steadily, samples_pending grows, samples_dropped_total starts climbing after max_retries. Logs show connection refused or DNS resolution failures. The pipeline is configured but not delivering.
2. Authentication drift. Symptom: HTTP 401 / 403 from the receiver, samples_dropped_total climbs, no DNS or TCP errors. A rotated credential, an IAM policy change, or a tenant ID change is the usual cause.
3. Receiver back-pressure. Symptom: HTTP 429s, queue depth grows, latency on the receiver side. retry_on_http_429 holds the samples; without it, the samples drop on first 429. The receiver is the bottleneck, not the sender.
4. WAL explosion on a long remote outage. Symptom: local disk fills during the outage; on recovery, replay takes hours and competing for CPU with scrapes. The remote is back but the local Prometheus is degraded until WAL drains.
5. Schema version mismatch on the receiver. Symptom: the receiver rejects entire batches with “unknown metric type” or “unknown label name”. Caused by a Prometheus 2.55 sender writing native-histogram samples to a receiver that does not understand them. Check the receiver changelog before enabling native histograms upstream.
6. HA replica labelling mistake. Symptom: two Prometheus writing the same remote_write target with the same external_labels (or none). The remote store receives duplicate samples for every series; query-time deduplication hides the duplication; capacity bills double. The external_labels: { replica: A } / { replica: B } split is mandatory for HA pairs writing the same target set.

How to troubleshoot it

1. Is the URL right? curl -v $REMOTE_URL/api/v1/push from the Prometheus host. A 401 is configuration; a connection refused is network; a 415 is schema.
2. Is the queue healthy? prometheus_remote_write_samples_ pending near zero means the sender is keeping up. A sustained value above ~30% of capacity * max_shards is a warning sign; near the limit is a drop is imminent.
3. Are samples dropping? increase(prometheus_remote_write_ samples_dropped_total[15m]) > 0 is a page. The first question is why: check the receiver HTTP error code, then the network, then the queue sizing.
4. What does the receiver say? On Mimir or Cortex, the Distributor’s cortex_distributor_samples_in_total and cortex_distributor_samples_rejected_total are the matching pair. A mismatch between Prometheus’s “sent” and the Distributor’s “received” is a network or auth issue; a “received vs accepted” mismatch is a tenant or schema issue.
5. Logs. Grep for “remote write”, “dropped samples”, “5xx”, “auth”. The sender logs the failure mode explicitly when one is available.

Security implications

• Credentials. The remote_write URL is a write-only endpoint; a leaked credential is a credential to write arbitrary metrics into the platform. Treat the password file or secret as you would treat a database root password: 0600, rotated, sourced from a secrets manager. SigV4 with instance roles or workload identity is preferable to long-lived bearer tokens.
• mTLS. On-prem Thanos Receiver, Mimir, and Cortex all support mTLS. The trade-off is operational: rotation cadence, CA trust, and cert expiry alerting. Without mTLS, the credential is the only line of defence against a network attacker.
• Tenant scoping. On Mimir and Cortex, the X-Scope-OrgID header (or equivalent auth claim) is the tenant boundary. A misconfigured header that emits the wrong tenant ID writes another tenant’s data — silent, hard to detect, expensive to reconcile. The Prometheus that is supposed to be tenant A must always carry tenant A’s ID, and only tenant A’s ID.
• Network path. The remote_write endpoint should not be internet-reachable. A common pattern is to bind the receiver to a private VPC or to require a private link / peering connection. The TLS layer protects the bytes; the network layer protects the attack surface.

Performance implications

remote_write is the most CPU-intensive feature of a production Prometheus server. A rough budget:

1 M samples/sec shipped   ~ 4-8 cores on the sender
                          ~ 4-8 GB RAM for queues
                          ~ 100-300 Mbit/s outbound
                          ~ 50-100 GB/day disk for WAL spill buffer

The dominant variables are the shard count (more shards, more parallel HTTP/2 streams, more memory) and the batch size (larger batches, lower per-sample CPU, higher per-batch latency). The trade-off between CPU and memory is capacity * max_shards: each pending sample costs ~32 bytes of queue memory.

Scrape interval matters more than cardinality for remote_write cost. Halving the scrape interval doubles the sample rate and roughly doubles the remote_write CPU. A team that has over-instrumented can solve the problem by dropping high- cardinality metrics at the source with metric_relabel_configs, not by scaling the sender.

Verification

You should now be able to answer:

• What does the remote_write data path look like from scrape to receiver?
• What does queue_config.capacity actually bound, and what happens when it is exceeded?
• Which metric tells you whether samples are being accepted by the remote, and which tells you whether they were dropped locally?
• What is the right discipline when the receiver returns HTTP 429?
• Why is the X-Scope-OrgID header not optional on Mimir or Cortex?

Quiz