Remote Write and Remote Receive

A single Prometheus with fifteen days of local disk is a complete monitoring system — right up until compliance asks for thirteen months, or three teams need one query across every cluster. Remote write is how a 2026-era Prometheus answers that: every sample is also streamed to a remote storage system whose whole job is long-term, global, multi-tenant retention. The local TSDB keeps doing what it does; the remote system does what the TSDB never will.

What it is

remote_write streams samples — after optional write_relabel_configs filtering — to an HTTP endpoint as snappy-compressed protobuf WriteRequest messages. The receivers you will actually meet in production: Thanos Receive, Grafana Mimir, Cortex, VictoriaMetrics (vminsert), and the hosted endpoints (Grafana Cloud and friends).

Two things it is not:

• Not replication. Your local TSDB is untouched; remote write is a one-way stream. There is no synchronisation back.
• Not query. Reading remote data back is remote_read, a separate, optional configuration that most people skip in favour of querying the receiver’s own query layer (Mimir, Thanos Querier) directly from Grafana.

WAL-backed delivery — the guarantee and its edge

The remote-write queue manager tails the same write-ahead log the head block uses, and it checkpoints its position as it goes. Restart Prometheus and it resumes where it stopped — as long as the WAL segment still exists.

That qualifier is the entire delivery guarantee. The WAL is truncated when the head is compacted (lesson 03), roughly every couple of hours. A receiver outage shorter than that horizon is fully replayed; an outage longer than that leaves a permanent gap in the remote store, even though your local retention is fine. Remote write buys you at-least-once delivery over hours, not days. Design alerts and expectations accordingly.

The queue model

Samples read from the WAL fan out into shards — in-memory queues, each with its own sender goroutine. The number of shards scales dynamically between min_shards and max_shards based on the rate of samples coming in versus going out. Each shard batches up to max_samples_per_send samples and flushes when the batch is full or batch_send_deadline expires, whichever comes first. Failures retry with exponential backoff between min_backoff and max_backoff.

How to configure it

remote_write:
  - url: 'https://mimir.example.internal/api/v1/push'
    remote_timeout: 30s
    headers:
      X-Scope-OrgID: platform        # tenant id for Mimir/Cortex-style receivers
    basic_auth:
      username: prometheus
      password_file: /etc/prometheus/secrets/mimir.pass
    tls_config:
      ca_file: /etc/prometheus/tls/ca.crt
    write_relabel_configs:
      # do not pay to ship runtime trivia
      - source_labels: [__name__]
        regex: 'go_.*'
        action: drop
    queue_config:
      capacity: 10000                # samples buffered per shard (default 2500)
      max_shards: 50                 # ceiling on parallel senders (default 200)
      min_shards: 1                  # idle floor (default 1)
      max_samples_per_send: 2000     # batch size (default 500; 2000 with the v2 protocol)
      batch_send_deadline: 5s        # flush a partial batch after this
      min_backoff: 30ms              # first retry delay after a failure
      max_backoff: 5s                # worst-case retry delay
      retry_on_http_429: true        # v1 protocol treats 429 as permanent otherwise
    # protobuf_message: io.prometheus.write.v2.Request   # opt into RW 2.0 in 2.55

Tuning from first principles. Sustained throughput is roughly shards × max_samples_per_send ÷ actual round-trip time. If shards_desired pins at max_shards and lag grows: raise max_samples_per_send first (fewer, bigger requests), then max_shards, then capacity. Do not just max everything: queue memory is capacity × max_shards samples held in RAM, and your receiver pays for every oversized burst. write_relabel_configs is the cheapest lever of all — every dropped series is egress and receiver spend saved forever.

How to validate it

# Config sanity
promtool check config /etc/prometheus/prometheus.yml

# The two numbers that matter most: how far behind are we?
curl -s 'localhost:9090/api/v1/query?query=prometheus_remote_storage_highest_timestamp_in_seconds'
curl -s 'localhost:9090/api/v1/query?query=prometheus_remote_storage_queue_highest_sent_timestamp_seconds'

# Reachability and auth from the Prometheus host
# (a POST-only endpoint answering 405 still proves TLS, DNS and credentials path)
curl -sv https://mimir.example.internal/api/v1/push -o /dev/null

The delivery-health metrics, all under prometheus_remote_storage_:

# lag: the single best "are we keeping up" signal — alert on this
prometheus_remote_storage_highest_timestamp_in_seconds
  - on() prometheus_remote_storage_queue_highest_sent_timestamp_seconds

prometheus_remote_storage_samples_total            # accepted
prometheus_remote_storage_samples_retried_total    # transient failures
prometheus_remote_storage_samples_failed_total     # permanent rejections (4xx)
prometheus_remote_storage_samples_dropped_total    # dropped before send (queue full)
prometheus_remote_storage_shards                   # current shard count
prometheus_remote_storage_shards_desired           # what the autoscaler wants
prometheus_remote_storage_bytes_total              # your egress bill

Then prove the round trip: query the receiver (Mimir, Thanos Querier) for a recent metric and confirm your external_labels arrived attached.

Outage semantics, precisely

Receiver down → sends fail → exponential backoff between min_backoff and max_backoff → shards fill → new samples beyond capacity × max_shards are dropped (samples_dropped_total) → on recovery, everything still in the WAL is resent → anything WAL truncation already reclaimed is gone for good.

Failure classes the metrics distinguish:

• 429 (rate limited): with the v1 protocol this is a permanent failure unless retry_on_http_429: true; the v2 protocol retries it by default. A 429 storm during receiver autoscaling with v1 defaults silently loses data.
• 400 (bad request): never retried. samples_failed_total grows until you fix the offending series or relabel it away. Classic cause: the sender’s clock skew puts samples outside the receiver’s acceptance window.
• Everything else (5xx, network): retried within the WAL horizon.

How it can fail

1. Receiver outage beyond the WAL horizon. Symptom: lag grows, then dropped_total grows; the long-term store has a permanent hole while local Grafana looks fine. Invisible unless you alert on the lag metric.
2. 429s with v1 defaults. Symptom: failed_total spikes that correlate exactly with receiver scaling events.
3. 400 rejections from bad samples. Symptom: failed_total grows steadily, and the Prometheus logs name the offending labels.
4. Undersized queue. Symptom: shards_desired pinned at max_shards, lag growing, CPU churning in small batches. Remote write “works” but is perpetually behind.
5. Credential or TLS expiry. Symptom: identical to an outage in the metrics, but curl -v shows 401s or handshake failures.
6. Queue memory during a long outage. capacity × max_shards samples held in RAM while the receiver is down; sized generously, the buffers themselves OOM the server mid-incident.

How to troubleshoot it

1. Classify first: keeping up, behind, or dropping? The lag expression plus failed vs retried vs dropped answers this in one dashboard row.
2. Read the logs. “remote storage” send errors include HTTP status codes and label hints.
3. Prove the path. Curl the URL from the Prometheus host; check certificate dates; check the receiver’s own ingestion metrics.
4. Check the WAL. A bloated wal/ directory during an outage tells you how much resend is pending.
5. Fix in order: restore the receiver, then let the queue drain. Do not restart Prometheus mid-drain unless forced — the resend survives restarts, but cold caches make it slower.

Security implications

The remote_write credentials can write metrics into your long-term store — metric injection poisons dashboards, SLOs and capacity models, and it does not show up anywhere local. Protect password_file (0640), mandate TLS across any WAN boundary, and scope tenant credentials (X-Scope-OrgID or equivalent) per team so one leaked password cannot write as everyone. A receiver deployed without authentication is an open write path into your history.

Performance implications

Egress bandwidth ≈ your ingestion rate, snappy-compressed. Marshalling CPU scales with samples per second. Queue memory is capacity × max_shards × sample size — at the tuned example above (10000 × 50) that is half a million samples held in RAM per remote endpoint. The receiver side pays ingestion cost for everything you send, which is why write_relabel_configs belongs in the cost conversation, not just the hygiene one.

Verification

You should now be able to answer:

• Where does remote-write durability come from, and exactly where does it end?
• What do capacity, max_shards, max_samples_per_send and batch_send_deadline each control?
• Which metric pair shows remote-write lag, and why is it the one to alert on?
• What is the delivery difference between an HTTP 400 and an HTTP 429 under the v1 protocol?
• Why does a six-hour receiver outage leave a permanent remote gap even with thirty days of local retention?

Quiz