Mimir Overview

50 Prometheus servers, each remote_write-ing to a Thanos bucket. The bucket is healthy. The Thanos Querier, however, has become a chokepoint: every dashboard query goes through one process, and the 4 pm reporting window regularly exceeds 30 seconds. Adding Querier replicas does not help, because each replica independently scans the entire bucket for every query.

Grafana Mimir is the answer that the Thanos architecture does not provide: a purpose-built, horizontally scalable, multi-tenant TSDB that splits the work by tenant, by time, and by series, with caches and query parallelism designed into the data path. The trade-off is operational complexity; the win is scale.

What it is

Grafana Mimir is an open-source, horizontally scalable, multi- tenant time-series database built by Grafana Labs and forked from Cortex in 2022. It accepts Prometheus’s remote_write protocol, stores samples in per-tenant blocks in an object store, and answers Prometheus-compatible query API requests at scale.

Mimir is not a sidecar architecture. The vanilla Prometheus servers keep scraping and shipping; the Mimir cluster takes ownership of ingest, query, retention, and rule evaluation. A single Mimir deployment can hold billions of active series across thousands of tenants.

The project ships as a single binary that can run in any combination of microservices modes (one process per component) or as a monolithic mode (one binary, all components). Production deployments of any meaningful size run microservices mode.

Why a sysadmin cares

Mimir is the right answer when Thanos has stopped scaling. The specific thresholds are:

• Active series above ~10 million. Thanos works; Mimir is designed for it.
• Tenants above ~10. The multi-tenancy machinery of Mimir (per-tenant limits, per-tenant query splitting, per-tenant retention) is the operational relief.
• Query concurrency above what a few Querier replicas can serve. Mimir’s Query-frontend splits queries, caches results, and parallelises execution across Queriers.

The trade-off is operational surface area. A Thanos deployment has Sidecar + Store + Querier + Compactor; Mimir has nine microservices plus a Consul / etcd backend for the hash-ring. That complexity is the price of horizontal scalability, and the reason Mimir requires more care to operate.

How it works

The components and their jobs:

                       Query-frontend
                  (split, cache, parallelise)
                                |
                                v
                          Queriers x N
                       (merge partials)
                                |
              +-----------------+-----------------+
              |                 |                 |
        Store gateway     Store gateway     Store gateway
        (block reader,    (block reader,    (block reader,
         per-tenant sharded) per-tenant sharded) per-tenant sharded)
              |                 |                 |
              +-----------------+-----------------+
                                |
                          Object store
                                ^
                                |
        +-----------------+-----------------+
        |               |                 |
   Ingesters x N  Ingesters x N  Ingesters x N
   (TSDB head,     (TSDB head,     (TSDB head,
    per-tenant)     per-tenant)     per-tenant)
        ^               ^                 ^
        |               |                 |
        +---------------+-----------------+
                        |
                 Distributors x N
              (hash tenant + series
               onto Ingesters, replicate)
                        ^
                        |
                Prometheus remote_write
                   (one per fleet)

The nine components, by role:

• Distributor — stateless; receives remote_write, validates the request, hashes each series to an Ingester replica, and forwards. Replicates to N Ingesters based on ingester.ring.replication_factor.
• Ingester — stateful; holds the head of each tenant’s TSDB in memory and on local disk. On a flush boundary (default 2 hours), writes the head as a block to the bucket. Ingesters are the heart of Mimir; their loss is data loss.
• Store gateway — stateless; reads blocks from the bucket on demand, indexes them in memory and in memcached, serves samples to the Querier. Per-tenant sharding keeps any one gateway from loading the entire bucket.
• Querier — stateless; merges partial results from Store gateways and Ingesters, evaluates PromQL, returns to Query- frontend.
• Query-frontend — stateless; the query optimiser. Splits long ranges into shorter ones, parallelises per-split, caches results in Redis or memcached, retries on partial failures.
• Compactor — singleton per tenant group; merges and downsamples blocks.
• Ruler — stateful; evaluates recording and alerting rules per tenant.
• Alertmanager — embedded; the alert evaluation and routing layer, replacing the external Alertmanager when Mimir owns alert state.
• Overrides-exporter — small; exports per-tenant runtime overrides for observability.

How to configure it

A minimal Mimir configuration (/etc/mimir/config.yaml):

# Common settings
common:
  storage:
    backend: s3
    s3:
      endpoint: s3.eu-west-1.amazonaws.com
      region: eu-west-1
      bucket_name: mimir-blocks-prod
      access_key_id: "${AWS_ACCESS_KEY_ID}"
      secret_access_key: "${AWS_SECRET_ACCESS_KEY}"
  ring:
    kvstore:
      store: consul
      consul:
        host: consul.mimir.svc:8500

# Distributor
distributor:
  pool:
    health_check_ingesters: true
  remote_timeout: 10s
  rate_limit:
    ingestion_rate: 25000       # samples/sec per tenant default
    ingestion_burst_size: 50000

# Ingester
ingester:
  ring:
    replication_factor: 3
    kvstore:
      store: consul
  lifecycler:
    ring_kvstore:
      store: consul
    min_ready_duration: 1m
    final_sleep: 1m
  blocks_storage_config:
    tsdb:
      dir: /var/lib/mimir/tsdb

# Querier
querier:
  query_timeout: 2m
  max_concurrent: 200

# Query-frontend
query_frontend:
  results_cache:
    backend: redis
    redis:
      endpoint: redis.mimir.svc:6379
  split_queries_by_interval: 24h
  parallelise_shardable_queries: true
  cache_results: true

# Store gateway
store_gateway:
  sharding_ring:
    replication_factor: 3
  index_cache:
    backend: memcached
    memcached:
      addresses: memcached-exporter.mimir.svc:11211
      max_item_size: 5MB

The four knobs that drive sizing:

ingester.ring.replication_factor        3 is standard; 2 reduces cost
                                        and risks data on Ingester loss
ingester.lifecycler.min_ready_duration  how long a new replica must be
                                        healthy before it accepts writes
                                        (prevents flapping during rollout)
store_gateway.sharding_ring.replication_factor
                                        how many Store gateways serve the
                                        same tenant + time range
query_frontend.results_cache.backend   memcached (low cost) vs redis
                                        (more features, slightly higher)

How to validate it

# Mimir is up and ready
curl -s mimir-frontend:8080/ready
curl -s mimir-frontend:8080/api/v1/status/config | jq '.status'

# A real query that proves the full path: tenant 42, last 24 hours
curl -sG mimir-frontend:8080/api/v1/query \
  -H 'X-Scope-OrgID: tenant-42' \
  --data-urlencode 'query=count(up{job="prometheus"})'

# Mimirtool: validate the configuration is sane
mimirtool check config /etc/mimir/config.yaml

# List blocks in the bucket from the Mimir side
curl -s mimir-frontend:8080/api/v1/status/tsdb \
  -H 'X-Scope-OrgID: tenant-42' | jq '.data'

Metrics that prove Mimir is healthy end-to-end:

# Distributor is accepting samples
rate(cortex_distributor_samples_in_total[1m])

# Ingesters are healthy (should equal replication_factor)
count(up{job="mimir-ingester"})

# Compactor is alive (exactly 1 instance per tenant group)
count(up{job="mimir-compactor"})

# Query latency
histogram_quantile(0.99,
  sum(rate(cortex_query_request_duration_seconds_bucket[5m])) by (le))

# No rejected samples due to per-tenant limits
rate(cortex_distributor_samples_rejected_total[5m])

How it can fail

1. Ingester ring unhealthy. Symptom: writes fail with “no healthy ingester” or 503s; some series are missing; the cortex_ingester_ring_members metric drops below the replication factor. The cause is usually a flapping replica that has not held min_ready_duration long enough to be marked healthy.
2. Compactor not running. Symptom: bucket fills with raw 1-minute blocks; old blocks never downsample; storage cost climbs. Same shape as the Thanos Compactor failure, same discipline: one Compactor per tenant group, alert on absence.
3. Query-frontend cache misconfiguration. Symptom: every query hits the Querier; the cache hit ratio (cortex_queryfrontend_cache_hits_total / cortex_queryfrontend_cache_requests_total) is near zero; query latency at p99 climbs. A wrong memcached endpoint, an unmarshalled value type, or a TTL too short are the usual causes.
4. Per-tenant limit overflow. Symptom: HTTP 429 from the Distributor; cortex_distributor_samples_rejected_total climbs for the affected tenant. The cause is a tenant exceeding ingestion_rate or max_global_series_per_user. The fix is either raising the limit or fixing the upstream scrape — the limit is the symptom, not the disease.
5. Hash-ring KV store unreachable. Symptom: Ingesters and Store gateways cannot register; ingest fails; queries return “no store gateways available”. Consul or etcd is the dependency; network partitions to it are the typical cause.
6. Multi-tenant ID drift. Symptom: a Prometheus is writing tenant A’s data with X-Scope-OrgID: tenant-B (a config typo). Tenant B silently receives Tenant A’s data; query results cross-contaminate; tenant billing is wrong. The fix is to make the tenant ID a deployment-time variable, not a hand-edited string.

How to troubleshoot it

1. Is Mimir up? /-/ready on every component.
2. Is the ring healthy? gossip_ring_members per replica; memberlist_debug blocks if the KV store is unreachable.
3. Are samples being accepted? cortex_distributor_samples_ in_total should match the sum of Prometheus remote_write send rates. A mismatch is the network between Prometheus and Mimir.
4. Are samples being persisted? cortex_ingester_memory_ series should match the tenant’s active series count; cortex_ingester_tsdb_head_series is the same number from the TSDB side.
5. Are queries merging? cortex_query_store_gateway_ touched_posting_groups_total should be non-zero for any query that hits historical data; if it is zero, the Store gateway is not being asked.
6. Logs. Mimir is loud on purpose; the structured log includes caller, tenant, and component. Grep by component, then by tenant.

Security implications

• Tenant authentication. Every request to the Distributor carries X-Scope-OrgID (or a JWT claim with the same meaning). Production deployments put an authenticating proxy in front of Mimir that maps an authn identity to the tenant header; the Mimir side does not trust the header alone. Cross-tenant writes are the worst-case bug — silent, hard to detect, expensive to clean.
• mTLS between components. The hash-ring traffic and the gRPC between Distributor / Ingester / Querier / Store gateway should be mTLS-encrypted. The Prometheus-facing HTTP API should sit on a private network or behind a proxy that does authentication.
• Per-tenant limits as a security control. A misbehaving tenant (or a malicious one) can DoS the cluster by exceeding ingestion_rate or by sending high-cardinality labels. Per-tenant limits are not just resource management; they are the trust boundary between tenants.
• Bucket credentials. Mimir needs s3:GetObject, s3:PutObject, s3:ListBucket, s3:DeleteObject. The same discipline as Thanos: prefer IAM roles, restrict DeleteObject to the Compactor.

Performance implications

Mimir is a heavier platform than Thanos. Production sizing is roughly:

Distributor        ~ 2-4 cores, 1-2 GB RAM per replica;
                   stateless; scale by replicas
                   ~ 50 000 samples/sec per replica at 15s scrape

Ingester           ~ 8-16 cores, 16-32 GB RAM, fast NVMe
                   Memory and head size dominate
                   A single replica can hold ~5 M active series

Querier            ~ 4-8 cores, 8 GB RAM
                   CPU-bound on expression evaluation

Query-frontend     ~ 2-4 cores, 1 GB RAM
                   Stateless; cache is the dominant cost

Store gateway      ~ 4-8 cores, 8-16 GB RAM
                   Per-tenant sharding is the dominant memory shape

The dominant cost variables are: number of active series (Ingester memory), per-tenant cardinality (Store gateway index size), and query concurrency (Querier CPU). The dominant scaling levers are: sharding by tenant in the Store gateways, replication factor in the Ingester ring, and TTL / hit ratio in the Query-frontend cache.

Verification

You should now be able to answer:

• What does each of Distributor, Ingester, Store gateway, Querier, and Query-frontend actually do?
• How is Mimir’s write path different from Thanos’s Sidecar + Store path?
• Why is the Ingester the unit of Mimir durability, and what does its loss cost?
• What does per-tenant sharding buy you that the Thanos Querier does not?
• Which Prometheus-side error is the first sign of an unhealthy Ingester ring?

Quiz