Loki Microservices Mode

A platform team runs Loki in simple-scalable mode at 800 GB per day. A distributor spike saturates the write-path nodes. Ingestion stalls. The read path is fine. The write path is fine until the distributor rejects pushes with 500s and the on-call engineer has no knob except to scale the entire read+write deployment together, doubling CPU cost on the querier side just to recover a distributor outage. They migrate to microservices the following quarter, three months later than they should have.

Microservices mode is the deployment shape that makes per-component scaling, per-component failure isolation, and per-component upgrades possible. It is also the shape that requires an SRE-shaped team and a metrics stack that can absorb twelve or more services with their own metrics endpoints, readiness checks, and HPA targets.

What it is

Loki microservices mode runs each component as its own Kubernetes Deployment (or its own process group outside Kubernetes). The binary is the same one used by simple-scalable and single-binary modes; the -target flag selects the component. The production/helm/loki chart and the production/ksonnet manifests under the Loki repository are the canonical templates.

The seven components are:

• distributor — front door for log pushes. Authenticates the request, enforces per-tenant rate limits, validates labels, hashes the stream to a set of ingesters, and forwards the stream to those ingesters. Stateless. Scales horizontally.
• ingester — receives streams from the distributor, builds chunks in memory or to the WAL, and ships closed chunks to the object store. Stateful. Lives in a hash ring with a configurable replication factor. Three replicas are the production minimum.
• querier — answers LogQL queries by asking ingesters for in-flight data and fetching historical chunks from the object store. Stateless. Scales horizontally.
• query-frontend — splits long queries by time interval, parallelises the sub-queries, and caches results in memcached or Redis. Stateless. Sits between the user (Grafana, logcli) and the queriers.
• index-gateway — answers index queries against the TSDB index in the object store. Stateless behind a small ring of its own in v3.x. Scales horizontally.
• compactor — merges and compacts TSDB index files, applies retention, and rewrites old chunks. Stateful because it owns a filesystem working directory and a singleton lock. Exactly one replica in production.
• ruler — evaluates recording and alerting rules against LogQL. Stateful because it stores its rule state in the object store. One or more replicas; rule evaluation is sharded by tenant and rule group.

The cache backends (memcached or Redis) are external services, not Loki components. The object store is external. The ring KV store (consul, etcd, or memberlist) is external. The seven components above are the parts Loki ships.

Why a sysadmin cares

The microservices topology is the production default for any Loki deployment above roughly 1 TB per day, and it is the right default for most teams below that mark if they expect to grow. The operational reasons are concrete:

1. Independent scaling. The distributor and query-frontend are CPU-bound under load. The ingester is memory-bound on chunk cache. The querier is network-bound on bucket fetches. The compactor is disk-bound on its working directory. One resource curve per component means one autoscaler per component, not one shared curve that has to satisfy all four.
2. Independent failure domains. An ingester crash that loses in-flight chunks does not affect queries. A querier crash does not affect ingestion. A compactor crash does not affect queries or ingestion. The blast radius of an incident shrinks to the component that broke.
3. Independent upgrades. A querier can roll forward to a new Loki version while ingesters stay on the old version, because the wire protocol is versioned. In simple-scalable mode the whole read path must upgrade as one unit.

The trade-off is operational headcount. Twelve Deployments means twelve /services endpoints to monitor, twelve HPA targets to tune, twelve version skew windows to plan around. Microservices is a better platform if you have the platform team to run it; it is a liability if you do not.

How it works

The flow of a write request and a read request through the seven components:

  Write path (push)                       Read path (query)

  client / Alloy /                         Grafana / logcli
  Promtail / OTel Collector                       |
        |                                         v
        v                                +-----------------+
  +---------------+  hash ring  +-----+  | query-frontend  |
  | distributor   | --------->  | ing |  | split by time,  |
  | validates,    |             | est |  | parallelise,    |
  | rate-limits,  |             | er  |  | cache results   |
  | hashes stream |             |     |  +--------+--------+
  +-------+-------+             +--+--+           |
          |                        |              v
          v                        |     +-----------------+
  +---------------+                |     | querier         |
  | ingester      |  flush  +-----+     | fetch blocks    |
  | builds chunks | ----->  | obj |     | from bucket,    |
  | WAL on disk   |         | st  |     | query ingesters |
  +---------------+         | or  |     | for in-flight   |
                            | e   |     +--------+--------+
                            +-----+              |
                                                 v
                                          +-----------------+
                                          | index-gateway   |
                                          | serves index    |
                                          | queries against |
                                          | TSDB in bucket  |
                                          +-----------------+

The compactor sits beside the bucket, not in either path. It reads index files and chunks, compacts and rewrites them, applies retention, and writes the result back. The ruler pulls rule groups from the object store, evaluates them against the querier, and writes alert state back.

The hash ring that coordinates the ingester fleet lives in the KV store (consul, etcd, or memberlist). Every ingester registers on startup and renews its heartbeat every ingester.lifecycler. heartbeat_period. A distributor that cannot reach the ring refuses to push — it cannot tell which ingesters should receive the stream.

How to configure it

The Helm chart is the reference configuration. Below is a minimal microservices manifest with each component in its own file. The common block is identical across components; each per-component block is consumed only by the deployment that needs it.

Distributor (stateless, scale horizontally)

# loki-distributor.yaml
auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9095

common:
  ring:
    kvstore:
      store: consul
      consul:
        host: consul.internal:8500
  instance_addr: loki-distributor-0.loki-distributor-headless:9095
  path_prefix: /var/lib/loki
  storage_backend: s3
  s3:
    s3: s3://eu-west-1
    bucketnames: prod-loki-chunks
    region: eu-west-1

schema_config:
  configs:
    - from: '2024-01-01'
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h

distributor:
  ring:
    kvstore:
      store: consul
      consul:
        host: consul.internal:8500
  rate_store:
    backend: redis
    redis:
      endpoint: redis.internal:6379

The distributor only needs a distributor and a common block. It is stateless. Replicas = N behind a load balancer.

Ingester (stateful, replication factor 3)

# loki-ingester.yaml
auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9095

common:
  ring:
    kvstore:
      store: consul
      consul:
        host: consul.internal:8500
  instance_addr: loki-ingester-0.loki-ingester-headless:9095
  path_prefix: /var/lib/loki
  storage_backend: s3
  s3:
    s3: s3://eu-west-1
    bucketnames: prod-loki-chunks
    region: eu-west-1

schema_config:
  configs:
    - from: '2024-01-01'
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h

ingester:
  lifecycler:
    ring:
      kvstore:
        store: consul
        consul:
          host: consul.internal:8500
      replication_factor: 3
  chunk_idle_period: 1h
  chunk_target_size: 1572864   # 1.5 MB
  wal:
    enabled: true
    dir: /var/lib/loki/wal
    truncate_frequency: 2h

The ingester needs wal.enabled: true in any deployment where losing in-flight chunks is unacceptable. Without WAL a process crash loses everything in memory since the last flush.

Querier (stateless, scale horizontally)

# loki-querier.yaml
auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9095

common:
  ring:
    kvstore:
      store: consul
      consul:
        host: consul.internal:8500
  instance_addr: loki-querier-0.loki-querier-headless:9095
  path_prefix: /var/lib/loki
  storage_backend: s3
  s3:
    s3: s3://eu-west-1
    bucketnames: prod-loki-chunks
    region: eu-west-1

schema_config:
  configs:
    - from: '2024-01-01'
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h

querier:
  frontend_address: loki-query-frontend:9095
  max_concurrent: 20

The querier only needs a querier and a common block. The frontend_address points at the query-frontend; without it the querier answers queries directly.

Query-frontend (stateless, with results cache)

# loki-query-frontend.yaml
auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9095

common:
  ring:
    kvstore:
      store: consul
      consul:
        host: consul.internal:8500
  instance_addr: loki-query-frontend-0.loki-qf-headless:9095
  path_prefix: /var/lib/loki
  storage_backend: s3
  s3:
    s3: s3://eu-west-1
    bucketnames: prod-loki-chunks
    region: eu-west-1

query_range:
  split_queries_by_interval: 24h
  parallelise_shardable_queries: true
  results_cache:
    cache:
      memcached:
        endpoint: memcached.internal:11211
        max_item_size: 5MB
        ttl: 24h

frontend:
  max_outstanding_per_tenant: 2048
  compress_responses: true

The query-frontend is the only place split_queries_by_interval, the results cache, and the per-tenant concurrency limit are configured. These belong to the query-frontend, not the querier.

Index-gateway (stateless behind its own small ring)

# loki-index-gateway.yaml
auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9095

common:
  ring:
    kvstore:
      store: consul
      consul:
        host: consul.internal:8500
  instance_addr: loki-index-gateway-0.loki-ig-headless:9095
  path_prefix: /var/lib/loki
  storage_backend: s3
  s3:
    s3: s3://eu-west-1
    bucketnames: prod-loki-chunks
    region: eu-west-1

index_gateway:
  mode: ring
  ring:
    kvstore:
      store: consul
      consul:
        host: consul.internal:8500
    replication_factor: 3

The index-gateway runs in ring mode in v3.x by default so that index queries can survive a single index-gateway pod loss. Two or three replicas is the production baseline.

Compactor (singleton, exactly one)

# loki-compactor.yaml
auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9095

common:
  ring:
    kvstore:
      store: consul
      consul:
        host: consul.internal:8500
  instance_addr: loki-compactor-0.loki-compactor-headless:9095
  path_prefix: /var/lib/loki
  storage_backend: s3
  s3:
    s3: s3://eu-west-1
    bucketnames: prod-loki-chunks
    region: eu-west-1

compactor:
  working_directory: /var/lib/loki/compactor
  compaction_interval: 30m
  retention_enabled: true
  retention_delete_delay: 1h
  delete_request_store: s3

The compactor is a singleton because it owns the compaction lock. Two compactors racing for the lock is an outage. The Helm chart sets replicas: 1 explicitly; do not raise it.

Ruler (one or more, with sharded evaluation)

# loki-ruler.yaml
auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9095

common:
  ring:
    kvstore:
      store: consul
      consul:
        host: consul.internal:8500
  instance_addr: loki-ruler-0.loki-ruler-headless:9095
  path_prefix: /var/lib/loki
  storage_backend: s3
  s3:
    s3: s3://eu-west-1
    bucketnames: prod-loki-chunks
    region: eu-west-1

ruler:
  storage:
    type: s3
    s3:
      bucketnames: prod-loki-ruler
  rule_path: /var/lib/loki/rules
  ring:
    kvstore:
      store: consul
      consul:
        host: consul.internal:8500
  flush_period: 1m

The ruler stores its rule groups and alert state in a separate bucket so that a compactor run does not touch the ruler files and so that rule eval state survives a compactor crash.

The right scale order

The production sequence for bringing up a microservices Loki:

1. Object store and ring KV store first. Without these two nothing else starts.
2. Compactor. Exactly one. It needs the bucket but does not need any other component.
3. Index-gateway. Two or three replicas. It needs the bucket.
4. Ingester. Three replicas minimum, replication factor 3. It needs the bucket and the ring.
5. Distributor. Two or more replicas behind the load balancer. It needs the ring and the ingester ring.
6. Querier. Two or more replicas. It needs the bucket and the ring.
7. Query-frontend. Two or more replicas. It needs the queriers.
8. Ruler. One or more replicas. It needs the queriers and its own bucket.

The order matters. The distributor cannot push until at least one ingester is JOINING the ring. The querier cannot answer until at least one ingester is ACTIVE. The query-frontend gives 502s until the queriers are ready. Bringing them up in this order is the difference between a clean startup and a noisy one.

How to validate it

Five checks confirm a microservices deployment is wired correctly. None of these require a load test.

# READ-ONLY: confirm each component knows its own target.
for pod in $(kubectl get pods -n loki -o name); do
  kubectl logs -n loki "$pod" --tail 50 | grep -m1 'target='
done
# expected: one line per pod, with the right target for the
# right pod (distributor -> distributor, etc.)

# READ-ONLY: confirm the ingester ring is healthy.
curl -s http://loki-ingester:3100/ring | jq '.shards[] |
  { id: .id, state: .state, registered: .registered }'
# expected: every replica in JOINED or ACTIVE state, none in
# LEAVING or PENDING.

# READ-ONLY: confirm the compactor is the singleton.
curl -s http://loki-compactor:3100/services | jq -r '.services[]'
# expected: compactor. The compactor is the only component that
# appears in its own /services without an instance ring listing.

# READ-ONLY: confirm the query-frontend is in front of the
# querier and the cache is wired.
curl -sG http://loki-query-frontend:3100/loki/api/v1/query \
  --data-urlencode 'query={job="varlog"}' \
  --data-urlencode 'limit=1' | jq '.status, .data.result | length'
# expected: status=success, result=array of streams.

# READ-ONLY: confirm the distributor and ingester are in the
# same ring view from the distributor side.
curl -s http://loki-distributor:3100/distributor/ring | jq .
# expected: ingester ring is visible with all three replicas.
# A distributor that cannot see the ingester ring is misrouted
# at the network layer.

A passing result on all five checks is the minimum bar. Anything less and a query against Grafana will return a different answer than a query against logcli.

How it can fail

Six failure shapes appear repeatedly in microservices Loki deployments. Each is mapped to a symptom the on-call engineer can recognise.

1. Ingester ring split. The KV store is reachable from some ingesters but not others. A subset of ingesters drops out of the ring. Symptom: loki_ingester_ring_members drops, streams for affected tenants start returning no chunks found from the querier, and pushes from the distributor return 500 because the hash cannot place the stream on a JOINED ingester.
2. Compactor duplicate. Two compactor pods run at once after a Helm rollback that did not wait for the old pod to terminate. Symptom: the new compactor logs another compactor is holding the lock and refuses to run. Retention stops advancing; index files accumulate uncompacted.
3. Index-gateway unreachable from querier. The NetworkPolicy is tightened on the index-gateway but the querier is not updated. Symptom: every LogQL query that hits the index returns 500 with index-gateway unreachable while raw chunk queries still work.
4. Distributor version skew with ingester. The distributor rolls forward to a new Loki version that introduces a new push field. Symptom: loki_distributor_push_errors_total spikes with rpc error: code = Unknown desc = unknown field until the ingesters catch up.
5. Query-frontend without cache backend. The cache block is removed during a config cleanup. Symptom: every query reaches the querier, query latency rises by an order of magnitude, and loki_query_frontend_results_cache_hits_total stays at zero.
6. Ruler evaluating against a stuck querier. The ruler endpoint points at a querier that has crashed and is being restarted by its HPA on every evaluation cycle. Symptom: loki_ruler_evaluations_failed_total rises and loki_ruler_evaluations_total plateaus. Alerts stop firing.

How to troubleshoot it

The diagnostic order is the same for every component:

1. Is the component running? kubectl get pods -n loki or the equivalent systemctl status for non-Kubernetes deployments. A missing pod is the most common cause of a missing component and the cheapest to fix.
2. Which target is the binary running? curl /services on the component’s listener shows the components registered in the process. If the list does not match the -target flag, the wrong config file is mounted.
3. Is the ring healthy? curl /ring shows the replicas and their state. A replica in PENDING is trying to join; a replica in LEAVING is being shut down; a replica missing from the listing is not reaching the KV store.
4. Is the bucket reachable? loki_objstore_request_errors_total and loki_objstore_request_duration_seconds on the component metrics endpoint show the latency and failure rate per request type. A spike in 5xx from the bucket affects every component that reads or writes it.
5. Is the rate limit firing? loki_distributor_lines_rejected_total and loki_discarded_samples_total{reason} show the shape of the rejection. The lesson on Loki performance troubleshooting walks through the per-reason breakdown.
6. Is the cache wired? loki_query_frontend_results_cache_hits_total should be non-zero under steady-state dashboard load. A flat counter means the cache backend is unreachable or the cache block was removed from the config.

Security implications

Each component has its own HTTP listener and its own metrics endpoint. Twelve components means twelve listeners to lock down. The minimum network policy in production:

• distributor, query-frontend — reachable from the application fleet (or from the Alloy / OTel Collector fleet) and from Grafana. No other source.
• ingester, index-gateway, compactor, querier, ruler — reachable only from inside the Loki namespace. The compactor in particular has no authentication on its admin endpoints and must not be exposed.

The ring KV store is an internal dependency. It does not need to be reachable from the application fleet. It does need to be reachable from every ingester, every distributor, every index-gateway, and every ruler. The object store credential is read-write for every component that writes (distributor via ingester, compactor, ruler) and read-only for every component that reads (querier, query-frontend). Splitting the credentials into read-write and read-only IAM roles is the standard production discipline.

Performance implications

The performance ceiling per component:

• distributor — CPU-bound on label validation and rate- limiting. 4 vCPU per pod handles roughly 100 MB per second of pushed streams.
• ingester — memory-bound on in-memory chunks. A 16 GiB pod holds roughly 2 TB per day of in-flight streams with WAL on; without WAL the safe ceiling is closer to 200 GB per day.
• querier — network-bound on bucket fetches and CPU-bound on LogQL execution. 4 vCPU per pod handles roughly 20 concurrent queries.
• query-frontend — CPU-bound on query splitting and parallelisation. With a results cache, the cache absorbs most of the dashboard query load; without it, the querier pool saturates.
• compactor — disk-bound on its working directory. An NVMe local disk at 3 GB per second sustains a 1 TB per day compaction rate.
• ruler — CPU-bound on rule evaluation. Mostly idle unless the rule set is large.
• index-gateway — network-bound on TSDB index fetches. Small CPU footprint per query.

Production guidance

• Plan for one compactor. Plan for three ingesters with replication factor 3. Plan for two or more of every other component.
• Use the Helm chart or the production/ksonnet manifests as the configuration baseline. Custom manifests drift; canonical manifests track the upstream upgrade cadence.
• Pin the Loki version per component during a rolling upgrade. Allow up to one minor version of skew between distributor and ingester, and between querier and index-gateway, but never skip more than that.
• Monitor loki_target_info for every component. A missing component in the metric set is the first signal of a missing pod.
• Document the topology in the runbook. The on-call engineer at 03:00 should not have to read twelve config files to know which component is failing.

Verification

You should now be able to answer:

• Which seven components make up Loki microservices mode, and which of them are stateful?
• Why does the compactor run as a singleton while every other component runs at two or more replicas?
• What is the correct order to bring the components up at startup, and what fails at each step if you get the order wrong?
• How does an ingester ring split manifest in user-facing symptoms?
• What is the operational cost difference between simple scalable and microservices in terms of components and on-call burden?

Quiz