Indexes

At 11:14 a Grafana query {service="api"} |= "timeout" returns the answer in under a second. At 11:47 the same query times out after sixty seconds. The on-call engineer inspects the querier metrics and sees loki_index_request_duration_seconds flat at zero and the querier CPU near saturation. The index-gateway is reporting loki_index_gateway_request_duration_seconds p99 at forty-five seconds. The cause is that a chart upgrade six hours earlier renamed the index-gateway service but the querier config still points at the old DNS. The querier never receives a response, but Loki does not error. The query appears to be working until the timeout fires.

This is the failure shape of the index: a silent dependency between querier and index-gateway that the dashboard does not surface.

What it is

The Loki index is the data structure that maps a label selector to the chunks that hold the matching streams. Loki does not index log content. A LogQL filter like |= "timeout" is applied at the chunk level, after the index has reduced the candidate set to a small number of streams. The index is the difference between “scan every byte in the bucket” and “scan a few hundred chunks”.

Two index stores have shipped with Loki:

   Store              Era         Index files            Cardinality ceiling
   ----------------   ---------   ---------------------  ---------------------
   boltdb-shipper     v11, v12    per-day BoltDB files   ~10 M series
                      uploaded to      per tenant
                      object store
   tsdb               v13+        single TSDB file       100+ M series
                      per day          per tenant per day
                      per tenant

The boltdb-shipper is the older store. Each ingester writes to a local BoltDB file for the day, then uploads the file to the object store at the day boundary. The querier pulls the BoltDB file for the queried time window and resolves the label selector against it. The store handles moderate cardinality but has a per-file cardinality ceiling that becomes painful past ten million series per tenant per day.

The TSDB index is the current store. Loki 3.x ships with the TSDB index as the default for v13. The TSDB index is a custom format designed by the Loki team to handle the cardinality shapes that the boltdb-shipper could not: services with hundreds of millions of distinct label values, long retention windows, and multi-tenant isolation.

Why a sysadmin cares

The index is the difference between a Loki that answers LogQL queries in milliseconds and a Loki that times out. Three operational pains appear in every Loki cluster that has not had its index store and configuration tuned:

1. Cardinality ceiling hit. A boltdb-shipper cluster that crosses ten million streams per tenant per day slows dramatically. The index file is loaded into memory per query; the memory cost is paid by every querier replica. The fix is to migrate to the TSDB index, not to add more memory.
2. Index-gateway bottleneck. The index-gateway fronts the TSDB index files for the queriers. A single index-gateway replica becomes the bottleneck before the queriers do. The fix is to scale the index-gateway in microservices mode, not to add more queriers.
3. Per-tenant index isolation leaks. The TSDB index is keyed by tenant. A misconfigured auth_enabled flag or a missing X-Scope-OrgID header causes one tenant’s queries to scan another tenant’s index. The symptom is cross-tenant data leaks, not performance.

How it works

The query path that uses the index has four steps:

   LogQL query
        |
        v
   +-------------------+
   | query-frontend    |  split, cache, fan-out
   +-------------------+
        |
        v
   +-------------------+
   | querier           |  fetch label-selector matches
   +-------------------+
        |
        v
   +-------------------+
   | index-gateway     |  serve per-tenant TSDB index files
   +-------------------+
        |
        v
   +-------------------+
   | object store      |  TSDB files keyed by tenant and day
   +-------------------+
        |
        v
   +-------------------+
   | back to querier   |  chunk list for matched streams
   +-------------------+
        |
        v
   +-------------------+
   | object store      |  chunks
   +-------------------+

The querier resolves the label selector against the index-gateway, receives a list of chunk IDs, and fetches the chunks from the object store. The index-gateway holds the TSDB index files in memory (or pages them in from the object store on demand). The chunk fetch is a separate round trip.

The per-tenant index is the security boundary. Every index file is keyed by tenant; every index-gateway request carries a tenant ID; the index-gateway refuses requests that do not match. In single-tenant mode the tenant ID is fake; in multi-tenant mode the ID is the X-Scope-OrgID header value.

   Object store layout for a single tenant:
   s3://prod-loki-chunks/
     fake/
       chunks/
         08/yyy.../01Hxx...   <- chunk file
         08/yyy.../01Hyy...
       index/
         08/
           19180            <- TSDB index file for day 19180
           19181            <- TSDB index file for day 19181
           ...

The index.period config (default 24h) declares how often a new index file is created. The index.prefix config (default index_) is the file name prefix.

How to configure it

The index store is declared in schema_config. A Loki 3.x deployment that uses the TSDB index has a single entry in schema_config.configs.

# /etc/loki/config-write.yaml (extract)

schema_config:
  configs:
    # From this date forward, use the TSDB index store, the s3
    # object store, and the v13 schema.
    - from: '2024-01-01'
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h

A cluster that was upgraded from Loki 2.x with v12 keeps the v12 entry as well:

schema_config:
  configs:
    # The v12 era. Boltdb-shipper index, old object store
    # namespace. Removing this entry breaks reads of data
    # written under this period.
    - from: '2022-01-01'
      store: boltdb-shipper
      object_store: s3
      schema: v12
      index:
        prefix: index_
        period: 24h

    # The v13 era. TSDB index. New object store namespace
    # (optional; many clusters keep the same bucket).
    - from: '2024-01-01'
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h

The index-gateway section (in microservices mode only):

# /etc/loki/config-index-gateway.yaml
# Mode: microservices, index-gateway target. -target=index-gateway.

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9095

common:
  ring:
    kvstore:
      store: consul
      consul:
        host: consul.loki.svc.cluster.local:8500
  instance_addr: loki-index-gateway-0.loki-index-gateway-headless.loki.svc.cluster.local
  path_prefix: /var/lib/loki
  storage_backend: s3
  s3:
    s3: s3://s3.eu-west-1.amazonaws.com
    bucketnames: prod-loki-chunks
    region: eu-west-1

schema_config:
  configs:
    - from: '2024-01-01'
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h

index_gateway:
  mode: ring

Three details to call out:

• index_gateway.mode: ring is the right choice for a microservices deployment. The querier hashes the tenant ID and forwards the request to the gateway replica that owns the hash range. A single gateway is fine for small clusters; a ring is mandatory for high-throughput ones.
• schema_config.configs is append-only. Removing an entry breaks reads of data written under that period. Adding a new entry is the way to switch index stores.
• index.prefix is shared by every object_store bucket. A cluster that has separate dev and prod buckets must use different prefixes to keep the index files apart.

How to validate it

Severity: READ-ONLY.

1. Confirm the schema config loaded into the running binary:

curl -s http://loki-write:3100/config | jq '.schema_config.configs'
# [
#   {
#     "from": "2024-01-01T00:00:00Z",
#     "store": "tsdb",
#     "object_store": "s3",
#     "schema": "v13",
#     "index": {
#       "prefix": "index_",
#       "period": 86400000000000
#     }
#   }
# ]

2. Confirm the index files exist in the bucket:

aws s3 ls s3://prod-loki-chunks/fake/index/2026-08-13/ \
  --recursive | head -5
# 2026-08-13 09:14:22      12345 fake/index/19180
# 2026-08-13 09:14:22       8934 fake/index/19181

3. Confirm the index-gateway is responsive (microservices mode):

curl -s -H 'X-Scope-OrgID: fake' \
  http://loki-index-gateway:3100/loki/api/v1/index/stats
# {"streams": 8123, "chunks": 184230, "bytes": 1843912841, ...}

curl -s http://loki-index-gateway:3100/ready
# ready

4. Confirm the querier can answer a label selector:

curl -sG -H 'X-Scope-OrgID: fake' \
  http://loki-read:3100/loki/api/v1/series \
  --data-urlencode 'match[]={job="nginx"}' | jq '.data | length'
# 184

5. Confirm the index request latency is healthy:

curl -s http://loki-read:3100/metrics \
  | grep loki_index_request_duration_seconds
# loki_index_request_duration_seconds_bucket{le="0.005"} 1842
# loki_index_request_duration_seconds_bucket{le="0.05"} 1842
# loki_index_request_duration_seconds_bucket{le="0.5"} 1842

How it can fail

Six shapes cover the most common index-related incidents:

1. Schema version mismatch. A new schema_config entry uses schema: v13 but the cluster has not been upgraded past the version that reads v13. Symptom: the binary refuses to start with unknown schema version v13. The loki_index_request_duration_seconds histogram is absent.
2. Index-gateway unreachable from querier. A chart rename changes the service name. Symptom: the querier logs connection refused; queries return after the timeout; loki_index_request_duration_seconds is flat at zero.
3. Boltdb-shipper file too large. A tenant with twenty million streams produces a per-day BoltDB file of several gigabytes. Symptom: the querier CPU is saturated; loki_index_request_duration_seconds p99 is in the tens of seconds.
4. TSDB index file corruption. A network error mid-write produces a partial file. Symptom: the index-gateway logs index file checksum mismatch and skips the file; queries for the affected day return no data.
5. Per-tenant isolation leak. auth_enabled: false in a multi-tenant deployment lets a tenant issue a query with no X-Scope-OrgID header; the default tenant is fake. Symptom: a tenant can read another tenant’s streams by omitting the header.
6. Compactor not merging index files. Old index files accumulate. Symptom: the index-gateway must page in more files per query; loki_index_gateway_request_duration_seconds p99 rises. The bucket bill rises because the old files take up storage.

How to troubleshoot it

The diagnostic order for an index-related incident:

1. Is the schema config loaded? curl /config | jq .schema_config. If the new entry is missing, the loki.yaml did not reload.
2. Is the index-gateway /ready? A 404 or a slow response points at a DNS or a chart-rename drift.
3. Is the querier reaching the gateway? Inspect the querier log for connection refused or no index-gateway available. The metric loki_index_request_duration_seconds is flat at zero when the path is broken.
4. Are the index files in the bucket? aws s3 ls against the expected prefix. A missing prefix means the period or the prefix in the config is wrong.
5. What is the per-tenant stream count? loki_ingester_streams{tenant="..."} and loki_index_gateway_request_duration_seconds. A growing stream count with flat query latency is healthy; a flat stream count with growing latency points at the index.
6. Is the compactor merging index files? loki_compactor_oldest_processed_age_seconds is the same metric as for chunks. A flat value means the compactor is stuck.

Security implications

The index is the per-tenant boundary. Three surfaces:

• Index files in the bucket. A leaked AWS key with s3:GetObject on the chunks bucket is a read breach. The index files alone are not enough to reconstruct the streams (the chunks are still needed), but the label sets in the index may carry tenant-identifying information.
• Per-tenant index isolation. auth_enabled: true and an authentication layer that issues X-Scope-OrgID headers per tenant. Without this, every query lands on the default tenant; the multi-tenant boundary is meaningless.
• Index-gateway mTLS. In microservices mode, the querier-to-index-gateway link should be mTLS. The link carries the tenant ID and the query parameters.

Performance implications

The performance cost of the index is paid at three points:

• Querier-to-index-gateway round trip. A query with a label selector must wait for the index-gateway to resolve the selector before it can fetch chunks. The round trip is a per-query overhead.
• Index file paging. A large per-tenant TSDB index means the index-gateway pages many files per query. The cost is in GetObject calls and per-file parsing.
• Compactor merge cost. The compactor merges index files for retired days. A bucket with many short-lived index files pays a high merge cost.

The right sizing:

• Single tenant, low cardinality. The defaults (store: tsdb, period: 24h) are fine.
• Multi-tenant. index_gateway.mode: ring to share the load across replicas.
• High cardinality. Verify the boltdb-shipper has been migrated to TSDB. A boltdb-shipper cluster past ten million streams per tenant per day is the wrong shape.

Production guidance

• Use the TSDB index for any Loki 3.x deployment. The boltdb-shipper is the legacy store; new deployments should start on TSDB.
• Keep schema_config.configs append-only. Removing an entry breaks reads of data written under that period.
• Pin the schema version to the version your Loki binary reads. v13 is the default for Loki 3.x.
• Scale the index-gateway in microservices mode. The gateway is the bottleneck before the querier is.
• Set auth_enabled: true for multi-tenant deployments and enforce the X-Scope-OrgID header at the gateway.
• Monitor loki_index_request_duration_seconds and loki_index_gateway_request_duration_seconds. p99 latency above 100 ms is a sign of an undersized gateway pool.

Verification

You should now be able to answer:

• What is the difference between the boltdb-shipper and TSDB index stores, and which schema version pins each?
• How does a Loki query resolve a label selector to a list of chunks?
• Why is per-tenant index isolation a security boundary, and what is the role of the X-Scope-OrgID header?
• What does the compactor do to the index over time?
• Which metric shows the querier-to-index-gateway latency?

Quiz