Caching

A platform team runs Loki at 200 GB per day with three querier pods and no cache. Grafana panels that poll a 6-hour LogQL query every 30 seconds account for 80 percent of the querier load. They add a memcached cluster with 1 GB total memory and a 24-hour TTL on the results cache. Querier load drops by two-thirds within a day. The cost of three memcached nodes is lower than the cost of the four additional querier pods they would otherwise have needed.

The Loki cache is the difference between scaling the read path by adding queriers and scaling the read path by adding memory. Knowing which cache lives where is the difference between a cache that absorbs dashboard load and a cache that misses every query.

What it is

Loki has three distinct cache layers, each owned by a different component:

1. Results cache — owned by the query-frontend. Stores the output of completed LogQL queries, keyed by the normalised query string, the tenant ID, and the query time window. A repeat query within the TTL returns the cached result without ever touching the querier pool. This is the cache that absorbs dashboard polling load.
2. Chunk cache — owned by the querier (and the ingester in some configurations). Stores decompressed chunk payloads after they are fetched from the bucket, so that the second query against the same chunk does not pay the decompression cost. This cache is most useful for queries that re-scan the same recent chunks repeatedly.
3. Index cache — owned by the index-gateway (and the querier for index lookups). Stores the TSDB index entries that resolve a stream selector into chunk references, so that the second lookup does not re-list the index files. This cache is most useful for queries with repeated stream selectors.

The three caches are configured in three different config blocks: query_range.results_cache on the query-frontend, storage_config.chunk_cache_config on the querier, and storage_config.index_queries_cache_config on the index- gateway. None of them share a backend unless the operator points them at the same memcached or Redis cluster.

Why a sysadmin cares

Caching is the operational discipline that determines whether the read path keeps up with Grafana. Three operational pains are specific to the cache layer:

1. Unbounded query cost. Without a results cache, every dashboard poll repeats the same query against the querier pool. A dashboard with 20 panels that polls every 30 seconds issues 1,200 queries per minute against the querier. The same dashboard with a 1-minute results cache TTL issues 20 per minute. The cache absorbs the load.
2. Bucket bandwidth. Without a chunk cache, every query refetches and re-decompresses every matching chunk from the bucket. A repeated query against a 24-hour window with the same stream selector fetches the same chunks twice. The chunk cache collapses the second fetch to a memory hit.
3. Index listing latency. Without an index cache, every stream-selector resolution re-lists the relevant TSDB index files in the bucket. The index cache collapses the second listing to a memory hit.

How it works

The three caches in the read path:

  Grafana / logcli
        |
        v
  +-----------------+   miss              +-----------------+
  | query-frontend  | ---------------+    | querier pool    |
  | results cache   | <------+       +--> | chunk cache     |
  +-----------------+        |       |    +--------+--------+
                             |       |             |
                  hit        |       |             v
                             |       |    +-----------------+
                             |       |    | index-gateway   |
                             |       |    | index cache     |
                             |       |    +--------+--------+
                             |       |             |
                             v       v             v
                       +-----------------------------+
                       | memcached or Redis cluster  |
                       +-----------------------------+

  Legend:
    results cache lives on query-frontend (per-component)
    chunk cache lives on querier (per-component)
    index cache lives on index-gateway (per-component)
    all three can share the same backend cluster

The query-frontend computes the cache key from the tenant ID, the normalised LogQL expression, and the query time window. A query that matches a cached entry returns the cached result without forwarding to the querier pool. The chunk cache on the querier is keyed by chunk ID and contains the decompressed chunk payload. The index cache on the index-gateway is keyed by stream selector and contains the resolved chunk references.

How to configure it

Three cache blocks, one per cache layer. The first is the most important — the results cache on the query-frontend.

Results cache on the query-frontend

# loki-query-frontend.yaml
query_range:
  split_queries_by_interval: 24h
  parallelise_shardable_queries: true
  results_cache:
    cache:
      memcached:
        endpoint: memcached.internal:11211
        max_item_size: 5MB
        ttl: 24h

frontend:
  max_outstanding_per_tenant: 2048
  compress_responses: true

Three production details to call out:

• max_item_size must be large enough to hold the largest legitimate query result. A wide LogQL query that returns megabytes of parsed log lines will exceed the default 1 MB and be silently dropped from the cache. 5 MB is the standard starting point; raise it for known wide queries.
• ttl is the trade-off between hit rate and freshness. A 5-minute TTL catches most dashboard polls but returns up to 5 minutes of stale data. A 24-hour TTL catches every poll but returns 24 hours of stale data on the first request after midnight.
• The cache backend is shared across all query-frontend replicas. The hash of the key lands on one memcached node; losing that node drops every cached entry that hashed to it.

Chunk cache on the querier

# loki-querier.yaml
storage_config:
  chunk_cache_config:
    memcached:
      endpoint: memcached.internal:11211
      max_item_size: 5MB
      ttl: 1h

The chunk cache lives in storage_config, not in querier. The block is consumed by the querier process. The TTL is short because the chunk cache is most useful for re-scans of the same recent chunks; longer TTLs waste memory on chunks that will not be re-read.

Index cache on the index-gateway

# loki-index-gateway.yaml
storage_config:
  index_queries_cache_config:
    memcached:
      endpoint: memcached.internal:11211
      max_item_size: 5MB
      ttl: 1h

The index cache uses the same storage_config block, with a separate index_queries_cache_config key. Like the chunk cache, this lives in storage_config and is consumed by the index-gateway process.

Redis as an alternative

memcached is the right default. Redis is the alternative when the operator needs persistence, replication, or eviction controls that memcached does not provide.

results_cache:
  cache:
    redis:
      endpoint: redis.internal:6379
      timeout: 500ms
      max_item_size: 5MB
      ttl: 24h

Redis requires an authentication block if the cluster is secured. memcached does not have native authentication; it relies on network isolation.

How to validate it

Six checks confirm the cache layer is wired correctly:

# READ-ONLY: confirm the cache backend is reachable from the
# query-frontend. A memcached miss on connectivity is the
# most common cache failure.
curl -s http://loki-query-frontend:3100/metrics \
  | grep '^loki_query_frontend_results_cache_hit_total '
# expected: a non-zero counter. A zero counter means either
# no queries have run yet or every query is missing the
# cache.

# READ-ONLY: confirm the hit rate. The ratio of hits to
# misses is the single most useful cache health metric.
HITS=$(curl -s http://loki-query-frontend:3100/metrics \
  | awk '/^loki_query_frontend_results_cache_hits_total/ {print $2}')
MISSES=$(curl -s http://loki-query-frontend:3100/metrics \
  | awk '/^loki_query_frontend_results_cache_misses_total/ {print $2}')
echo "scale=$((HITS + MISSES)) hits=$HITS misses=$MISSES"
# expected: under steady-state dashboard load, hits should
# dominate misses by at least 3:1. A 1:1 ratio means the TTL
# is too short or the cache is too small.

# READ-ONLY: confirm the chunk cache is wired on the querier.
curl -s http://loki-querier:3100/metrics \
  | grep '^loki_cache_request_total{component="querier"'
# expected: a non-zero counter. The chunk cache should be
# serving at least some requests under repeated queries.

# READ-ONLY: confirm the cache backend itself is alive.
echo 'stats' | nc -q 1 memcached.internal 11211 | head -20
# expected: stats output including 'curr_items' (current
# key count) and 'curr_connections'. An empty response
# means the memcached port is not reachable.

# READ-ONLY: confirm the cache does not have oversized items
# being dropped. A counter called something like
# chunk_cache_dropped_items_total or the memcached
# 'evictions' counter is the signal.
echo 'stats' | nc -q 1 memcached.internal 11211 \
  | grep -E 'evictions|bytes_read|bytes_written'
# expected: evictions non-zero but stable. A sudden rise in
# evictions means the cache is undersized.

# READ-ONLY: confirm the cache backend list in the config is
# actually a list (a common Loki misconfiguration is to pass
# a single backend as a map, which Loki silently ignores).
yq '.query_range.results_cache.cache' \
  /etc/loki/config-query-frontend.yaml
# expected: the cache key is a map containing either
# memcached, redis, or embedded_cache. A missing key means
# the cache is disabled.

How it can fail

Five shapes appear repeatedly:

1. Cache backend unreachable. The cache block points at a memcached or Redis endpoint that the query-frontend cannot reach. Symptom: loki_query_frontend_results_cache_misses_total dominates the hit ratio, every query falls through to the querier pool, the pool saturates.
2. Max item size too small. The cache drops every result above max_item_size. Wide LogQL queries with many log lines exceed the default 1 MB and never cache. Symptom: hit rate stays high for narrow queries but drops to zero for wide ones, dashboards with wide queries slow down after the cache is added.
3. TTL too short. A 1-minute TTL on a dashboard that polls every 30 seconds means the cache misses every other poll. Symptom: hit rate near 50 percent despite a healthy cache backend, latency barely improves after the cache is added.
4. Cache backend eviction storm. The memcached cluster is undersized for the working set, evictions rise, the hit ratio collapses. Symptom: evictions counter rising on the memcached stats, hit rate falling in step.
5. Cache configured on the wrong component. A results cache block appears in the querier config instead of the query-frontend config. Symptom: the cache silently does nothing — every query reaches the querier, the loki_query_frontend_results_cache_* counters are flat zero on the query-frontend, the local cache on the querier has no effect because the cache key is computed at the query-frontend boundary.

How to troubleshoot it

The diagnostic order:

1. Is the cache backend reachable? echo stats | nc -q 1 memcached.internal 11211 returns the memcached stats. An empty response means the port is blocked or the host is down.
2. Is the cache wired? curl /metrics | grep cache_hits. A zero counter means either the cache is not configured or no queries have hit the cache yet.
3. What is the hit rate? Hits / (hits + misses). Below 50 percent means the TTL is too short or the working set is larger than the cache.
4. Is the cache evicting? Memcached stats shows evictions. A non-zero counter is normal; a rising counter is undersized.
5. Is max item size blocking the working set? Compare cache hits for narrow queries (expected high) versus wide queries (expected low). If narrow hits but wide misses, raise max_item_size.
6. Is the cache on the right component? Check the deployment’s config for the right block in the right section. The query_range.results_cache block belongs on the query-frontend. The storage_config.chunk_cache_config block belongs on the querier. The storage_config.index_queries_cache_config block belongs on the index-gateway.

Security implications

The cache stores query results and chunk payloads in memory. The security implications:

• Tenant isolation. The cache key includes the tenant ID. One tenant cannot read another tenant’s cached results. Memcached and Redis do not implement per-key access control; network isolation and authentication on the cache backend are the only defences.
• Sensitive data in cache. Cached results contain log content, including any PII or secrets in the log lines. The cache backend should be treated with the same data classification as the bucket itself.
• Cache poisoning. A cache backend reachable from outside the Loki namespace is an attack surface. A malicious entry that matches a popular query could return stale or fabricated data until the TTL expires. Lock the cache backend behind a NetworkPolicy that allows only the Loki components to reach it.

Performance implications

The cache is memory-bound. A 1 GB memcached node holds roughly 200,000 entries at 5 KB per entry. The working set for a modest Loki deployment (100 dashboards, 5 panels per dashboard, 1 query per panel per minute, 24-hour TTL) is roughly 5 million entries. Plan for at least 30 GB of cache memory for that workload; more for higher cardinality or longer TTLs.

memcached is the right default. Redis is the right alternative when the operator needs persistence, replication, or eviction controls. A memcached cluster scales horizontally by adding nodes; a Redis cluster scales horizontally but requires sharding to be configured.

Production guidance

• Add memcached or Redis before raising the querier pool above three replicas. The cache absorbs load that the pool cannot.
• Set max_item_size to at least 5 MB. The default 1 MB drops every wide query result.
• Set the results cache TTL to at least 5 minutes for dashboard workloads. A 1-minute TTL misses every other poll on a 30-second dashboard.
• Monitor the hit rate and the eviction rate. Hit rate below 50 percent means the cache is too small or the TTL is too short. Eviction rate rising means the working set exceeds the cache memory.
• Document the cache backend in the runbook. The on-call engineer at 03:00 should know where the cache lives and how to restart it.

Verification

You should now be able to answer:

• Which Loki component owns the results cache, the chunk cache, and the index cache?
• What is the difference between max_item_size, ttl, and the memcached memory budget, and how does each affect hit rate?
• Why does a results cache on the querier do nothing?
• What is the failure mode when the cache backend goes down, and what is the correct response?
• How does a hit rate of 80 percent translate into a saving on the querier pool size?

Quiz