Loki at Scale Configuration

A platform team sees query latency rise from 2 seconds to 12 seconds after a Grafana upgrade adds 40 new dashboard panels. They add four more querier pods. Latency improves briefly, then returns to 12 seconds. They add a memcached cluster with a 24-hour results cache. Latency drops to 1 second within an hour. The four extra queriers are decommissioned two weeks later.

The choice between scaling and tuning is the operational decision that determines whether the next month of Loki work is about Kubernetes or about config. Knowing the answer for each bottleneck is the difference between a budget spent on infrastructure and a budget spent on the wrong infrastructure.

What it is

Loki tuning is the discipline of choosing between three moves when a bottleneck appears:

1. Scale up. Add replicas of the component that is saturating. The most expensive move in terms of infrastructure cost. The right move when the bottleneck is throughput that grows linearly with workload and the component is cheap to add.
2. Tune the config. Change a knob on the component that is saturating. The cheapest move in terms of infrastructure cost. The right move when the bottleneck is the way the workload interacts with a default value that does not fit.
3. Rewrite the workload. Change the LogQL query, the dashboard poll, or the per-tenant rate limit. The cheapest move in terms of infrastructure cost and the most expensive move in terms of organisational cost. The right move when the bottleneck is the workload shape, not the platform capacity.

The mistake is treating the three moves as interchangeable. They are not. Scaling a component that is bottlenecked by a query shape does not help; the new replicas are saturated by the same shape. Tuning a knob that does not match the bottleneck does not help; the new setting adds CPU to a component that already had CPU headroom.

Why a sysadmin cares

The scale-vs-config decision is the operational decision that recurs every month in any Loki deployment above 200 GB per day. The reasons it matters:

1. Cost discipline. A new Loki pod costs roughly the same per month as the right config change. The right answer is often the cheaper one.
2. Time-to-fix. A config change can be rolled out in minutes; a horizontal scale event takes an hour of HPA reconciliation plus the underlying node capacity. A config change is faster when the right knob exists.
3. Failure surface. Every new replica is a new failure domain. A bottleneck fixed by config is a bottleneck that cannot recur on a pod failure. A bottleneck fixed by scaling is a bottleneck that recurs every time a pod fails.

How it works

The three moves are not interchangeable. Each addresses a different bottleneck:

  Bottleneck                         First move
  -------------------------------    ------------------------------
  Distributor CPU saturation         Scale up: add distributor pods
  Ingester memory pressure           Tune: lower chunk_target_size
                                     or raise chunk_idle_period
  Querier pool saturation            Tune: add results cache, lower
                                     max_concurrent, raise query-
                                     frontend parallelism
  Bucket bandwidth exhausted         Tune: lower split_queries_by_
                                     interval; rewrite: narrow the
                                     LogQL query
  Query-frontend CPU saturation      Scale up: add query-frontend
                                     pods (stateless)
  Compactor disk pressure            Tune: raise compaction_interval
                                     and lower retention_delete_delay
  Ruler eval failures                Rewrite: simplify the rule
                                     expression or raise
                                     flush_period

Two patterns emerge:

• CPU-bound components scale linearly. The distributor and the query-frontend are CPU-bound and stateless. Adding a replica costs the same CPU as raising the existing replica size, and adding a replica is operationally simpler. Scale is the first move for these components.
• Memory-bound components tune first. The ingester is memory-bound on chunk cache. Adding replicas does not reduce per-pod memory pressure; the pressure comes from the workload, not the replica count. Tuning chunk_target_size and chunk_idle_period changes the memory ceiling. Tune is the first move for the ingester.

How to configure it

The per-bottleneck tuning set.

Distributor CPU saturation

The distributor’s CPU scales linearly with the number of pods. Scale is the right first move. The config knob to consider if scaling alone is not enough is the rate limiter: a Redis-backed rate store lets the distributor enforce per-tenant limits without holding per-tenant state in memory.

# loki-distributor.yaml
distributor:
  rate_store:
    backend: redis
    redis:
      endpoint: redis.internal:6379
  pool:
    health_check_ingesters: true

Ingester memory pressure

The ingester is memory-bound on chunk cache. Tuning the chunk shape changes the memory ceiling without raising CPU cost.

# loki-ingester.yaml
ingester:
  chunk_target_size: 1048576      # 1 MB (down from 1.5 MB)
  chunk_idle_period: 30m          # shorter window closes chunks sooner
  max_chunk_age: 1h               # hard cap on chunk age
  wal:
    enabled: true
    dir: /var/lib/loki/wal

Three knobs to understand:

• chunk_target_size controls the byte-size at which a chunk is closed and flushed. Lower values close chunks sooner; the per-pod memory ceiling drops in proportion.
• chunk_idle_period controls the idle time at which a chunk is closed. Lower values close chunks sooner when a stream goes quiet.
• wal.enabled does not change memory but changes durability; WAL adds a small CPU and disk cost but eliminates in-flight chunk loss on crash.

Querier pool saturation

The querier is the cheapest component to scale and the most expensive to operate under an unbounded query. The order of operations when the pool saturates is:

# loki-query-frontend.yaml
query_range:
  split_queries_by_interval: 12h       # narrower than 24h
  parallelise_shardable_queries: true
  results_cache:
    cache:
      memcached:
        endpoint: memcached.internal:11211
        ttl: 24h

# loki-querier.yaml
querier:
  frontend_address: loki-query-frontend:9095
  worker_parallelism_factor: 8          # up from 4
  max_concurrent: 10                    # down from 20
  query_timeout: 30s                    # tighter than 60s

The order of operations:

1. Add or expand the results cache. This is the highest- leverage move.
2. Lower split_queries_by_interval. More sub-queries, more parallelism, lower per-worker cost.
3. Raise worker_parallelism_factor. More chunks fetched in parallel per query.
4. Lower max_concurrent. A saturated pool at max_concurrent: 100 becomes a saturated pool at max_concurrent: 10 after this change, but the saturated pool rejects new work fast instead of hanging.
5. Add querier pods. Only after the cache and the parallelism are already tuned.

Bucket bandwidth exhausted

Bucket bandwidth is a shared resource across the querier pool. Scaling the pool does not add bandwidth; it just adds more clients competing for the same bucket. The tuning answer is narrower queries.

# loki-query-frontend.yaml
query_range:
  split_queries_by_interval: 6h        # narrower sub-queries
  parallelise_shardable_queries: true
  results_cache:
    cache:
      memcached:
        endpoint: memcached.internal:11211
        ttl: 1h                         # shorter TTL catches
                                         # repeat queries within
                                         # the narrow window

If narrowing the queries does not help, the bottleneck is the workload shape, not the platform. The next move is a LogQL rewrite: shorter time ranges, narrower stream selectors, and heavier filters applied first.

Compactor disk pressure

The compactor runs on a single host with a local NVMe working directory. Disk pressure means the compaction interval is too short for the working directory size.

# loki-compactor.yaml
compactor:
  compaction_interval: 1h              # longer than 30m default
  retention_delete_delay: 6h           # longer than 1h default
  working_directory: /var/lib/loki/compactor

A longer interval means fewer compactions per day, less disk turnover, and a larger working directory. If the working directory is full, the compactor cannot start a new compaction and retention stops. Add disk before raising the interval.

Ruler eval failures

The ruler evaluates LogQL on a schedule. Eval failures are usually rule-shape problems, not capacity problems.

# loki-ruler.yaml
ruler:
  flush_period: 1m                      # shorter for stricter rules
  evaluation_delay: 30s                 # stagger evaluations
  rule_path: /var/lib/loki/rules

A simpler rule that completes inside flush_period is the right answer. A rule that needs more CPU should be split into two rules with narrower time ranges.

How to validate it

Five checks confirm the tuning is taking effect:

# READ-ONLY: confirm the ingester memory ceiling has dropped
# after the chunk_target_size change.
curl -s http://loki-ingester:3100/metrics \
  | grep '^loki_ingester_chunk_age_seconds_count'
# expected: a stable chunk count that reflects the new chunk
# target size. A reading still at the old target means the
# change has not rolled out to the running pod.

# READ-ONLY: confirm the results cache hit rate after the
# cache change. The hit rate should rise within minutes of
# the cache going live.
HITS=$(curl -s http://loki-query-frontend:3100/metrics \
  | awk '/^loki_query_frontend_results_cache_hits_total/ {print $2}')
MISSES=$(curl -s http://loki-query-frontend:3100/metrics \
  | awk '/^loki_query_frontend_results_cache_misses_total/ {print $2}')
echo "scale=$((HITS + MISSES)) hits=$HITS misses=$MISSES"
# expected: hits dominating misses. A 1:1 ratio means the
# cache is missing the working set.

# READ-ONLY: confirm the querier pool is not saturated.
curl -s http://loki-querier:3100/metrics \
  | grep '^loki_querier_concurrent_queries ' | head -3
# expected: in-flight queries well below max_concurrent. A
# reading at the limit means the tuning is not effective.

# READ-ONLY: confirm the bucket request rate has dropped
# after the cache and split-by-interval changes.
curl -s http://loki-querier:3100/metrics \
  | grep '^loki_objstore_request_duration_seconds_count'
# expected: a lower rate than before the tuning. A rate that
# has not changed means the tuning is not reaching the bucket
# path.

# READ-ONLY: confirm the compactor disk usage is below the
# working directory capacity.
df -h /var/lib/loki/compactor
# expected: usage below 80 percent. A reading above 90
# percent means the compactor is about to stop accepting new
# compactions.

How it can fail

Five shapes appear repeatedly when the scale-vs-config decision goes wrong:

1. Scaled the wrong component. The querier pool saturates. The team adds distributor pods. Symptom: distributor pods sit at 10 percent CPU while the querier pool continues to saturate. The cost rises without the latency dropping.
2. Tuned the wrong knob. The ingester memory pressure is from too many concurrent streams, not from chunk size. Lowering chunk_target_size does not help. Symptom: per- pod memory stays at the ceiling; CPU rises because chunks close more often. The bottleneck does not move.
3. Added replicas without fixing the cache. The querier pool saturates. The team adds pods. The new pods see the same query stream and saturate too. Symptom: pool size doubles, latency stays the same, infrastructure cost doubles.
4. Split-by-interval too narrow. Lowering split_queries_by_interval from 24h to 5m increases parallelism but also increases the per-query round-trip overhead. Symptom: latency rises as more sub-queries queue; total CPU on the pool rises faster than the latency falls.
5. Rewrote the LogQL but kept the dashboard polling interval. The narrow query is fast per call, but the dashboard still polls every 30 seconds and the cache TTL is 1 minute. Symptom: hit rate stays around 50 percent; latency drops per call but does not drop per minute on the dashboard.

How to troubleshoot it

The diagnostic order for the scale-vs-config decision:

1. Where is the bottleneck? Read loki_request_duration_seconds p99 by component. The component with the high p99 is the bottleneck.
2. Is the bottleneck CPU, memory, network, or disk? kubectl top pod for CPU and memory; node_filesystem_* for disk; node_network_* for network. The resource type determines the right move.
3. Is the bottleneck throughput or query-shape? A throughput bottleneck rises with workload volume; a query- shape bottleneck rises with a specific dashboard, query, or tenant. Compare the metrics before and after isolating the suspect.
4. Is the cache wired? A read-path bottleneck with a flat results cache hit rate is a missing or misconfigured cache. Add or fix the cache before scaling.
5. Has the knob already been turned? curl /config shows the active config. A setting that is not in the running config has not been rolled out.
6. Has the workload been rewritten? A bottleneck that persists across cache, parallelism, and scale changes is the workload. The LogQL, the dashboard, or the per-tenant limit is the next thing to look at.

Security implications

The scale-vs-config decision has security implications beyond the immediate bottleneck:

• Per-tenant rate limits are a tuning knob. Raising the ingestion rate limit to absorb a spike exposes the platform to a noisy tenant. The right move is a per-tenant limit that protects the platform.
• Bucket IAM scopes scale with replicas. Every querier replica needs read access to the bucket. Scaling the pool expands the IAM surface. Read-only credentials limit the blast radius of a leaked credential.
• Cache eviction under load. A memcached cluster under pressure evicts aggressively; a cache miss under load means every query reaches the bucket. A bucket credentials leak during an incident response window has a wider window when the cache is missing.

Performance implications

The performance ceiling of each tuning move:

• Scale. Linear with replicas. Doubling the distributor replicas doubles the distributor throughput. Doubling the ingester replicas does not double the ingester memory budget.
• Cache. Geometric. A 5-minute results cache absorbs the vast majority of dashboard polling load; a 1-minute cache misses every other poll.
• Parallelism. Sub-linear. Raising worker_parallelism_factor from 4 to 8 speeds up wide queries but increases the per-query bucket request burst. The trade-off is bandwidth vs latency.
• Query rewrite. Depends on the query. A query with a filter that excludes 99 percent of chunks before the heavy regex is 100 times faster than the unfiltered version. The effort is in the rewrite, not the platform.

Production guidance

• Start with the cache. The results cache on the query- frontend is the highest-leverage tuning move in production.
• Tune before scaling. A misconfigured component does not get fixed by adding replicas.
• Scale CPU-bound components first (distributor, query- frontend). Tune memory-bound components first (ingester).
• Rewrite before scaling the bucket. A narrow query reaches the bucket less often; scaling the bucket is more expensive than rewriting the query.
• Document the scale-vs-config decision in the runbook. The on-call engineer at 03:00 should know the order of moves.

Verification

You should now be able to answer:

• Why does scaling the ingester pool not reduce per-pod memory pressure?
• What is the right order of tuning moves when the querier pool saturates?
• When is rewriting the LogQL query the right answer instead of adding replicas?
• Why does the cache precede every other move?
• How do you tell the difference between a throughput bottleneck and a query-shape bottleneck?

Quiz