Loki Performance Troubleshooting

A platform team gets paged: Grafana is slow. The on-call engineer opens Loki, runs the same query as Grafana, and gets the result back in 800 ms. They check the cache hit rate: 4 percent. They check the querier pool: every pod is at max_concurrent. They check the bucket request duration: p99 is 14 seconds. The diagnostic is “missing cache plus exhausted bucket”. The fix is a memcached cluster and a narrower split_queries_by_interval. The fix takes an hour. The page-to-fix time would have been minutes if the diagnostic order had been followed.

Loki performance troubleshooting has three locations where a problem lives and a strict diagnostic order that maps symptom to location. Knowing the order is the difference between a one- hour investigation and a five-minute one.

What it is

A Loki performance issue has three locations:

1. Write path. Distributor, ingester, WAL. Symptom is rejection (pushes return 429 or 500) or latency (pushes take seconds to acknowledge).
2. Read path. Query-frontend, querier, ingester head-chunk fetch. Symptom is slow queries, timeouts, or 504s.
3. Storage path. Index-gateway, compactor, bucket. Symptom is slow historical queries, retention not advancing, or bucket errors.

The diagnostic order is the same regardless of location:

1. Service health. Is the component up?
2. Service view. Which components are registered in this process? Is the binary running the target you think it is?
3. Metrics view. What does the per-component loki_request_duration_seconds say? Where is the p99?
4. Log view. What do the component logs say about the failing operation?
5. Bucket view. Is the object store reachable, and at what latency?

The order matters. Most Loki performance investigations skip straight to the bucket view (because “the slow query must be the storage”) and miss the actual cause (usually the cache, or the pool, or the query shape).

Why a sysadmin cares

Loki performance is the operational discipline that determines whether the platform is usable under load. Three operational pains are specific to performance:

1. Hidden bottlenecks. Loki has twelve components with their own metrics. A bottleneck on one component can be invisible on every other component’s metrics until the cascade hits the user-facing endpoint.
2. Symptom overlap. Slow queries, slow ingestion, and slow retention all manifest as “Loki is slow” from the user’s perspective. The diagnostic order is what separates them.
3. Cost of misdiagnosis. Adding a querier pod to fix a cache miss wastes the pod’s CPU. Adding an ingester pod to fix a bucket exhaustion wastes the pod’s memory. The correct diagnosis is the correct fix.

How it works

The diagnostic order maps to the three locations:

  User: "Loki is slow"
        |
        v
  +-------------------+   component not ready?
  | service health    | ---> fix the deployment
  | (kubectl get pods,
  |  curl /ready)
  +---------+---------+
            |
            v
  +-------------------+   component list wrong?
  | service view      | ---> fix the -target flag
  | (curl /services)
  +---------+---------+
            |
            v
  +-------------------+   high p99 on which
  | metrics view      |     component?
  | (request_duration |
  |  by component)
  +---------+---------+
            |
   +--------+--------+--------+--------+
   |        |        |        |        |
   v        v        v        v        v
  write    read    cache   bucket   query
  path     path     miss   exhausted  shape
  high     high     80%    14s p99   wide
   p99     p99     hits     bucket    range
   |        |        |        |        |
   v        v        v        v        v
  ingester querier  add or  narrow  rewrite
  pool     pool     tune   query    LogQL
  exhausted exhausted cache  + cache

The arrows from metrics view to the five possible diagnoses are the most useful pattern in the lesson. Each component metric has a characteristic signature:

• Write path high p99 — loki_distributor_request_duration_seconds and loki_ingester_request_duration_seconds both rise. The ingester is the bottleneck.
• Read path high p99 — loki_request_duration_seconds rises on the querier and on the query-frontend. The querier pool is saturated.
• Cache miss — loki_query_frontend_results_cache_hits_total is flat or the hit rate is below 50 percent.
• Bucket exhausted — loki_objstore_request_duration_seconds p99 rises into the multi-second buckets.
• Query shape — pool saturated but bucket, cache, and ingester all healthy. The query is the cause.

How to configure it

There is no single config block for performance. The relevant configs are the per-component blocks already covered in the other lessons. The diagnostic discipline is in the order of the checks, not in a config.

# The five knobs that affect performance, and where they live.
# This block is a reminder, not a configuration.

# 1. Results cache on the query-frontend.
query_range:
  results_cache:
    cache:
      memcached:
        endpoint: memcached.internal:11211
        max_item_size: 5MB
        ttl: 24h

# 2. Querier pool size and concurrency.
querier:
  max_concurrent: 20
  worker_parallelism_factor: 4
  query_timeout: 60s

# 3. Query-frontend split-by-interval.
query_range:
  split_queries_by_interval: 24h
  parallelise_shardable_queries: true

# 4. Per-tenant limits.
limits_config:
  ingestion_rate_mb: 16
  max_streams_per_user: 100000
  max_query_parallelism: 32

# 5. Ingester chunk shape.
ingester:
  chunk_target_size: 1572864
  chunk_idle_period: 1h

Five production details to call out:

• The cache is the highest-leverage knob. A missing cache makes every other knob less effective.
• max_concurrent is the pool size. A pool at the limit is a pool that is rejecting work.
• split_queries_by_interval controls how wide a sub-query is. A narrow interval raises parallelism; a wide interval lowers it.
• Per-tenant limits protect the platform from one tenant. Raising them globally does not raise the ceiling for the tenant that needs it.
• The ingester’s chunk shape controls memory pressure. A noisy tenant with a wide stream selector needs a tighter chunk idle period.

How to validate it

Seven checks run in the diagnostic order. Each one is the answer to one question.

# READ-ONLY: 1. Service health.
for component in distributor ingester querier query-frontend \
                 index-gateway compactor ruler; do
  curl -s --max-time 3 \
    "http://loki-${component}:3100/ready" \
    | jq -c "{component: \"${component}\", ready: .}"
done
# expected: every component returns {"<component>": "ready"}.
# A not-ready component is the first problem to fix.

# READ-ONLY: 2. Service view.
curl -s http://loki-querier:3100/services | jq -r '.services[]'
# expected: querier (and nothing else on a microservices
# querier pod). A wrong component list means the -target flag
# is wrong.

# READ-ONLY: 3. Metrics view (write path).
curl -s http://loki-distributor:3100/metrics \
  | grep '^loki_distributor_request_duration_seconds_sum'
# expected: a low sum-per-scrape ratio. A high ratio means
# the distributor is slow, which means pushes are slow.

# READ-ONLY: 3. Metrics view (read path).
curl -s http://loki-querier:3100/metrics \
  | grep '^loki_request_duration_seconds_sum'
# expected: a low sum-per-scrape ratio. A high ratio means
# the querier is slow.

# READ-ONLY: 3. Metrics view (cache hit rate).
HITS=$(curl -s http://loki-query-frontend:3100/metrics \
  | awk '/^loki_query_frontend_results_cache_hits_total/ {print $2}')
MISSES=$(curl -s http://loki-query-frontend:3100/metrics \
  | awk '/^loki_query_frontend_results_cache_misses_total/ {print $2}')
TOTAL=$((HITS + MISSES))
if [ "$TOTAL" -gt 0 ]; then
  echo "hit rate: $((HITS * 100 / TOTAL))%"
fi
# expected: hit rate above 50 percent under dashboard load.
# A hit rate below 50 percent means the cache is missing.

# READ-ONLY: 4. Log view.
kubectl logs -n loki deploy/loki-querier --tail 200 \
  | grep -E 'level=(error|warn)' | head -10
# expected: no recent errors or warnings. A pattern of errors
# against a specific gRPC client points at a specific
# component.

# READ-ONLY: 5. Bucket view.
curl -s http://loki-querier:3100/metrics \
  | grep '^loki_objstore_request_duration_seconds_sum'
# expected: a low sum-per-scrape ratio. A high ratio means
# the bucket is the bottleneck.

The seven checks run in order. The first check that returns a non-healthy answer is the cause. If all seven checks are healthy, the issue is the workload shape, not the platform.

How it can fail

Six shapes appear repeatedly:

1. Missing results cache. The query-frontend has no cache block, or the cache backend is unreachable. Symptom: hit rate near zero, every query reaches the querier pool, the pool saturates within minutes under dashboard load.
2. Saturated querier pool. max_concurrent is too low or the queries are too wide. Symptom: pool at the limit, new queries time out, Grafana returns 504.
3. Bucket exhausted. The querier pool’s bucket requests exceed the bucket’s per-prefix throughput. Symptom: loki_objstore_request_duration_seconds p99 rises into the multi-second buckets.
4. Ingester memory pressure. The chunk cache exceeds the ingester’s memory budget. Symptom: loki_ingester_chunk_age_seconds_count plateaus, ingester pods OOM.
5. Compactor not running. Two compactors are racing for the lock, or the compactor pod has crashed. Symptom: retention stops, index files accumulate uncompacted, historical queries slow down as the index grows.
6. Query shape problem. A Grafana dashboard polls a wide LogQL query every few seconds. Symptom: cache hit rate low, querier pool saturated, bucket bandwidth normal. The query is the cause.

How to troubleshoot it

The diagnostic order, with the per-step actions:

1. Service health. Is every component ready? A not-ready component is the first problem to fix. The /ready endpoint is the cheapest diagnostic.
2. Service view. Is the binary running the target you think it is? The /services endpoint lists the components in the process. A wrong list means the -target flag is wrong or the config file is wrong.
3. Metrics view. Where is the p99 high? Read loki_request_duration_seconds per component. The component with the high p99 is the bottleneck. Pair with the cache hit rate and the bucket request duration.
4. Log view. What do the component logs say? A pattern of rpc error against a specific gRPC client points at a specific component. A pattern of OOM in the ingester logs points at memory pressure.
5. Bucket view. Is the object store reachable, and at what latency? loki_objstore_request_duration_seconds p99 is the answer. A spike in 5xx from the bucket is the answer.
6. Workload view. Is the cache miss because the cache is missing, or because the workload bypasses the cache? A dashboard that polls a unique query every second bypasses the cache by design. The fix is in the dashboard, not the platform.
7. Limit view. Has a per-tenant limit fired? Check loki_discarded_samples_total{reason} and the query-side error rates. A tenant hitting the rate limit sees 429s; the platform sees a partial outage for that tenant.

Security implications

Performance troubleshooting has security implications:

• Cache backend access. Inspecting the cache state via nc -q 1 memcached.internal 11211 is read-only but exposes cached results, which may contain PII or secrets. Limit the diagnostic access to the Loki operators.
• Bucket credential scope. A bucket latency spike may be caused by a throttled credential. Throttling is a signal that the IAM policy is wrong. Inspecting the IAM policy is a security review as much as a performance review.
• Per-tenant limits as a security control. A tenant that exceeds max_streams_per_user may be emitting labels that should not be in the logs (per-request UUIDs, secrets). The limit is the first signal; the application review is the second.

Performance implications

The diagnostic order has a performance implication: each step runs against the platform, not against the user’s traffic. Reading /metrics adds one HTTP call per scrape; reading logs adds one kubectl logs per pod; reading the bucket adds one network call against the bucket endpoint. The total cost of running the diagnostic order is well under one second, which is cheaper than the wrong fix.

Production guidance

• Document the diagnostic order in the runbook. The on-call engineer at 03:00 should run the order, not improvise.
• Wire the cache hit rate and the bucket p99 to a dashboard. A hit rate that drops below 50 percent or a bucket p99 that rises above one second are the first signals of a performance issue.
• Test the diagnostic order on a staging cluster. The order only works if every step returns a meaningful value; a staging cluster is the place to confirm.
• Pair the diagnostic with the cache, parallelism, and scaling lessons. The diagnostic is the what; the lessons are the how.

Verification

You should now be able to answer:

• What are the three locations where a Loki performance issue can live, and what is the symptom of each?
• Why is the diagnostic order strict, and what is the cost of skipping it?
• What is the most common Loki performance cause in production?
• How does the cache hit rate metric pair with the bucket latency metric to identify the cause?
• What is the difference between a workload-shape performance issue and a platform-capacity performance issue?

Quiz