Loki Validation

A new Loki cluster is rolled out. The pods are running. The service has endpoints. The on-call engineer declares it healthy. The application fleet is reconfigured to push to the new endpoint. Six hours later the on-call engineer is paged: the dashboards are empty. The Loki cluster accepted the pushes, returned 204s, and silently dropped every line because the runtime config file was not loaded. The validation step was “the pods are Running”. That is not a validation. That is a status check.

This lesson is the validation that catches the failure before the application fleet is pointed at it.

What it is

Loki validation is the set of checks that prove a deployment is doing what the operator intended. There are four classes:

• Process status. The binary is running. This is the Kubernetes Running check or the systemd active (running) state. It does not prove the cluster is functional.
• Endpoint status. The HTTP endpoints respond. The readiness endpoint, the metrics endpoint, the config endpoint, and the services endpoint. These prove the binary parsed the config and started the components.
• Functional status. A push reaches the ingester, a query retrieves it, and the chunk lands in the object store. This is the only check that proves the cluster is doing what the application fleet needs.
• Semantic status. The retention rule is what the compliance requirement asks for, the rate limit is what the largest client needs, and the runtime overrides are loaded. This is the policy check that the configuration matches the runbook.

A validation script covers all four. None of them alone is sufficient.

Why a sysadmin cares

A sysadmin cares because “the pods are Running” is the false negative that ships every other Loki incident. The Loki distributor returns 204 on a successful push; it returns 204 on a push that was rejected by a label-length validation that silently dropped the line; it returns 204 on a push that was delivered to the ingester but the ingester’s WAL directory is full and the chunk could not be written to the bucket. The client cannot distinguish success from failure without reading the response body or querying the logs back.

The validation script catches all four of these. It does so by pushing a line, querying it back, and reading the response. The push succeeds at the API layer; the query succeeds at the read layer; the bucket console shows the object exists. Three layers of proof that the cluster is functional.

How it works

  Validation script
        |
        +--- 1. Process status
        |       systemctl status loki || kubectl get pods -n loki
        |
        +--- 2. Endpoint status
        |       curl /ready
        |       curl /metrics
        |       curl /services
        |       curl /config
        |
        +--- 3. Functional status
        |       curl -X POST /loki/api/v1/push  (synthetic line)
        |       logcli query '{job="smoke"}'  (query it back)
        |       aws s3api head-object --key <chunk-key>  (bucket check)
        |
        +--- 4. Semantic status
                diff config against runbook
                check retention_period against compliance doc
                check rate limits against largest client peak

How to configure it

The validation does not require Loki configuration. It requires logcli on the operator’s workstation and a curl client. The Helm chart includes a logcli container; the Docker Compose example includes it as a sidecar.

# /etc/loki/config.yaml
# Validation does not change this file. The endpoints that
# the validation script calls are the defaults.
auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9095

common:
  ring:
    kvstore:
      store: consul
      consul:
        host: consul.loki.svc.cluster.local:8500
  instance_addr: loki-0.loki-headless.loki.svc.cluster.local
  path_prefix: /var/lib/loki
  storage_backend: s3
  s3:
    s3: s3://s3.eu-west-1.amazonaws.com
    bucketnames: prod-loki-chunks
    region: eu-west-1

schema_config:
  configs:
    - from: '2024-01-01'
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h

limits_config:
  retention_period: 2160h
  ingestion_rate_mb: 32
  ingestion_burst_size_mb: 48
  reject_old_samples: true
  reject_old_samples_max_age: 168h

How to validate it

A complete validation script. Run it after every config change, after every upgrade, and before every production cutover.

#!/usr/bin/env bash
# scripts/validate-loki.sh
# READ-ONLY: validates a Loki deployment end to end.
set -euo pipefail

LOKI_URL="${LOKI_URL:-http://loki.loki.svc.cluster.local:3100}"
BUCKET="${BUCKET:-prod-loki-chunks}"
REGION="${REGION:-eu-west-1}"
TENANT="${TENANT:-smoke}"
LABEL_JOB="${LABEL_JOB:-smoke}"

# 1. Process status. Confirm the pod or systemd unit is running.
echo "[1/4] Process status"
kubectl get pods -n loki -l app=loki --no-headers | awk '{print $3}' \
  | grep -q '^Running$' && echo "  ok" || { echo "  fail"; exit 1; }

# 2. Endpoint status. Every component must report ready.
echo "[2/4] Endpoint status"
READY=$(curl -fsS "${LOKI_URL}/ready")
echo "  /ready: ${READY}"
echo "$READY" | jq -e 'all(.[]; . == "ready")' > /dev/null \
  || { echo "  fail: not all components ready"; exit 1; }

curl -fsS "${LOKI_URL}/services" | jq -e '.services | length > 0' > /dev/null \
  || { echo "  fail: /services returned no components"; exit 1; }

CONFIG=$(curl -fsS "${LOKI_URL}/config")
echo "  /config: $(echo "$CONFIG" | jq -r '.common.storage_backend')"

# 3. Functional status. Push a synthetic line and query it back.
echo "[3/4] Functional status"
NOW_NS=$(date +%s)000000000
curl -fsS -X POST "${LOKI_URL}/loki/api/v1/push" \
  -H 'Content-Type: application/json' \
  -d "{
    \"streams\": [{
      \"stream\": { \"job\": \"${LABEL_JOB}\", \"tenant\": \"${TENANT}\" },
      \"values\": [[ \"${NOW_NS}\", \"smoke-test-${NOW_NS}\" ]]
    }]
  }" > /dev/null
echo "  push: ok"

# Query the line back. Wait up to 5 seconds for the chunk to flush.
sleep 2
RESULT=$(curl -fsS -G "${LOKI_URL}/loki/api/v1/query" \
  --data-urlencode "query={job=\"${LABEL_JOB}\"}" \
  --data-urlencode "limit=1" | jq -r '.data.result[0].values[][1]')
echo "  query: ${RESULT}"
[ "${RESULT}" = "smoke-test-${NOW_NS}" ] \
  || { echo "  fail: query did not return the synthetic line"; exit 1; }

# Confirm the chunk landed in the bucket.
COUNT=$(aws s3api list-objects-v2 --bucket "${BUCKET}" \
  --prefix "fake/${TENANT}/" --max-items 5 \
  --query 'KeyCount' --output text)
echo "  bucket: ${COUNT} objects in prefix"
[ "${COUNT}" -gt 0 ] \
  || { echo "  fail: no objects in the bucket prefix"; exit 1; }

# 4. Semantic status. Diff the running config against the runbook.
echo "[4/4] Semantic status"
RETENTION=$(echo "$CONFIG" | jq -r '.limits_config.retention_period')
RATE=$(echo "$CONFIG" | jq -r '.limits_config.ingestion_rate_mb')
echo "  retention_period: ${RETENTION}"
echo "  ingestion_rate_mb: ${RATE}"

echo "Validation passed."

The script above is the canonical smoke test. The sections can be split into separate CI steps; the order matters because each step depends on the previous step passing.

# READ-ONLY: validate the configuration without starting the binary.
loki -config.file=/etc/loki/config.yaml -verify-config
# expected: "config is valid" on stdout; exit code 0.
# A non-zero exit means the YAML failed parsing or a required
# key is missing.

# READ-ONLY: logcli query against a known stream.
logcli --addr "${LOKI_URL}" \
  --query '{job="smoke"}' \
  --since 1h --limit 10
# expected: a table of log lines. An empty result means the
# synthetic push did not make it into the cluster.

How it can fail

Six failure modes cover the most common “validation passed but the cluster is broken” patterns.

1. Push returns 204 but the line is not queryable. The distributor accepted the push; the ingester accepted the stream; the chunk was not flushed because the ingester has not reached chunk_idle_period or max_chunk_age. Symptom: the query returns empty. The fix is to wait for the flush window or to force a flush via the admin API.

2. Query returns the line but the bucket is empty. The chunk was flushed to a local cache but the S3 PUT failed silently. Symptom: the in-memory chunk is served by the querier for query_ingester_within (default 30 minutes); after that window, the query returns chunk not found. The fix is to inspect the S3 IAM policy and the bucket CORS configuration.

3. Readiness endpoint returns 200 but the querier is empty. The querier reports ready but cannot serve queries because the index-gateway is not loaded. Symptom: queries return no chunks found. The fix is to check the loki_index_request_duration_seconds histogram.

4. Config validation passes but the runtime config is not loaded. loki -verify-config only validates the main config file; it does not load the runtime config. Symptom: the per-tenant overrides are ignored. The fix is to inspect curl /runtime-config and confirm the overrides section is populated.

5. Bucket policy is correct but the bucket is in the wrong region. Symptom: every request returns 301 PermanentRedirect. The fix is to set s3.region to the correct region.

6. Retention period passes validation but is shorter than compliance. The validation script does not enforce compliance. Symptom: the compactor deletes data that the compliance requirement says should be retained. The fix is to add the compliance check to the semantic status section.

How to troubleshoot it

The diagnostic order for a validation failure:

1. Which step failed? The validation script prints the step number. Step 1 is process status; step 2 is endpoint status; step 3 is functional status; step 4 is semantic status.
2. Step 1 failed. kubectl describe pod or journalctl -u loki -n 100. The pod is in CrashLoopBackOff or the service exited.
3. Step 2 failed. curl /ready | jq. Identify the component that is not ready. The component’s logs explain the cause.
4. Step 3 push failed. curl -v -X POST /loki/api/v1/push. The HTTP status code and response body show the rejection reason.
5. Step 3 query failed. logcli --stats shows the query path. The summary line shows the per-step latency (ingester, store, query). A long store latency means the bucket is the suspect.
6. Step 3 bucket check failed. aws s3api head-object on a known chunk key. The 4xx or 5xx response explains the cause.

Security implications

Validation has a security face:

• Authentication is not tested. The validation script uses the unauthenticated /loki/api/v1/push endpoint by default. In a multi-tenant cluster, the push requires the X-Scope-OrgID header. The validation script must be updated to include the header for production clusters.
• The validation pushes synthetic data. The synthetic line is visible in the bucket. If the bucket has a compliance constraint, the validation script must use a label that is excluded from retention or the operator must add the synthetic data to a retention exemption.
• The /config endpoint exposes the full configuration. A misconfigured Loki with auth_enabled: false and an exposed admin port leaks credentials in the /config response. The validation script must not run against an exposed Loki endpoint.

The lesson in 06-loki-hardening covers the full hardening checklist.

Performance implications

The validation script’s performance cost is negligible. A single push and a single query against an empty stream completes in under 5 seconds. The validation script should run in under 30 seconds end to end. A validation script that takes longer is a signal that the cluster is unhealthy.

Production guidance

• Run the validation script after every config change. The CI pipeline should fail if any step fails.
• Run the validation script as part of the pre-deploy hook. A cluster that fails validation should not be promoted to production.
• Run the validation script against the production cluster after every maintenance window. The maintenance window may have introduced a regression that the cluster view does not show.
• Store the validation output in a log bucket of its own. The validation history is the audit trail for “when did this cluster last pass”.
• Add the compliance check to the semantic status section. A retention period shorter than the compliance requirement is a deployment blocker.

Verification

You should now be able to answer:

• What are the four classes of validation that distinguish a Loki cluster that is up from one that works?
• Which Loki endpoint exposes the per-component readiness state, and why is a 200 from the endpoint not sufficient?
• Why is the push-to-bucket round-trip the only check that proves the cluster is functional?
• Which metric shows that the compactor is making progress through the marker table?
• What is the difference between loki -verify-config and a full end-to-end validation?

Quiz