Skip to main content
RunBook Academy

← All checklists in Observability

Before deploymentloki-readiness

Loki Production Readiness

24 items ·18 critical ·6 warn ·0 info

Run this before Loki takes production logs, and again after any change to the schema, the storage backend, the limits, the retention settings or the tenancy model. Those five are where the irreversible mistakes live; everything else can be adjusted while the cluster runs.

The items are ordered as configuration, then labels, then limits, then retention, then tenancy and recovery. That order is deliberate: the label discipline decides whether the cluster survives at all, the limits decide who gets rejected when it does not, and retention decides what you still have afterwards.

Reading the running configuration

Most items ask the process what it loaded rather than reading the file, because Loki starts happily on a file with a broken section. The configuration endpoint serves YAML, so the commands below parse it with yq; where a section is missing from the output, the key was never loaded and the component is running on its built-in default. Substitute your own component addresses, bucket names and storage profile — the names below are the shapes this course uses throughout, not an estate that exists.

Where a target runs several components in one process, the same endpoint answers for all of them. Where they are separate, ask each one, because the whole point of several of these items is that the separate files can disagree.

The two irreversible settings

Two items on this list can destroy data on their own. The schema list is append-only: removing a historical period makes everything written under it unreadable. Retention deletes on the compactor’s next sweep, with no confirmation and no undo beyond bucket versioning. Both deserve a second reviewer on the pull request, and both are the reason the versioning item is here even though it looks like storage housekeeping.

Sign-off

  • Reviewer: ________________ Date: ___________
  • Platform owner: ___________ Date: ___________
  • Largest log producer: ______ Date: ___________

Critical18 items

  1. loki -config.file=/etc/loki/config-write.yaml -target=write -verify-config
  2. for h in loki-write loki-read loki-backend; do echo "== $h"; curl -s http://$h:3100/config | yq '{"backend": .common.storage_backend, "schema": .schema_config.configs}'; done
  3. curl -s http://loki:3100/config | yq '.schema_config.configs'
  4. curl -s http://loki:3100/config | yq '.common.storage_backend'
  5. logcli --addr=http://loki:3100 labels
  6. curl -s http://loki:3100/config | yq '{"names": .limits_config.max_label_names_per_series, "name_len": .limits_config.max_label_name_length, "value_len": .limits_config.max_label_value_length}'
  7. curl -s http://loki:3100/config | yq '{"rate_mb": .limits_config.ingestion_rate_mb, "burst_mb": .limits_config.ingestion_burst_size_mb}'
  8. curl -s http://loki-distributor:3100/metrics | grep '^loki_discarded_samples_total'
  9. curl -s http://loki:3100/config | yq '{"period": .limits_config.retention_period, "enabled": .compactor.retention_enabled, "delay": .compactor.retention_delete_delay}'
  10. curl -s http://loki:3100/config | yq '.limits_config.retention_stream'; curl -s http://loki:3100/runtime_config | yq '.overrides'
  11. curl -s http://loki-backend:3100/metrics | grep loki_compactor_oldest_processed_age_seconds
  12. aws s3api get-bucket-lifecycle-configuration --bucket prod-loki-chunks --profile loki-storage
  13. curl -s http://loki:3100/runtime_config | yq '.overrides | keys'
  14. curl -s http://loki-distributor:3100/config | yq '.auth_enabled'
  15. curl -s -o /dev/null -w "%{http_code}\n" -X POST http://loki-distributor:3100/loki/api/v1/push -H "Content-Type: application/json" -d '{"streams":[{"stream":{"job":"probe"},"values":[["1700000000000000000","probe"]]}]}'
  16. aws s3api get-bucket-policy --bucket prod-loki-chunks --profile loki-storage | jq -r '.Policy' | jq '[.Statement[].Action] | flatten'
  17. cat /var/backups/loki/drill/last-drill.txt
  18. curl -s -G http://meta-prom:9090/api/v1/query --data-urlencode 'query=up{job=~"loki.*"}' | jq -r '.data.result[] | "\(.metric.instance) \(.value[1])"'

Warning6 items

  1. curl -s http://loki-write:3100/services; curl -s http://loki-read:3100/services
  2. curl -s http://loki:3100/config | yq '{"rate": .limits_config.per_stream_rate_limit, "burst": .limits_config.per_stream_rate_limit_burst}'
  3. curl -s http://loki:3100/config | yq '{"reject": .limits_config.reject_old_samples, "max_age": .limits_config.reject_old_samples_max_age}'
  4. aws s3api get-bucket-versioning --bucket prod-loki-chunks --profile loki-storage; aws s3api get-bucket-replication --bucket prod-loki-chunks --profile loki-storage
  5. curl -s http://loki-ruler:3100/config | yq '{"alertmanager": .ruler.alertmanager_url, "interval": .ruler.evaluation_interval, "storage": .ruler.storage.type}'
  6. openssl s_client -connect loki.example.internal:8443 </dev/null 2>&1 | grep -E 'Protocol|Cipher'