Loki and Tempo Config as Code

A Loki deploy lands at 14:00 with the new compactor block enabled against a backend bucket whose prefix changed from loki-chunks/ to loki-v2/chunks/ last week. The compactor loads the config, scans the new prefix, and finds zero chunks. The retention worker has nothing to compact. Storage costs quietly grow for six hours before anyone notices. The chunks are still being written to the new prefix; the old prefix is no longer being pruned because the compactor cannot see those keys. This is what loki.yaml as code prevents — the change to the prefix is a one-line edit that must flow through review, not a console tweak.

This lesson is about Loki 3.x and Tempo configuration files as code. The discipline is the same as for Prometheus and Alertmanager — files in Git, linter in CI, credentials external, and a hot reload (or restart, for these services) on every merge. The differences are in the schema and the validation command.

What it is

Loki’s loki.yaml and Tempo’s tempo.yaml are the YAML files that each binary reads on startup. Each declares:

• For Loki: the auth_enabled, server, distributor, ingester, querier, query_range, ruler, compactor, limits_config, schema_config, and storage_config blocks.
• For Tempo: the server, distributor, ingester, compactor, querier, storage.trace, receivers (OTLP, Jaeger, Zipkin, OpenCensus), and metrics_generator blocks.

Both files are plain YAML. Both can be validated offline with the binary’s -verify-config flag. Both interpolate environment variables for credentials.

Why a sysadmin cares

Loki and Tempo are the big back-ends of the log and trace planes. When their config is wrong, the cost is paid in two distinct failure shapes:

1. Storage cost blowup. A misconfigured compactor or a wrong retention period fills the bucket. The bill rises; nobody notices because the dashboards still work. The fix is a CI rule that asserts schema version, retention period, and bucket prefix are within known values.
2. Ingest path broken. A receiver disabled in the YAML, an OTLP endpoint bound to the wrong interface, a TLS misconfiguration on the gRPC listener. Ingests stop; logs and traces queue or are lost. The fix is a CI smoke test that pushes a test line / test span and asserts the receiver responded.

Each is preventable. Each is amplified when the file is not under configuration management.

How it works

The mental model is “the binary reads the YAML at startup, validates it, and starts the components”:

  loki.yaml (in Git)
       |
  loki -config.file=loki.yaml -verify-config  (CI gate)
       |
  ConfigMap / file mount   (deploy time)
       |
  loki binary starts, opens 3100 (gRPC), 9095 (HTTP)
       |
  dist, ingester, querier, compactor  come up
       |
  /ready returns 200 once schema_config is verified against the bucket

  tempo.yaml (in Git)
       |
  tempo -config.file=tempo.yaml -verify-config  (CI gate)
       |
  ConfigMap / file mount   (deploy time)
       |
  tempo binary starts, opens 3100 (OTLP gRPC), 4317 (OTLP HTTP)
       |
  distributor, ingester, querier, compactor  come up
       |
  /ready returns 200 once the storage backend is reachable

Two details. First, both binaries support -verify-config to parse the file offline without starting any components. CI runs this on every change. Second, neither binary supports hot reload; changes to the YAML require a process restart. The restart preserves in-flight ingests only if the ingester has a WAL configured and the bucket is still reachable.

How to configure it

The shape of the configuration for a small Loki + Tempo deployment:

infra/observability/
  loki/
    loki.yaml
    rules/
      api-recording.rules.yaml
  tempo/
    tempo.yaml

A small but production-shaped loki.yaml:

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9095
  log_level: info

common:
  ring:
    kvstore:
      store: memberlist

distributor:
  receivers:
    grpc: { max_recv_msg_size: 10485760 }
    http: {}

ingester:
  lifecycler:
    ring:
      kvstore:
        store: memberlist
  chunk_idle_period: 15m

schema_config:
  configs:
    - from: 2024-01-01
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h

storage_config:
  aws:
    s3: s3://eu-west-1/loki
    s3forcepathstyle: true
    bucketnames: loki-prod
    region: eu-west-1
  tsdb_shipper:
    active_index_directory: /loki/index
    cache_location: /loki/cache

limits_config:
  ingestion_rate_mb: 10
  ingestion_burst_size_mb: 20
  retention_period: 744h         # 31 days
  max_entries_limit_per_query: 5000

compactor:
  working_directory: /loki/compactor
  compaction_interval: 1h
  retention_enabled: true
  delete_request_store: s3

ruler:
  alertmanager_url: http://alertmanager:9093
  storage:
    type: s3
    s3:
      bucketnames: loki-prod
      region: eu-west-1

A small but production-shaped tempo.yaml:

server:
  http_listen_port: 3100

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318

ingester:
  trace_idle_period: 10s
  max_block_duration: 20m

compactor:
  compaction:
    block_retention: 744h           # 31 days

storage:
  trace:
    backend: s3
    wal:
      path: /var/tempo/wal
    s3:
      bucket: tempo-prod
      region: eu-west-1
      access_key: ${AWS_ACCESS_KEY_ID}
      secret_key: ${AWS_SECRET_ACCESS_KEY}

The CI gate:

.PHONY: loki-tempo
loki-tempo:
  loki  -config.file=loki/loki.yaml -verify-config
  tempo -config.file=tempo/tempo.yaml -verify-config

Both -verify-config flags exit non-zero on schema or storage connection failure. CI runs them on every change.

How to validate it

Four checks: two in CI, two in the prod runtime.

# 1. CI: schema validates offline
loki  -config.file=loki/loki.yaml -verify-config
tempo -config.file=tempo/tempo.yaml -verify-config

# 2. CI: a synthetic push through the receiver returns 202
logcli --addr http://loki-prod-01:3100/loki/api/v1/push \
  --bearer-token-file /dev/null \
  --send-line 'INFO test logcli push'

# 3. Prod: the ingester is accepting writes
curl -fsS http://loki-prod-01:3100/ready
curl -fsS http://loki-prod-01:3100/metrics | grep loki_distributor_*

# 4. Prod: the OTLP receiver is reachable
grpcurl -plaintext tempo-prod-01:4317 list
curl -fsS http://tempo-prod-01:3100/ready

A -verify-config exit non-zero blocks merge. A push that does not return 202 Accepted (Loki) or a grpc.health.v1.Health/Check that returns NOT_SERVING (Tempo) is a config deployment failure that the GitOps controller must surface.

How it can fail

Six concrete failure modes appear repeatedly.

1. Bucket prefix drift. The schema_config prefix changes but the compactor’s working_directory is still pointed at the old prefix; the compactor cannot find chunks; retention fails silently and storage costs grow. The fix is a CI rule that the schema_config.configs[*].index.prefix and the bucket prefix match the deployment chart’s expectations.
2. OTLP receiver bound to the wrong interface. A Tempo receiver bound to 0.0.0.0 from inside a private VPC exposes the gRPC endpoint to anyone in the VPC; a receiver bound to 127.0.0.1 accepts no traffic at all. The fix is a CI rule that the OTLP endpoint is a private CIDR.
3. WAL path not mounted. A Tempo restart loses the WAL on a non-persistent volume; recent blocks are corrupted; ingests from the previous 10 minutes are dropped. The fix is storage.trace.wal.path on a persistent volume, asserted in CI.
4. compactor.retention_enabled: false and no separate process. The retention worker never runs; the bucket grows unbounded. The fix is to run the compactor as a separate process in production, with retention_enabled: true.
5. Schema version pinned too low. schema: v11 is too low for Loki 3.x; ingests start failing under load with schema version mismatch. The fix is to keep at least one version ahead of the binary minimum and to validate that the schema version is in the supported list.
6. Inline AWS credentials in tempo.yaml. A secret_key: AKIA... literal in Git. The fix is ${AWS_SECRET_ACCESS_KEY} and a CI gitleaks rule.

Security implications

Both YAML files are credentials stores in disguise. The discipline:

• All credentials via ${VAR} interpolation; the binary reads them from environment variables at startup.
• The OTLP receivers on Loki and Tempo must bind to private interfaces. 0.0.0.0 is acceptable only behind a reverse proxy that enforces mTLS.
• The S3 / GCS / Azure client uses TLS by default. mTLS configuration lives at storage_config.aws.s3 and storage.trace.s3. Verify the bucket policy enforces TLS as a precondition.
• Both /metrics and /ready endpoints expose service health to the network. Bind them to private listeners, or front with a reverse proxy that enforces auth.
• auth_enabled: true in Loki enables multi-tenant mode; the X-Scope-OrgID header must be set by the data source configuration, never trusted from the network.

Performance implications

The performance cost of the config file is defined by the contents, not the file. The big knobs:

• schema_config.configs[*].schema — the v13 TSDB schema is the current best practice; older schemas are slower.
• storage_config.tsdb_shipper.active_index_directory — must be on fast local disk; the cache caches recent index files.
• limits_config.ingestion_rate_mb and ingestion_burst_size_mb — cap the per-tenant rate; the binary drops ingests over the cap with a 429-equivalent.
• compactor.compaction_interval — short interval means cheaper retention cost (chunks freed promptly); long interval means lag.
• compactor.block_retention — controls how long Tempo keeps blocks. The storage cost scales with this; the SLI cost is the inverse.

The size of the YAML file itself is not the bottleneck; both binaries parse it in milliseconds.

Production guidance

• One loki.yaml and one tempo.yaml per cluster, in Git.
• -verify-config in CI on every change.
• compactor as a separate process in production, not as a side-car of the all-in-one.
• All credentials via ${VAR} interpolation. No plaintext in Git.
• OTLP receivers bind to private VPC IPs.
• WAL path on a persistent volume.
• Schema version pinned to the current major.
• Reload via the GitOps controller: deploy, restart, /ready returns 200, smoke test pushes a sample line / span.

Verification

You should now be able to answer:

• What does loki -verify-config check, and what does it not check?
• Why is compactor.retention_enabled: true with the compactor as a separate process the production posture?
• Why is ${AWS_SECRET_ACCESS_KEY} interpolation in tempo.yaml the discipline, and what fails silently if the env var is unset?
• Why should the OTLP receiver endpoint bind to a private interface rather than 0.0.0.0?

Quiz