Compactor

A platform team inherits a Loki cluster from a previous team. The storage bill is steady. The team adds a new tenant and the bill doubles within a month. The on-call engineer inspects the compactor metrics and sees loki_compactor_oldest_processed_age_seconds flat at zero for the new tenant. The compactor is not running the retention sweep for that tenant. The cause is a compactor.compaction_interval of 1m combined with a compactor.retention_delete_worker_count of 1: the compactor is so slow that it never finishes a cycle for the new tenant before the next interval begins. The bucket grows without bound.

This is the failure shape of the compactor: a singleton with finite capacity and a single-tenant-per-cycle model that silently falls behind when the workload grows.

What it is

The compactor is the singleton Loki component that enforces retention and compacts the TSDB index. It is a writes-only path: it reads from the object store, decides what to delete or merge, and writes the result back to the object store. It does not participate in the write or read path of the cluster; it operates as a sidecar that ages the bucket over time.

The compactor has three jobs:

   Job                       What it does                      When it runs
   ------------------------  --------------------------------  -------------------
   index compaction          merge small TSDB files into       every compaction
                             larger ones                       interval
   retention sweep           delete chunks older than          every compaction
                             retention_period                  interval (if enabled)
   delete-request execution  apply operator-initiated          every compaction
                             deletions                         interval (if enabled)

All three run during the same cycle, on a per-tenant basis. The compactor acquires a lock per tenant, runs its jobs for that tenant, releases the lock, and moves on to the next tenant.

Why a sysadmin cares

The compactor is the role that determines whether the bucket grows unbounded or stays within the retention window. Three operational pains appear in every Loki cluster that does not have its compactor tuned:

1. Compactor not running. A Loki cluster with retention_enabled: true but a missing or wedged compactor grows without bound. The bucket bill is the first sign; the query latency is the second.
2. Retention value too short. A config that says retention_period: 744h (31 days) when compliance requires 365 days. The next compactor sweep deletes everything older than 31 days. The data is gone; the compactor did exactly what it was told.
3. Two compactors competing. Two compactor pods running against the same bucket compete for the per-tenant lock. The loser logs lock already held and stops working. Retention halts until the duplicate is removed.

How it works

A compactor cycle has six steps:

   +-------------------+
   | acquire tenant    |
   | lock              |
   +-------------------+
            |
            v
   +-------------------+
   | list object       |
   | store prefixes    |
   | for the tenant    |
   +-------------------+
            |
            v
   +-------------------+
   | scan marker       |
   | files for the     |
   | day owned by     |
   | this cycle       |
   +-------------------+
            |
            v
   +-------------------+
   | for each marker:  |
   |   compute         |
   |   effective       |
   |   retention       |
   |   (max of global, |
   |    per-tenant,    |
   |    per-stream)    |
   +-------------------+
            |
            v
   +-------------------+
   | if marker is      |
   | expired:          |
   |   delete marker   |
   |   and chunk       |
   +-------------------+
            |
            v
   +-------------------+
   | merge index       |
   | files for the     |
   | tenant            |
   +-------------------+
            |
            v
   +-------------------+
   | release tenant    |
   | lock              |
   +-------------------+

The cycle is bounded by compaction_interval. If the cycle takes longer than the interval, the next interval starts immediately. The compactor does not run cycles in parallel for the same tenant; one tenant is processed at a time.

The writes-only path is the key design choice. The compactor never reads from the write path or the read path; it never serves a query; it never accepts a push. It reads from the object store, makes decisions, and writes back. A failed compactor cycle does not affect the write or read paths; it only affects the bucket size.

The per-tenant lock

The lock is a marker file in the object store:

   s3://prod-loki-chunks/<tenant>/compactor/compactor.lock

A compactor that wants to run a cycle for a tenant writes a file with a UUID and a timestamp. Another compactor that finds the file reads the timestamp; if it is older than compactor.compaction_interval * 2, the lock is considered stale and the second compactor overwrites it. If the lock is fresh, the second compactor skips the tenant and tries again next interval.

This is why two compactors compete but neither corrupts the bucket. The lock guarantees that at most one compactor is running a cycle for a tenant at a time. The losers wait.

The marker file

A marker file is a small JSON object that the ingester writes alongside each chunk. The marker records:

• the chunk ID
• the stream label set
• the tenant ID
• the timestamp

The compactor scans the markers for the day it owns, computes the effective retention for each, and deletes the chunks and their markers when they expire.

   s3://prod-loki-chunks/<tenant>/markers/<day>/<marker_id>

The delete-request path

The delete-request path is an operator-initiated deletion. The operator POSTs a query selector and a time range to /loki/api/v1/delete; Loki records the request in the bucket as a marker; the compactor applies the request on the next cycle. The path is used for right-to-erasure compliance and for one-off drops of bad data.

How to configure it

The compactor is configured in the compactor block, with limits in limits_config.

# /etc/loki/config.yaml (extract)

auth_enabled: false

server:
  http_listen_port: 3100

common:
  ring:
    kvstore:
      store: consul
      consul:
        host: consul.loki.svc.cluster.local:8500
  instance_addr: loki-backend-0.loki-backend-headless.loki.svc.cluster.local
  path_prefix: /var/lib/loki
  storage_backend: s3
  s3:
    s3: s3://s3.eu-west-1.amazonaws.com
    bucketnames: prod-loki-chunks
    region: eu-west-1

schema_config:
  configs:
    - from: '2024-01-01'
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h

# Global limits. retention_period is the floor for every
# tenant; per-tenant overrides can extend it, not shorten it.
limits_config:
  retention_period: 2160h
  retention_stream:
    - selector: '{service="audit"}'
      priority: 1
      period: 8760h

compactor:
  working_directory: /var/lib/loki/compactor
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 2h
  retention_delete_worker_count: 50
  delete_request_store: s3
  apply_retention_in_background: true

Three details to call out:

• working_directory must be persistent across restarts. Losing the directory forces a full rescan on the next cycle, which can take hours for a large bucket.
• retention_delete_delay is the safety buffer between a chunk being written and being eligible for deletion. The default of 2h gives the ingester time to flush; a smaller value risks deleting chunks before they are flushed.
• apply_retention_in_background: true is the right choice for production. The compactor applies retention during the compaction cycle rather than synchronously on each push.

How to validate it

Severity: READ-ONLY.

1. Confirm the compactor is /ready:

curl -s http://loki-backend:3100/ready
# ready

2. Confirm the compactor is making progress:

curl -s http://loki-backend:3100/metrics \
  | grep loki_compactor_oldest_processed_age_seconds
# loki_compactor_oldest_processed_age_seconds 86400

# A value rising over time means progress; a flat value means
# the compactor is stuck.

3. Confirm the compactor is processing every tenant:

curl -s http://loki-backend:3100/metrics \
  | grep loki_compactor_tenants_processed_total
# loki_compactor_tenants_processed_total 142

# Each tenant should be processed at least once per
# compaction_interval.

4. Confirm retention is enabled and the value matches the config:

curl -s http://loki-backend:3100/config \
  | jq '.limits_config.retention_period, .compactor.retention_enabled'
# "2160h"
# true

5. Confirm the per-tenant overrides are loaded:

curl -s http://loki-backend:3100/runtime-config | jq '.overrides'
# {
#   "tenant-a": { "retention_period": "8760h" },
#   "tenant-b": { "retention_period": "4320h" }
# }

6. Inspect the compactor log for errors:

journalctl -u loki-backend -n 100 | grep -E '(compactor|retention|delete)'
# compactor.go: compacting tenant fake for day 19790
# compactor.go: deleted 18423 expired chunks
# compactor.go: merged 12 index files

How it can fail

Six shapes cover the most common compactor-related incidents:

1. retention_enabled: true with a too-short retention_period. The operator intended 365 days; the config says 31. Symptom: the loki_compactor_oldest_processed_age_seconds metric drops to the retention value on the next sweep. Data older than the configured value is gone from the bucket.
2. Two compactor pods running. One holds the lock; the other logs lock already held and refuses to run. Symptom: loki_compactor_tenants_processed_total is half of what it should be; loki_compactor_oldest_processed_age_seconds stops advancing.
3. working_directory lost. The pod was rescheduled to a new node without the persistent volume mounted. Symptom: the compactor logs working directory does not exist and refuses to start.
4. Compaction interval too short. A 1m interval with a 5m cycle means the compactor is always starting the next cycle before the previous one finishes. Symptom: the compactor logs previous compaction still in progress and tenants fall behind.
5. Bucket lifecycle rule expires objects faster than the compactor deletes them. A lifecycle rule that expires objects at 7 days against a 30-day retention produces silent data loss. Symptom: queries that span 8 to 30 days return chunk not found; the compactor’s metric says retention is working; the bucket console disagrees.
6. Per-tenant retention label missing from the selector. A push carries retention_period: "8760h" in its labels, but limits_config.retention_stream has no selector that matches {service="audit"}. Symptom: the audit logs are deleted at the global retention, not at the stream retention.

How to troubleshoot it

The diagnostic order for a compactor-related incident:

1. Is retention_enabled: true? curl /config | jq .compactor.retention_enabled. If false, nothing is being deleted regardless of any other setting.
2. Is the compactor running? kubectl get pods -n loki -l app=loki-compactor. The compactor must be a singleton.
3. Is the per-tenant lock held? Inspect the compactor log for lock acquired or another compactor is holding the lock. The metric loki_compactor_compaction_interval_seconds shows whether cycles are completing.
4. What is the oldest processed age? The metric loki_compactor_oldest_processed_age_seconds. A rising value is progress; a flat or falling value is a stuck compactor.
5. Are the per-tenant overrides loaded? curl /runtime-config. If the overrides section is empty, the runtime config file is not being served.
6. Is the bucket lifecycle rule interfering? Inspect the bucket’s lifecycle configuration. The lifecycle rule must expire objects at or after the retention_period, never before.

Security implications

The compactor is the role that performs deletes. Three surfaces:

• Bucket credentials. The compactor needs s3:GetObject, s3:PutObject, s3:ListBucket, and s3:DeleteObject on the chunks bucket. A leaked key with DeleteObject is a silent log wipe.
• Delete-request endpoint. DELETE /loki/api/v1/delete must be locked down with the same authentication as the rest of the API. An open delete-request endpoint is an attacker-controlled log wipe.
• Retention as a compliance boundary. A retention value shorter than the compliance requirement is a violation. PCI requires 1 year for audit trails. HIPAA requires 6 years for healthcare records. The configuration must satisfy both the floor and the per-request delete path.

Performance implications

The performance cost of the compactor is paid by the compactor and by the bucket:

• Compactor CPU and memory. Each sweep processes one day of marker files. A 90-day retention at 500 GB/day produces 45 TB of marker files. The compactor must hold the marker table in memory during a sweep. The compactor.retention_delete_worker_count controls the parallelism of the DELETEs.
• Bucket request cost. The DELETEs are billed per object. A sweep that deletes 45 TB of data is millions of DELETE requests. The cost is amortised over the compaction interval but is the largest single contributor to the bucket bill for a long-retention deployment.
• Compaction interval. A shorter interval means more frequent sweeps; a longer interval means sweeps take longer and the compactor holds memory for longer.

The right sizing:

• Single tenant, short retention. The defaults (compaction_interval: 10m, retention_delete_worker_count: 50) are fine.
• Multi-tenant, long retention. Raise retention_delete_worker_count to speed up the DELETEs. The default of 50 is conservative; 200 is a common production value.
• High cardinality. The compactor’s index-merge pass is the bottleneck. The metric loki_compactor_index_merge_duration_seconds shows the per-cycle merge time. A rising p99 is a sign of index growth.

Production guidance

• Set retention_period to the compliance requirement, not to the storage budget. The storage budget is sized to the retention requirement, not the other way around.
• Always edit retention_period and retention_enabled together. Both in the same pull request. Both reviewed.
• Enable bucket versioning. The recovery story from a wrong retention value depends on it.
• Take a bucket snapshot before every retention change. The snapshot is the only way to recover from a too-short value.
• Monitor loki_compactor_oldest_processed_age_seconds. Alert if it stops advancing for more than one compaction_interval.
• Run exactly one compactor. Two compactors competing for the lock is a silent retention halt.
• Document the retention value in the runbook. The on-call engineer at 03:00 should not need to read the config to know what retention is in effect.

Verification

You should now be able to answer:

• Why is the compactor a singleton, and what is the role of the per-tenant lock?
• What is the difference between the retention sweep and the delete-request path?
• What is the writes-only path, and why does a failed compactor cycle not affect the read or write paths?
• Which metric shows the compactor is making progress, and what does a flat value mean?
• What is the operational risk of a bucket lifecycle rule that expires objects faster than the retention sweep?

Quiz