Storage Tiering

A team’s Loki bill at the end of Q2 is 2.3x what they budgeted for. The bucket is 80 TB. The team enabled a lifecycle policy that transitions objects to Standard-IA after 30 days and to Glacier after 90 days, thinking they were saving money. What actually happened: their incident review dashboards routinely span 60 days, and every query past day 30 paid the Standard-IA retrieval fee; their compliance pull ran every quarter against 90 days of data, and every retrieval paid the Glacier restore fee. The retrieval fees at the end of the quarter exceeded the storage saving.

Tiering without measuring the access pattern is guessing with money.

What storage tiering is

Storage tiering is the practice of placing data in the storage tier that matches its access pattern, and moving data between tiers as the access pattern changes.

   Hot tier (0-30 days)     Warm tier (30-90 days)     Cold tier (90+ days)
   +---------------------+   +---------------------+   +--------------------+
   | Local NVMe          |   | S3 Standard         |   | S3 Glacier         |
   | Sub-second queries  |   | Seconds to minutes  |   | Hours for restore  |
   | $0.20-0.50 /GB-mo   |   | $0.02-0.04 /GB-mo   |   | $0.001-0.005 /GB-mo|
   +---------------------+   +---------------------+   +--------------------+
            |                          |                          |
            v                          v                          v
       Dashboards,              Incident reviews,           Compliance pulls,
       live alerts              ad-hoc queries             quarterly exports

The access pattern of each observability signal is different:

• Metrics. Mostly sub-second dashboard queries on the last 24 hours. Recording rules into longer-retention aggregates extend the useful retention to months.
• Logs. Search-driven; queries span hours to weeks depending on the use case. Per-stream retention overrides cover compliance needs.
• Traces. Browse-driven; queries span minutes to hours after an incident. Long retention is for rare incidents and compliance.

The tiering policy must match the access pattern. A policy that tiers too aggressively inflates the query cost; a policy that tiers too conservatively inflates the storage cost.

Why a sysadmin cares

Storage tiering is the lever that controls the storage bill. Three failure shapes appear when the tiering policy is under-designed.

1. The bill that is dominated by retrieval. A team transitions objects to Glacier after 7 days but queries routinely span 30 days. The retrieval fees at the end of the month exceed the storage saving. Symptom: the bill is dominated by retrieval fees; the storage class is the cause.
2. The dashboard that loads in 30 seconds. A team transitions Loki chunks to Standard-IA after 7 days. Their dashboards routinely query the last 30 days. Every dashboard load pays the IA retrieval latency. Symptom: dashboard latency rises from 200 ms to 5 s; user complaints about “slow Grafana.”
3. The retention that compliance needs but the platform cannot answer. A regulator asks for 90 days of audit logs. The platform keeps 30. The data exists in the ingesters but was evicted from the hot tier before the export ran. Symptom: audit data is unqueryable after 30 days; a regulatory finding follows.

How it works

The three tiers

Hot tier. Local NVMe or SSD on the host running the backend. Sub-second query latency. Single-digit to low-tens of days of retention. The expensive place to keep data; the only place to query it cheaply.

Warm tier. Object storage with a standard storage class. Seconds-to-minutes query latency. Weeks to months of retention. The cheap place to keep data; a slower place to query it.

Cold tier. Object storage with an infrequent-access or archive storage class. Minutes-to-hours query latency. Months to years of retention. The cheapest place to keep data; the worst place to query it.

Lifecycle policies

A lifecycle policy is a set of rules that apply to objects in a bucket. The rules can transition objects between storage classes and expire (delete) objects after a specified age.

{
  "Rules": [
    {
      "Id": "loki-tiering",
      "Status": "Enabled",
      "Filter": { "Prefix": "tenant-a/" },
      "Transitions": [
        { "Days": 30, "StorageClass": "STANDARD_IA" },
        { "Days": 90, "StorageClass": "GLACIER" }
      ],
      "Expiration": { "Days": 365 }
    }
  ]
}

The policy transitions objects to Standard-IA after 30 days, to Glacier after 90 days, and deletes them after 365 days. The transition is asynchronous and may take several hours; the deletion is asynchronous.

Per-stream retention

Loki supports per-stream retention overrides. The override is a stream selector and a per-stream priority.

limits_config:
  retention_period: 744h
  retention_stream:
    - selector: '{job="compliance-audit"}'
      priority: 1
      period: 2160h

The compliance-audit stream keeps 90 days; everything else keeps 31 days. The priority 1 is the override value; without a priority, the global retention applies.

Prometheus recording rules

Prometheus does not have a cold tier in the same sense as Loki or Tempo; the local TSDB is the only copy. Long retention for Prometheus is achieved through recording rules that aggregate recent metrics into longer-retention aggregates.

groups:
  - name: long-retention-aggregates
    interval: 5m
    rules:
      - record: instance:cpu_usage:5m
        expr: avg by (instance) (rate(node_cpu_seconds_total[5m]))
      - record: instance:cpu_usage:1h
        expr: avg by (instance) (avg_over_time(rate(node_cpu_seconds_total[5m])[1h]))
      - record: instance:cpu_usage:1d
        expr: avg by (instance) (avg_over_time(rate(node_cpu_seconds_total[5m])[1d]))

The 5-minute aggregate has the same resolution as the scrape interval. The 1-hour aggregate has 12x less cardinality in the time dimension. The 1-day aggregate has 288x less. The lower-resolution aggregates keep longer-retention visibility at a small fraction of the storage cost.

Tempo block retention

Tempo supports per-block retention. The retention is configured in the compactor.

compactor:
  compaction:
    block_retention: 744h
    compaction_window: 6h

Tempo’s block_retention is the period after which blocks are removed by the compactor. The compaction_window is the window size for compaction; smaller windows mean more frequent compactions but faster queries on recent data.

Under the hood

The cost trade-off between storage class and retrieval fees is the central engineering decision.

                   $/GB-month     $/1K GET requests     $/1K PUT requests
S3 Standard        $0.023         $0.0004               $0.005
S3 Standard-IA     $0.0125        $0.001                $0.01
S3 Glacier Instant $0.004         $0.01                 $0.02
S3 Glacier Flex    $0.0036        $0.01 (min 250 KB)    $0.03
S3 Glacier Deep    $0.00099       $0.02 (min 250 KB)    $0.05

The numbers are illustrative. The shape is the thing: the cheaper the storage class, the higher the retrieval fee.

A useful calculation: the break-even access rate is the rate at which the retrieval fee equals the storage saving.

For 1 TB kept for 30 days:

Standard:    1024 * 0.023 / 30 = $0.78 / day
Standard-IA: 1024 * 0.0125 / 30 = $0.43 / day + retrieval fee
Saving:      $0.35 / day
Retrieval:   $0.001 per 1000 GETs
Break-even:  $0.35 / $0.001 * 1000 = 350 K GETs / day

For 1 TB of Loki chunks with average chunk size 1 MB, 350 K GETs per day is roughly 100 queries per second against the warm tier. A team that queries more than 100 queries per second against the warm tier should not be on Standard-IA; the retrieval fee will exceed the storage saving.

How to configure it

Loki with three-tier storage:

# /etc/loki/loki-config.yaml  -- chunks in S3, lifecycle on bucket
common:
  ring:
    kvstore:
      store: memberlist
  replication_factor: 3
  compactor_address: loki-compactor:3100

schema_config:
  configs:
    - from: 2026-01-01
      store: tsdb
      object_store: s3
      chunks: tsdb
      index: tsdb

storage_config:
  aws:
    s3: s3://eu-west-1/loki-prod
    bucketnames: loki-prod
    region: eu-west-1

limits_config:
  retention_period: 744h
  retention_stream:
    - selector: '{job="compliance-audit"}'
      priority: 1
      period: 2160h

compactor:
  working_directory: /loki/compactor
  compaction_interval: 10m
  retention_enabled: true

The bucket lifecycle policy (managed at the S3 layer):

{
  "Rules": [
    {
      "Id": "loki-tiering",
      "Status": "Enabled",
      "Filter": { "Prefix": "" },
      "Transitions": [
        { "Days": 30, "StorageClass": "STANDARD_IA" },
        { "Days": 90, "StorageClass": "GLACIER" }
      ],
      "Expiration": { "Days": 365 }
    }
  ]
}

Tempo with three-tier storage:

storage:
  trace:
    backend: s3
    s3:
      bucket: tempo-prod
      region: eu-west-1

compactor:
  compaction:
    block_retention: 744h
    compaction_window: 6h

The Tempo bucket has its own lifecycle policy that transitions blocks to IA at 30 days and to Glacier at 90 days.

Prometheus with downsampling for long retention. The hot-tier path and retention are flags; the rule files are configuration:

# /etc/default/prometheus  -- local TSDB
ARGS="--storage.tsdb.path=/var/lib/prometheus \
      --storage.tsdb.retention.time=7d"

# /etc/prometheus/prometheus.yml
rule_files:
  - /etc/prometheus/rules/*.yml

The local TSDB keeps 7 days of full-resolution samples. Recording rules produce 5-minute, 1-hour, and 1-day aggregates that keep longer in Mimir via remote_write.

How to validate it

# READ-ONLY: S3 lifecycle policy is in place.
aws s3api get-bucket-lifecycle-configuration --bucket loki-prod
# (the policy as JSON)

# READ-ONLY: S3 objects are in the expected storage class.
aws s3api list-objects --bucket loki-prod --max-keys 100 \
  --query "Contents[].StorageClass"
# ["STANDARD", "STANDARD", "STANDARD_IA", "GLACIER", ...]

# READ-ONLY: Loki retention configuration.
curl -fsS http://loki:3100/config | \
  jq '.limits_config.retention_period, .limits_config.retention_stream'
# 744h
# [{"selector":"{job=\"compliance-audit\"}","priority":1,"period":2160h}]

# READ-ONLY: Tempo block retention.
curl -fsS http://tempo:3200/config | \
  jq '.compactor.compaction.block_retention'
# "744h"

# READ-ONLY: Prometheus recording rules are evaluated.
curl -fsS http://prometheus:9090/api/v1/rules | \
  jq '.data.groups[].rules[].name'
# ["instance:cpu_usage:5m", "instance:cpu_usage:1h", "instance:cpu_usage:1d"]

A clean validation: the lifecycle policy is in place; the objects are in the expected storage class for their age; the per-stream retention overrides are configured; the recording rules are producing the expected aggregates.

How it can fail

The most expensive tiering failures, in order of how often they appear in incident reviews.

1. Aggressive IA transition inflates the retrieval fee. A team transitions Loki chunks to Standard-IA after 7 days but queries routinely span 30 days. Symptom: the bill is dominated by retrieval fees; the storage saving is negative.
2. Aggressive Glacier transition inflates the restore fee. A team transitions Loki chunks to Glacier after 30 days but compliance pulls routinely span 90 days. Symptom: the quarterly compliance pull pays a large Glacier restore fee; the storage saving is negative.
3. No per-stream override for compliance. A team sets Loki retention to 30 days globally. A regulated stream needs 90 days. The override is not configured. Symptom: audit data is unqueryable after 30 days; a regulatory finding follows.
4. Lifecycle policy transitions noncurrent versions. A team enables S3 versioning on the Loki bucket. The lifecycle policy transitions the current version to IA but not the noncurrent versions. Symptom: the noncurrent versions accumulate; the bucket size grows without bound.
5. Lifecycle policy transitions delete markers. A team has a lifecycle policy that transitions delete markers to IA. The transitions fail silently because delete markers are 0-byte objects. Symptom: the bucket size appears to grow because delete markers accumulate; the bucket audit shows the unexpected growth.
6. Lifecycle policy expires the wrong prefix. A team sets a lifecycle policy that expires the prefix temp/ after 1 day. The prefix matches the index files, not the chunks. Symptom: the index files are deleted; the chunks become unqueryable; the bucket audit shows the unexpected deletion.

How to troubleshoot it

The diagnostic order is “is the policy in place?”, “is the data in the right storage class?”, “are the queries paying the retrieval fee?”, “is the retention matching the access pattern?”.

1. Start with the policy. aws s3api get-bucket-lifecycle-configuration --bucket <bucket>. Confirm the policy is in place and the transitions match the access pattern.
2. Check the storage class. aws s3api list-objects --bucket <bucket> --query "Contents[].StorageClass". Confirm the objects are in the expected storage class for their age.
3. Check the retrieval metric. The loki_objctl_bucket_request_duration_seconds histogram or the S3 CloudWatch metric for GET requests. A p99 that spikes is a sign of throttling or of a query pattern that does not match the storage class.
4. Check the retention. For Loki, curl http://loki:3100/config | jq .limits_config. For Tempo, curl http://tempo:3200/config | jq .compactor.compaction.block_retention.
5. Re-run the access-pattern analysis. The access pattern changes every quarter; the tiering policy must change with it.

Security implications

• Lifecycle policies do not affect encryption. Objects in any storage class are encrypted with the bucket default (SSE-S3 or SSE-KMS). The transition does not re-encrypt the object.
• Lifecycle policies do not affect access control. The bucket policy applies regardless of storage class. An object in Glacier has the same access policy as an object in Standard.
• Retrieval requires the same permissions as read. A Glacier restore call requires s3:GetObject and s3:RestoreObject. A bucket policy that restricts read permissions also restricts restores.
• Compliance retention overrides security retention. A retention requirement from compliance overrides a retention preference from security. The retention policy must be agreed between the two teams.

Performance implications

• Retrieval latency is the cold tier cost. Standard IA has millisecond GET latency. Glacier Instant has millisecond GET latency. Glacier Flexible has minute-scale retrieval latency. Glacier Deep Archive has hour-scale retrieval latency. The right storage class for the workload is the class whose retrieval latency the workload can tolerate.
• Rehydration from cold tier is a one-time cost. A bulk retrieval from Glacier creates a temporary copy in Standard that lasts for the configured duration. The cost of the temporary copy is the storage cost of Standard for the duration.
• Transition overhead is a small cost. The transition between storage classes is a copy operation. The cost is roughly $0.01 per 1 K objects transitioned. A team that transitions millions of small objects pays a non-trivial transition fee.

Production guidance

• Match the tiering to the access pattern. Standard for daily queries. Standard-IA for weekly queries. Glacier for compliance and rare incidents.
• Right-size the transition ages. A starting point is 30/90/365 days; adjust based on the observed query range.
• Use per-stream overrides for compliance. The compliance stream needs longer retention; the bulk does not. The override is the way to express that.
• Use Prometheus recording rules for long retention. The local TSDB keeps 7 days; the aggregates keep 30/90 days in Mimir.
• Re-evaluate the tiering policy quarterly. The access pattern changes every quarter; the policy must change with it.

Verification

You should now be able to answer:

• What are the three storage tiers and what access pattern is each one suited for?
• How does the per-stream retention override in Loki work?
• How does Prometheus achieve long retention without a cold tier?
• What is the break-even access rate for Standard-IA vs Standard?
• Why must the tiering policy be re-evaluated quarterly?

Quiz