Storage Architecture Basics

A team runs Loki on a single VM with a 2 TB NVMe. They ingest at roughly 150 MB per second. The disk fills on day 14; the ingester starts returning out of order errors; dashboards go blank at 02:00. They grow the volume to 4 TB. It fills again on day 29. They grow it to 8 TB. At month three the cost review shows that their observability bill is now 28% of their cloud spend, and they are keeping 60 days of logs that nobody has ever queried.

The mistake was treating storage as a single tier. The right shape is three tiers, each tuned for a different access pattern.

What storage architecture is

Storage architecture is the placement of observability data across three tiers, the protocols that move data between tiers, and the policies that decide how long data lives in each tier.

              Ingest
                |
                v
         +-------------+     +-------------+     +-------------+
         |   Hot tier  |---->|  Warm tier   |---->|  Cold tier  |
         | local disk  |     | object store|     | object store|
         | sub-second  |     | seconds to  |     | minutes to  |
         |   queries   |     |  minutes    |     |   hours     |
         +-------------+     +-------------+     +-------------+
                |                  |                   |
                |                  |                   |
                v                  v                   v
          Grafana queries    Grafana queries      Grafana queries
          (live dashboards)  (incident review)    (compliance pull)

• Hot tier. Local NVMe or SSD on the host running the backend. Sub-second query latency. Single-digit to low-tens of days of retention. The expensive place to keep data; the only place to query it cheaply.
• Warm tier. Object storage (S3, GCS, Azure Blob, MinIO) using a standard storage class. Seconds-to-minutes query latency. Weeks to months of retention. The cheap place to keep data; a slower place to query it.
• Cold tier. Object storage using an infrequent-access or archive storage class (S3 Glacier, GCS Coldline, Azure Archive). Minutes-to-hours query latency. Months to years of retention. The cheapest place to keep data; the worst place to query it.

A backend does not have to use all three. Prometheus uses one (hot). Loki uses two (warm and cold, with a small hot index). Tempo uses two (warm, with a WAL on local disk for hot). The tiered layout is the design space, not the default.

Why a sysadmin cares

Storage is the line item that grows fastest and is hardest to undo. Three operational failures appear repeatedly when storage architecture is under-designed.

1. The disk that fills at 02:00. Retention is set to a number of days the disk cannot hold. The team does not have monitoring on storage growth. The first time they hear about it is when dashboards go blank and an on-call engineer gets paged. The fix is monitoring on the storage metric of the relevant backend with an alert at 70% of the tier capacity.
2. The bill that doubles after one service ships verbose logging. A new release sets log level to debug for one service. The log volume doubles overnight. There is no per-stream ingest cap and no cardinality limit. The bill arrives at the end of the month. The fix is per-stream limits on the ingester and an alert on the rate of ingest.
3. The retention that compliance needs but the platform cannot answer. A regulator asks for 90 days of audit logs. The platform keeps 30. The data exists in the ingesters but was evicted from the hot tier before the export ran. The fix is per-stream retention overrides and a documented mapping between regulated streams and their retention values.

How it works

The three tiers are not interchangeable. Each one has a different cost model, a different access pattern, and a different failure mode.

The hot tier

The hot tier is local disk attached to the backend host. Typical hardware in 2026 production is NVMe SSD with provisioned IOPS in the tens of thousands. The hot tier serves queries with sub-second latency because the data is memory-mapped and iterated on read.

Hot tier is fast, expensive per byte, and limited in capacity. The cost per gigabyte-month of provisioned NVMe is roughly 10-30x the cost per gigabyte-month of S3 standard. The cost per IOPS is roughly 100-1000x. The capacity is whatever the host can hold, which is tens of terabytes at the upper end.

The warm tier

The warm tier is object storage with a standard storage class. The bucket is reachable over HTTPS from the backend; writes are multipart uploads; reads are GET requests with per-object range support. The S3 API is the de facto standard; GCS and Azure Blob implement it with a thin compatibility shim. MinIO implements it natively and runs on premises.

Warm tier is cheap per byte, slow per request, and effectively unlimited in capacity. The cost per gigabyte-month is roughly 1/10 to 1/30 of local NVMe. The cost per GET request is non-trivial at scale (a few fractions of a US cent per thousand requests). The capacity is unbounded from the operator perspective.

The cold tier

The cold tier is the same object storage with a different storage class. S3 Glacier Deep Archive costs roughly 1/100 of S3 standard per gigabyte-month but charges for retrieval latency in hours and per-request retrieval fees that can dominate the bill at high query rates.

Cold tier is the cheapest place to keep data and the worst place to query it. The trade-off is real: a single restore of a 1 TB block from Glacier Deep Archive takes 12 hours and costs a few tens of US dollars. Use it for data that is written once and queried by exception.

Tier        Latency (read)        $/GB-month    Typical retention
Hot         <100 ms               $0.20 - $0.50  1-30 days
Warm        50-500 ms             $0.02 - $0.04  30-180 days
Cold        minutes to hours      $0.001 - $0.005 180 days - 7 years

The numbers are illustrative. The actual price depends on the cloud, the region, the volume, and the negotiated contract. The shape of the table is the thing to internalise: latency, cost, and retention are coupled.

Under the hood

Each observability backend makes a specific choice about how to use the tiers. The choices are not arbitrary; they follow from the access pattern of the signal.

Prometheus writes samples to a memory-mapped head block on local disk and compacts the head into immutable on-disk blocks every two hours. The blocks are mmap’d on read. The local TSDB is the only copy Prometheus keeps. Prometheus does not write directly to object storage; it writes via remote_write to Mimir or Thanos, which then store the samples in their own bucket.

Loki writes compressed chunks of log lines to object storage keyed by tenant, stream, and time window. The index that records “which chunks exist for which stream” is a TSDB on local disk (since Loki 2.9) or another bucket. The index is small (kilobytes per stream) and fast to query; the chunks are large (megabytes to gigabytes per object) and slow to query.

Tempo writes trace blocks to object storage. Each block is the trace tree for a time window. The block metadata is a small file; the block itself is the trace data. Tempo also keeps a write-ahead log (WAL) on local disk to absorb S3 write outages without losing traces.

Mimir writes samples and chunk files to object storage across multiple buckets (one per component: ingester chunks, store-gateway blocks, ruler state). Mimir is the metrics backend that was designed to put all data in object storage from day one.

Backend     Hot tier (local)              Warm/cold tier (object store)
----------- ----------------------------- -------------------------------
Prometheus  TSDB head and blocks          None (or remote_write to Mimir)
Loki        Index (TSDB)                  Chunks (S3/GCS/Azure/MinIO)
Tempo       WAL                           Blocks (S3/GCS/Azure/MinIO)
Mimir       Ingester WAL                  Chunks and blocks (S3/GCS/Azure)

The pattern is the same in every case: the index or the WAL that lets the backend find the data is local; the data itself is in the bucket.

How to configure it

Storage architecture is a configuration choice before it is a cost choice. The example shape below shows where each tier is declared.

# /etc/default/prometheus  -- local TSDB path and retention.
# Both are flags; prometheus.yml has no key for either.
ARGS="--storage.tsdb.path=/var/lib/prometheus \
      --storage.tsdb.retention.time=30d \
      --storage.tsdb.retention.size=200GB"

# /etc/prometheus/prometheus.yml  -- reloadable configuration
global:
  scrape_interval: 15s
  external_labels:
    cluster: eu-west-1-prod

Prometheus here is a thin shipper. The hot tier is the local TSDB. The warm tier is reached through remote_write to Mimir, configured separately.

# /etc/loki/loki-config.yaml  -- chunks in S3, index on disk
common:
  ring:
    kvstore:
      store: memberlist
  replication_factor: 3
  compactor_address: loki-compactor:3100

schema_config:
  configs:
    - from: 2026-01-01
      store: tsdb
      object_store: s3
      chunks: tsdb
      index: tsdb

storage_config:
  tsdb_shipper:
    active_index_directory: /loki/tsdb-index
    cache_location: /loki/tsdb-cache
  aws:
    s3: s3://eu-west-1/loki-prod
    bucketnames: loki-prod
    region: eu-west-1

limits_config:
  retention_period: 744h

Loki here is the canonical two-tier shape: a small index on local disk, the chunks in S3 standard. The cold tier (Glacier) is reached through an S3 lifecycle policy on the bucket, not through Loki itself.

# /etc/tempo/tempo.yaml  -- blocks in S3, WAL on disk
storage:
  trace:
    backend: s3
    s3:
      bucket: tempo-prod
      region: eu-west-1
    wal:
      path: /var/tempo/wal
    pool:
      max_workers: 200
      queue_depth: 8000

compactor:
  compaction:
    block_retention: 744h

Tempo here mirrors Loki: a WAL on local disk to absorb write outages, the trace blocks in S3 standard. The compactor merges blocks and removes blocks older than block_retention.

How to validate it

# READ-ONLY: Prometheus TSDB retention and path are correct.
curl -fsS http://prometheus:9090/api/v1/status/runtimeinfo | jq .data
# {"data":{"GOGC":"100","GOMAXPROCS":"4","storageRetention":"30d,200GB"}}

# READ-ONLY: Prometheus TSDB head is healthy.
curl -fsS http://prometheus:9090/api/v1/status/tsdb | jq '.data.headStats'
# {"numSeries": 1234567, "chunkCount": 89, ...}

# READ-ONLY: Loki is healthy and the bucket is reachable.
logcli ready
curl -fsS http://loki:3100/config | jq '.storage_config.aws.bucketnames'
# ["loki-prod"]

# READ-ONLY: Loki returns a query result.
logcli query '{job="node"}' --since=1h --limit=1

# READ-ONLY: Tempo is healthy and the bucket is reachable.
curl -fsS http://tempo:3200/ready
curl -fsS http://tempo:3200/api/status | jq .ingester

A clean validation: the TSDB head count is not climbing without bound, the object store bucket exists and accepts writes from the backend IAM role, and a round-trip query returns within the expected latency.

How it can fail

The most expensive storage architecture failures, in order of how often they appear in incident reviews.

1. Single tier for everything. The team keeps all data on local disk because “the queries are fast.” The disk fills; the cost grows linearly with retention; the eventual migration is a quarter-long project. Symptom: dashboards go blank when the disk fills; the bill grows faster than the data volume.
2. Wrong tier for the workload. The team puts Loki chunks on local NVMe because they read the Loki docs and saw the filesystem storage option. Local NVMe is fast but unscalable and per-host. The hot tier is for the index; the chunks belong in object storage. Symptom: each new ingester adds disk cost; queries do not improve.
3. Cold tier without retrieval budget. A team configures S3 Glacier for compliance logs and forgets that retrieval is charged per request. A single compliance pull of 10 TB costs more than a year of standard storage. Symptom: the compliance team calls the platform team because their quarterly restore took three days and cost more than the data itself.
4. Hot tier queried at cold-tier scale. A team keeps 90 days of data in the hot tier by paying for 30 TB of NVMe. The queries scan all 30 TB on every dashboard load. The hot tier’s IOPS saturate. Symptom: dashboards take 30+ s to load; the backend reports IOPS saturation in the storage metrics.
5. Retention set without a per-stream override. The team sets Loki retention to 30 days globally. A regulated stream needs 90 days. The override is not configured. Symptom: audit data is unqueryable after 30 days; a regulatory finding follows.
6. Index lost, data orphaned in the bucket. The Loki ingester crashes and its local index is corrupted. The chunks still exist in the bucket, but the index no longer records their location. Symptom: queries for the affected streams return empty results even though the data exists.

How to troubleshoot it

The diagnostic order is “is the hot tier healthy?”, “is the warm tier reachable?”, “is the policy configured?”, “is the tier being used for the right data?”.

1. Start at the hot tier. prometheus_tsdb_storage_blocks_bytes, loki_ingester_memory_chunks, tempo_ingester_*. A metric that is climbing without bound is the failure.
2. Check the warm tier. aws s3api head-bucket or the equivalent in your cloud. If the bucket returns 403, the IAM role is the problem. If the bucket is reachable but empty, the backend is not flushing to it.
3. Check the policy. For Loki, curl http://loki:3100/config | jq .limits_config.retention_period. For Tempo, curl http://tempo:3200/config | jq .compactor.compaction.block_retention.
4. Reproduce the query. logcli query '\{job="node"\}' --since=30d for Loki. If the query takes 30 s, the warm tier is being read at the cold tier latency; either move the data back to hot or shorten the time range.
5. Check the bucket storage class. aws s3api list-objects --bucket loki-prod --query "Contents[].StorageClass". If the chunks are in Glacier when they should be in Standard, the lifecycle policy is wrong.

Security implications

• Object storage credentials are high-value secrets. The IAM role that writes to S3 should not be the same role that reads from S3. Separation of read and write makes credential rotation simpler and limits blast radius if a credential leaks.
• Encryption at rest is the default, not the option. S3 server-side encryption with KMS-managed keys (SSE-KMS) is the baseline. Client-side encryption adds CPU cost and is rarely justified for observability data.
• Network isolation. The backend egress to S3 should traverse a VPC endpoint, not the public internet. An aws:SourceVpce condition on the bucket policy prevents cross-VPC writes and stops a leaked credential from being used outside the VPC.
• Tenant separation in shared storage. Multi-tenant Loki and Tempo use the bucket key prefix to separate tenants. A misconfigured path_prefix in the storage configuration is a cross-tenant data leak.

Performance implications

• Hot tier IOPS matter. Prometheus TSDB writes are random 4 KB writes; NVMe is the right choice. EBS gp3 volumes with provisioned IOPS are acceptable; network-attached storage (NFS, CIFS) is not.
• Object store GET latency is the warm-tier floor. Loki queries are bounded below by the bucket GET p99; a bucket in a different region adds 50 ms to every GET. The bucket should be in the same region as the querier.
• Compactor is a CPU and memory consumer. Loki and Tempo compactors keep days of data in memory during compaction. The compactor instance type should be sized for the worst-day compaction, not the average day.
• Cardinality multiplies at every tier. A label added at ingest multiplies through every query and every compaction. Storage cost scales with cardinality even when the data volume does not.

Production guidance

• Start with the canonical tiered shape. Local TSDB on NVMe for Prometheus hot. S3 standard for Loki chunks and Tempo blocks. Glacier only when the query pattern is rare-and-large.
• Right-size the hot tier to 7-30 days. The hot tier exists for the queries that drive dashboards. Anything older belongs in the warm tier.
• Right-size the warm tier to 30-180 days. The warm tier exists for the queries that drive incident reviews and compliance pulls. Anything older belongs in the cold tier.
• Set retention per-stream, not just globally. Default retention applies to the bulk of streams; per-stream overrides cover compliance and audit needs.
• Test restore from object storage. A backup that has never been restored is a backup that does not exist. The production drill is to restore one Loki chunk and one Tempo block from S3 every quarter and confirm the data is queryable.

Verification

You should now be able to answer:

• What are the three storage tiers and what access pattern is each one suited for?
• Which cost dominates in each tier (per-byte, per-IOPS, per-request)?
• Why does Prometheus keep its TSDB on local disk and not on S3?
• What does Loki put in object storage and what does it keep locally?
• What is the most common cause of a storage bill that doubles overnight?

Quiz