Storage Architecture

A team’s Prometheus instance runs on a single VM with a 1 TB EBS volume. The retention is set to 90 days. The TSDB fills the disk on day 73; the head block starts returning storage_tsdb_out_of_order errors; the team gets paged at 02:00. They discover that the metrics volume grew 40% after a service added a high-cardinality label. They do not have a plan to grow the volume. They do not have a plan to shrink the data. They have a 02:00 pager and a 30-minute customer-impacting dashboard gap.

The storage architecture is the set of decisions about where each signal lives, how long it lives there, and how it gets there. Get it right and the data is honest, the cost is predictable, and the pager is quiet. Get it wrong and the pager is the loudest signal the platform emits.

What storage architecture is

Storage architecture is the placement of data across the three storage tiers and the protocols that move data between them.

                +-------------+      +---------------+
   Producer --->|   Hot tier  |----->|  Warm tier    |---> S3 (cold)
                | (local disk)|      | (object store)|     archive
                +-------------+      +---------------+
                       |
                       |  (queries served from hot)
                       v
                  Grafana queries

• Hot tier. Local disk. Sub-second queries. Single-digit days of retention. The expensive place to keep data; the only place to query it cheaply.
• Warm tier. Object storage (S3, GCS, Azure Blob, MinIO). Seconds-to-minutes query latency. Weeks to months of retention. The cheap place to keep data.
• Cold tier. Object storage with infrequent-access or archive storage class. Minutes-to-hours query latency. Months to years of retention. The cheapest place to keep data; the worst place to query it.

Each backend in the observability stack makes a different choice about the tiers.

Why a sysadmin cares

Three failure shapes appear when the storage architecture is under-designed.

1. The disk that fills at 02:00. A Prometheus with 90-day retention on a fixed-size volume. The volume fills; the TSDB refuses to compact; queries return storage_tsdb_out_of_bounds_time. The fix is monitoring on prometheus_tsdb_storage_blocks_bytes with an alert at 70% of disk and an automated plan to either grow the disk or shorten the retention.
2. The S3 credentials that drift. A Loki or Tempo that uses IAM instance profiles for S3 access. The IAM role is rotated; the new role does not have s3:GetObject. Writes fail with 403 AccessDenied; ingester logs pile up; queries return resource_exhausted. The fix is IAM policy audits in CI and an alert on the backend’s per-ingester error rate.
3. The retention policy that nobody owns. A team sets Loki retention to 30 days for cost reasons. The compliance team later requires 90 days for a regulated log stream. The retention setting is global; the override is per-stream; the documentation does not say which streams are exempt. Symptom: data older than 30 days is unqueryable for the team that needs it most.

How it works

Prometheus local TSDB

Prometheus writes samples to a memory-mapped head block. Every two hours, the head is compacted into a series of immutable on-disk blocks. The on-disk blocks are 2-hour, 6-hour, or default 25-hour windows, depending on configuration. Each block is mmap’d on read; queries are answered by iterating the blocks.

The TSDB is single-host. A single Prometheus instance owns its own TSDB; the TSDB is not shared with another Prometheus. Replication is achieved by running two Prometheus instances (each with its own TSDB), or by running Thanos / Mimir which read Prometheus’s remote_write stream and re-shard it.

Loki chunks + index

Loki separates the index from the data. The index is a TSDB (boltdb or TSDB-on-S3 since Loki 2.9) that records “which log streams exist and where their chunks live.” The chunks are gzip-compressed batches of log lines, written to object storage keyed by tenant, stream, and time window.

A query against Loki resolves to: “find the chunks whose labels match this stream selector, decompress them, filter by line content.” The bottleneck is the chunk-decompression rate and the object store’s GET latency.

Tempo blocks

Tempo stores traces as immutable blocks in object storage. Each block contains the trace tree for a time window. The block is written once and read many times by the querier. A search query (“show me traces with this service and this tag”) walks the search index (stored on local disk as a bloom filter) to find candidate blocks, then reads those blocks from object storage.

The remote_write stream

Prometheus’s remote_write is the only place in the stack where data crosses from one backend’s storage into another backend’s storage. The stream is gRPC with snappy compression; each batch contains up to 1000 series for a single scrape window.

remote_write is the seam between the hot tier and the warm tier for metrics. It is also the seam where a Prometheus becomes a thin shipper and Mimir / Thanos / Cortex become the real backend.

How to configure it

Each backend has a different configuration for storage. The production shape is shown below.

# /etc/default/prometheus  -- local TSDB path and retention.
# Both are flags; prometheus.yml has no key for either.
ARGS="--storage.tsdb.path=/var/lib/prometheus \
      --storage.tsdb.retention.time=30d \
      --storage.tsdb.retention.size=200GB"

# /etc/prometheus/prometheus.yml  -- remote_write
global:
  scrape_interval: 15s
  external_labels:
    cluster: eu-west-1-prod

remote_write:
  - url: https://mimir.example.com/api/v1/push
    basic_auth:
      username: ${REMOTE_WRITE_USERNAME}
      password_file: /etc/prometheus/remote_write_password
    tls_config:
      ca_file: /etc/prometheus/ca.crt
    queue_config:
      capacity: 10000
      min_shards: 4
      max_shards: 50
      batch_send_deadline: 5s
    write_relabel_configs:
      - source_labels: [__name__]
        regex: go_gc_.*
        action: drop

The hot tier is /var/lib/prometheus with 30 days of retention and a 200 GB cap, both set by flag. The warm tier is Mimir, reached by remote_write. The write_relabel_configs drop unwanted series before they hit the WAN.

# /etc/loki/loki-config.yaml  -- chunks in S3, index in TSDB
common:
  ring:
    kvstore:
      store: memberlist
  replication_factor: 3
  compactor_address: loki-compactor:3100

schema_config:
  configs:
    - from: 2026-01-01
      store: tsdb
      object_store: s3
      chunks: tsdb
      index: tsdb

storage_config:
  tsdb_shipper:
    active_index_directory: /loki/tsdb-index
    cache_location: /loki/tsdb-cache
  aws:
    s3: s3://eu-west-1/loki-prod
    bucketnames: loki-prod
    region: eu-west-1

limits_config:
  retention_period: 744h
  retention_stream:
    - selector: '{job="compliance-audit"}'
      priority: 1
      period: 2160h

The retention_stream block overrides the global retention for specific streams. The compliance-audit stream keeps 90 days; everything else keeps 31 days.

# /etc/tempo/tempo.yaml  -- blocks in S3
storage:
  trace:
    backend: s3
    s3:
      bucket: tempo-prod
      region: eu-west-1
    wal:
      path: /var/tempo/wal
    pool:
      max_workers: 200
      queue_depth: 8000

compactor:
  compaction:
    block_retention: 744h

querier:
  frontend_address: tempo-querier:9095

Tempo’s compactor periodically merges small blocks into larger ones and removes blocks older than block_retention. The WAL (the write-ahead log on local disk) absorbs S3 write outages without losing traces.

How to validate it

# READ-ONLY: Prometheus TSDB is healthy.
curl -fsS http://prometheus:9090/api/v1/status/runtimeinfo | jq .data
# {"data":{"GOGC":"100","GOMAXPROCS":"4","storageRetention":"30d,200GB"}}

# READ-ONLY: Prometheus TSDB head is not stuck.
curl -fsS http://prometheus:9090/api/v1/status/tsdb | jq '.data.headStats'
# {"numSeries": 1234567, "chunkCount": 89, ...}

# READ-ONLY: Loki is healthy and writing to S3.
logcli ready
curl -fsS http://loki:3100/config | jq '.storage_config.aws.bucketnames'
# ["loki-prod"]

# READ-ONLY: Loki index and chunks are reachable.
logcli query '{job="node"}' --since=1h --limit=1

# READ-ONLY: Tempo is healthy and writing to S3.
curl -fsS http://tempo:3200/ready
curl -fsS http://tempo:3200/api/status | jq .ingester

# CONFIGURATION: a synthetic end-to-end trace round-trip.
tempo-cli query trace-id 0e8e5c1f...

A clean validation: the TSDB head count is not climbing, the object store bucket exists and accepts writes from the backend’s IAM role, and a round-trip from ingest to query returns within the expected latency.

How it can fail

The most expensive storage-tier failure modes, in order of how often they appear in incident reviews.

1. Disk fills, head block stops. Prometheus local TSDB hits the disk cap; the head block returns storage_tsdb_out_of_order. Symptom: dashboards show gaps for the duration of the outage; alerts evaluate over staleness.
2. S3 credentials rot. A Loki or Tempo instance’s IAM role is updated; the new policy omits s3:GetObject. Writes return 403 AccessDenied. Symptom: ingester logs fill with auth errors; queries return no data even for recent time ranges.
3. remote_write queue overflow. Prometheus remote_write to Mimir falls behind; the queue fills past its capacity. Symptom: prometheus_remote_storage_queue_high_water hits its cap; samples are dropped at the sender.
4. Retention policy that nobody owns. Loki retention is global; a compliance-driven stream needs longer retention; the override is not configured. Symptom: audit data is unqueryable after 31 days; a regulatory finding follows.
5. Compactor falls behind. Loki or Tempo compactor falls behind its schedule; the index or block count grows without bound. Symptom: ingester memory rises; query latency rises because more blocks must be scanned.
6. Hot tier queried beyond its scope. A team queries Loki for a 30-day range with line filters; every chunk in 30 days is fetched and decompressed. Symptom: query latency spikes to 30+ seconds; the querier pool is exhausted.

How to troubleshoot it

The diagnostic order is “is the hot tier up?”, “is the warm tier reachable?”, “is the policy configured?”.

1. Start at the hot tier. For Prometheus, prometheus_tsdb_storage_blocks_bytes. For Loki, loki_ingester_memory_chunks. For Tempo, tempo_ingester_*. A metric that is climbing without bound is the failure.
2. Check the warm tier. loki_objctl or aws s3api head-bucket. If the bucket returns 403, IAM is the problem.
3. Check the policy. For Loki retention, curl http://loki:3100/config | jq .limits_config.retention_period. For Tempo retention, curl http://tempo:3200/config | jq .compactor.compaction.block_retention.
4. Reproduce the query. logcli query '\{job="node"\}' --since=30d for Loki. If the query takes 30 s, the warm tier is being read at the cold tier’s latency; either move the data back to hot or shorten the time range.
5. Check the queues. For Prometheus remote_write, prometheus_remote_storage_pending_samples and _failed_samples. A queue that is consistently high means the remote receiver is the bottleneck.

Security implications

• Object storage is the durable copy. The IAM role that writes to S3 is a high-privilege role; it should not be the same role that reads from S3. Separation of read and write makes credential rotation simpler and limits blast radius.
• Encryption at rest is the default, not the option. S3 server-side encryption with KMS-managed keys (SSE-KMS) is the baseline. Client-side encryption adds CPU cost and is rarely justified for observability data.
• Network isolation. The backend’s egress to S3 should traverse a VPC endpoint, not the public internet. A aws:SourceVpce condition on the bucket policy prevents cross-VPC writes.
• Tenant separation in shared storage. Multi-tenant Loki and Tempo use the bucket key prefix to separate tenants. A misconfigured path_prefix in the S3 configuration is a cross-tenant data leak.

Performance implications

• Hot tier IOPS matter. Prometheus TSDB writes are random 4 KB writes; NVMe is the right choice. EBS gp3 volumes with provisioned IOPS are acceptable; network-attached storage (NFS, CIFS) is not.
• Object store GET latency is the warm-tier floor. Loki queries are bounded below by the bucket’s GET p99; a bucket in a different region adds 50 ms to every GET. The bucket should be in the same region as the querier.
• Compactor is a CPU and memory consumer. Loki and Tempo compactors keep days of data in memory during compaction. The compactor instance type should be sized for the worst-day compaction, not the average day.
• Cardinality multiplies at every tier. A label added at ingest multiplies through every query and every compaction. Storage cost scales with cardinality even when the data volume does not.

Production guidance

• Local TSDB on NVMe. Chunks in S3. Blocks in S3. The canonical hot tier is local NVMe; the canonical warm tier is S3 in the same region as the backend; the cold tier is infrequent-access S3 with a restore-on-read window.
• Retention is per-stream, not global. Default retention applies to the bulk of streams; per-stream overrides cover compliance and audit needs.
• Monitor the disk, the queue, and the bucket. Alerts on prometheus_tsdb_storage_blocks_bytes at 70% of disk, prometheus_remote_storage_pending_samples at 80% of queue cap, and loki_ingester_wal_bytes at 70% of disk catch the three most common failures before they page.
• Test restore from object storage. A backup that has never been restored is a backup that does not exist. The production drill is to restore one Loki chunk and one Tempo block from S3 every quarter and confirm the data is queryable.

Verification

You should now be able to answer:

• Why does Prometheus keep its TSDB on local disk instead of on S3?
• What does Loki put in object storage and what does it keep locally?
• When should retention be set per-stream rather than globally?
• Which storage metric should alert before the disk fills?

Quiz