The Backend Layer

A Grafana dashboard shows a five-minute gap in the checkout-failure-rate panel. The on-call engineer opens the panel and confirms the gap. They open Prometheus and the metrics are present every thirty seconds around the gap. They open Loki and the log entries are present in the same window. They open Tempo and the trace store returns zero traces for the window. The on-call engineer has three backends, one of which silently dropped a five-minute window of trace data.

The backend layer is where the data lives. The five-minute gap is a backend failure, not a workload failure. The lesson is that “the backend is up” is not the same as “the backend is answering the question the operator is asking.”

What it is

The backend layer is the set of storage engines that own one signal. In this course:

• Prometheus for metrics. Single-server TSDB with a pull-based ingestion model. The canonical choice for a single-region, single-team deployment.
• Mimir for metrics at scale. Horizontally-scalable, Prometheus-compatible, multi-tenant. The choice for a multi-team or multi-region deployment.
• Loki for logs. Multi-tenant, label-based log store. The query language is LogQL. Index is the labels; payload is the log line.
• Tempo for traces. Multi-tenant trace store. The query is by trace ID or by service tags. Storage is a blob store (S3, GCS, Azure Blob).

The backend layer is the only layer that holds data. The collector forwards; the backend stores, indexes, and serves queries. The presentation layer reads; the backend answers.

Why a sysadmin cares

The backend is the storage layer. The four operational risks:

1. Cardinality explosion. The TSDB is the cardinality sink. Once a high-cardinality label reaches the TSDB, the cost is paid for the lifetime of the retention.
2. Retention mismatch. Logs are retained for 30 days because the index is cheap. Metrics are retained for 15 days because the TSDB is expensive. Traces are retained for 7 days because the volume is the largest. The mismatch is a documentation problem, not a configuration problem.
3. Storage fill. The on-disk volume fills. The TSDB starts returning write errors. Loki starts rejecting pushes. Tempo starts shedding blocks. The on-call engineer has hours to fix it.
4. Single replica. A single Prometheus is a SPOF. A single Loki is a SPOF. A single Tempo is a SPOF. The scale-out answer is Mimir for metrics and the multi-tenant modes of Loki and Tempo.

The right mental model is: each backend owns one signal, and the team that owns the backend owns the storage budget, the retention policy, and the failover plan.

How it works

The three backends with the storage model each one uses:

  Signal   |  Backend  |  Storage model           |  Index         |  Payload
  ---------|-----------|--------------------------|----------------|---------------
  metrics  | Prometheus| on-disk TSDB (per-server) | labels         | float samples
  metrics  | Mimir     | object store + ingester   | labels         | float samples
  logs     | Loki      | chunk store + index       | stream labels  | log lines
  traces   | Tempo     | block store + generator   | tags + trace ID| spans

The Prometheus TSDB is two files per block: an index of (label set → chunk) and a chunk of float samples. A two-hour block is compacted into a long-term block. The default retention is 15 days; the storage cost is bounded by the cardinality, not the volume.

The Loki index is a label-based inverted index. The index stores {service="checkout", level="error"} → chunk IDs. The chunk store holds the log lines compressed. A query is “select chunks matching label selector, return lines matching line filter”. The cost is the chunk count, not the line count.

The Tempo trace store is a blob store. Every trace is a block of spans. The trace ID is the lookup key. A query by service tags is an inverted index on the trace generator. The cost is the trace volume, dominated by the number of spans per trace.

Under the hood

The Prometheus storage path:

  scrape  ->  Appender  ->  head block (WAL)  ->  2h block (memory)
                                                  ->  mmap flush
                                                  ->  compaction (long-term block)
                                                  ->  retention delete

The head block is the in-memory write target. The WAL files back the head block for crash recovery. Every two hours, the head is flushed to disk and a new head starts. The compaction then merges blocks. The retention is enforced by deleting the oldest blocks.

The Loki storage path:

  push (distributor)  ->  ingester (in-memory stream)
                          ->  chunk flush (object store)
                          ->  index (BoltDB or TSDB)
                          ->  query (query-frontend, querier)

The distributor is the entry point. The ingester holds the in-memory stream and the WAL. The chunk is flushed to the object store when the chunk is full or the age threshold is reached. The query path goes through the query-frontend for splitting and caching.

The Tempo storage path:

  OTLP push (distributor)  ->  generator (tag search index)
                              ->  block builder (?)
                              ->  block (object store)
                              ->  query (querier)

The trace is stored as a block of spans. The block generator is the index. The block store is the payload. The query is by trace ID or by tag.

How to configure it

Prometheus 2.55.x. Minimal production config with a single-node TSDB and a remote write for redundancy:

# /etc/prometheus/prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    cluster: prod-eu-west-1

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']
  - job_name: node
    file_sd_configs:
      - files: ['/etc/prometheus/targets/node/*.yml']
  - job_name: app
    file_sd_configs:
      - files: ['/etc/prometheus/targets/app/*.yml']

remote_write:
  - url: http://mimir-distributor.monitoring.svc:9009/api/v1/push
    basic_auth:
      username: prometheus
      password_file: /etc/prometheus/remote_write_password

rule_files:
  - /etc/prometheus/rules/*.yml

The TSDB path and the two retention bounds are command-line flags, not keys in prometheus.yml:

# /etc/default/prometheus
ARGS="--storage.tsdb.path=/var/lib/prometheus \
      --storage.tsdb.retention.time=15d \
      --storage.tsdb.retention.size=200GB"

--storage.tsdb.retention.time and --storage.tsdb.retention.size are both upper bounds. The first to hit wins. The remote_write is the redundancy path; the local TSDB is the local query path.

Loki 3.x. Single-binary mode with the chunks in the object store and the index in BoltDB:

# /etc/loki/loki-config.yaml
auth_enabled: false

server:
  http_listen_port: 3100

common:
  storage:
    s3:
      s3: s3://eu-west-1-prod-loki
      bucketnames: loki-chunks
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2024-01-01
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h

limits_config:
  retention_period: 1440h
  ingestion_rate_mb: 50
  ingestion_burst_size_mb: 100
  per_stream_rate_limit: 25MB
  reject_old_samples: true
  reject_old_samples_max_age: 168h

compactor:
  working_directory: /var/loki/compactor
  retention_enabled: true
  retention_delete_delay: 2h
  delete_request_store: filesystem

query_range:
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 200

Tempo. Single-binary mode with a local block store:

# /etc/tempo/tempo.yaml
server:
  http_listen_port: 3200

storage:
  trace:
    backend: local
    local:
      path: /var/tempo/blocks
    wal:
      path: /var/tempo/wal

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318

ingester:
  trace_idle_period: 10s
  max_block_duration: 5m

compactor:
  compaction:
    block_retention: 168h

metrics_generator:
  registry:
    external_labels:
      cluster: prod-eu-west-1
  storage:
    path: /var/tempo/generator/wal

The retention for Tempo is 7 days (block_retention: 168h). The ingester holds the trace in memory for 10 seconds (trace_idle_period) before flushing to the WAL. The compactor enforces the retention.

How to validate it

CONFIGURATION — validate the configs before reload.

promtool check config /etc/prometheus/prometheus.yml
# SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config

promtool check rules /etc/prometheus/rules/*.yml
# SUCCESS: /etc/prometheus/rules/services.yml is valid

loki -config.file=/etc/loki/loki-config.yaml -verify-config
# config valid

tempo -config.file=/etc/tempo/tempo.yaml -verify-config
# config valid

READ-ONLY — confirm the backends are ready.

curl -sf http://prometheus.monitoring.svc:9090/-/ready
# Prometheus is Ready.

curl -sf http://loki.monitoring.svc:3100/ready
# ready

curl -sf http://tempo.monitoring.svc:3200/ready
# ready

READ-ONLY — confirm the backends are answering queries.

curl -sf 'http://prometheus.monitoring.svc:9090/api/v1/query?query=up' | jq '.data.result | length'
# 240

curl -sf 'http://tempo.monitoring.svc:3200/api/search?tags=service.name%3Dcheckout&limit=1' | jq '.traces | length'
# 1

logcli --addr=http://loki.monitoring.svc:3100 \
  query '{cluster="prod-eu-west-1"} | line "ready"' --since=1h
# 2026-08-13T... {cluster="prod-eu-west-1", ...} ready

READ-ONLY — confirm the storage is healthy.

curl -sf http://prometheus.monitoring.svc:9090/api/v1/status/tsdb | jq '.data.heapChunks,.data.numSeries'
# { "chunkCount": 12, "numSeries": 184000 }

du -sh /var/tempo/blocks
# 234G    /var/tempo/blocks

READ-ONLY — confirm cardinality is bounded.

curl -sf 'http://prometheus.monitoring.svc:9090/api/v1/query?query=count(count%20by%20(__name__)({__name__!=""})' | jq '.data.result[0].value[1]'
# "412"
# 412 series is the budget for this workload class.

How it can fail

1. Disk full. The TSDB stops accepting new blocks. The WAL grows. The next flush fails. The on-call engineer sees prometheus_tsdb_storage_blocks_bytes at the limit.
2. Retention over-run. The TSDB is configured for 15 days. The chain is changed to 30 days. The TSDB has to delete the 15-day-old blocks but the disk is full. The deletion blocks the write path.
3. WAL corruption. The head block is rebuilt from the WAL. The WAL is corrupted. The head block is lost. The last 2 hours of data is missing.
4. Loki ring inconsistency. The ingester is restarted. The new ingester reads the WAL but the previous ingester did not flush. The stream is missing.
5. Tempo block flush failure. The block builder cannot write to the object store. The traces are queued in memory. The queue fills. The distributor rejects new traces.
6. Compactor is not running. The retention is set in config but the compactor is not deployed. The blocks pile up. The disk fills.

How to troubleshoot it

The diagnostic order for “the backend is up but not answering the query the operator is asking”:

1. Is the backend ready? /-/ready is the first check. If the backend is not ready, the symptom is upstream.
2. Is the query well-formed? The query language is the first place to look. A typo in a label selector returns empty. The operator assumes the data is missing.
3. Is the data in the backend? The Loki query count_over_time and the Prometheus query count by (__name__) show the data shape. The comparison against the collector’s self-telemetry tells the operator whether the data is in the backend.
4. Is the retention covering the window? A query for “30 days ago” returns empty for a 15-day retention. The fix is the query, not the backend.
5. Is the storage full? df on the data volume. The prometheus_tsdb_storage_blocks_bytes and the loki_ingester_chunks_bytes are the early-warning metrics.
6. End-to-end probe. Trigger a single trace and search for it. The journey from the workload to the backend is observable.

Security implications

Each backend has its own authentication and authorization model.

• Prometheus. Basic auth on the /api/v1/write endpoint. The remote-write URL holds a credential. The basic_auth.password_file is the production default.
• Loki. Multi-tenant by HTTP header. The X-Scope-OrgID header is the tenant identifier. The distributor enforces the tenant limit. The injection of the header is a collector responsibility.
• Tempo. The query API is multi-tenant. The tenant identifier is the X-Scope-OrgID header. The trace payload is opaque; the tenant model is on the metadata.
• Object storage. The S3 credentials are the most sensitive secret in the stack. The credentials are read from the environment or a secret file. The bucket policy is the second layer of defence.
• Internal ports. The TSDB, Loki, and Tempo internal ports (compactor, ingester, distributor) are not for external traffic. The network policy is the rule.

Performance implications

The backend is the cost centre. The five knobs:

• Retention. The default is 15 days for metrics, 30 for logs, 7 for traces. Raise it for compliance; lower it for cost. The trade-off is operational visibility.
• Storage size. The Prometheus TSDB is bounded by the retention size. The Loki chunks are bounded by the retention period. The Tempo blocks are bounded by the block retention.
• Ingestion rate. The Loki distributor has a per-tenant rate limit. The Prometheus has a per-scrape cardinality budget. The Tempo has a per-ingester queue size.
• Query cost. The Prometheus query is bounded by the series count. The Loki query is bounded by the chunk count. The Tempo query is bounded by the trace count.
• Compaction. The compactor is the background process that enforces the retention. The right pattern is to monitor the compactor’s last success time.

Production guidance

• Document the retention. The retention is the operational contract. A Grafana panel that queries “30 days ago” is broken if the retention is 15 days.
• Monitor the storage. The disk usage is the early-warning signal. The trend is the actionable signal.
• Run a compactor. The retention is configured; the compactor enforces it. The compactor is a service that must run.
• Validate the query. A query for “the last 30 days” is a query for the last 30 days of retention. The Prometheus query is bounded by the retention.
• Scale out at scale. Mimir for metrics, the multi-tenant modes of Loki and Tempo. The single-server mode is the starting point; the scale-out mode is the production target.

Verification

You should now be able to answer:

• What is the storage model for Prometheus, Loki, and Tempo?
• What is the right default retention for each backend?
• Why is a single backend a single point of failure?
• What is the failure mode when the WAL is corrupted?
• What is the difference between ingestion rate and query cost?

Quiz