Telemetry Cost Overview

The finance team forwarded the observability bill. It was 38 percent larger than the previous quarter. Nobody on the platform team could say which signal, which team, or which change produced the growth. That gap — between a number that grew and an operator that cannot explain it — is what this lesson teaches you to close.

What telemetry cost is

Telemetry cost is the steady-state monthly spend on storing and querying the three signal types: metrics, logs and traces. It has three independent levers:

• Ingest rate — the volume of new data arriving per second. Sampled points per second for metrics, lines per second for logs, spans per second for traces.
• Retention — how long the platform keeps the data before it is deleted or moved to cold storage. Hot retention is the expensive part; cold retention is often ten times cheaper per byte.
• Bytes per sample — the average size of one unit on the wire and on disk. For Prometheus this is roughly constant per scrape; for Loki it varies by line; for Tempo it varies by span.

The model that connects them is straightforward:

monthly_bytes     = ingest_rate_per_second * bytes_per_sample *
                    seconds_per_month
monthly_storage   = hot_bytes  * hot_cost_per_byte
                  + cold_bytes * cold_cost_per_byte
monthly_cost      = monthly_storage
                  + query_compute_cost
                  + ingest_compute_cost
                  + egress_cost

The equation is honest about what it does not capture: query cost, ingest-side compute and network egress. Each of those is itself a separate budget item. The lesson treats them as additive because in production they are.

Why a sysadmin cares

The sysadmin is the person who gets the alert when prometheus_tsdb_head_series climbs ten percent in a week, or when the loki_ingester_streams_active panel shows a sudden step. The sysadmin is also the person who has to defend the spend to finance when the bill arrives. Both jobs require the same skill: translating between the cost equation and the live platform.

Three operational pains the cost model prevents:

1. The surprise monthly bill. A team added verbose logging to five services during a refactor. Nobody cut the retention. The log-storage line of the bill doubled in 90 days.
2. The undersized quota. A platform was sized for 30 days of retention and then left at 30 days while traffic grew. The store hit its head-series limit, started returning scrape errors, and the team discovered the problem as missing alerts.
3. The cross-team attribution gap. Finance asked which team owned the cost. The platform team could not answer because there were no per-tenant labels and no separate signal budgets.

How the cost model works

The mental model is the same for every signal. Bytes per second arrive at an ingest gateway. Bytes per second are written to a sequence of stores: hot, then warm, then cold, then deleted. The monthly cost is the integral of bytes-in over time, weighted by the per-byte price of the store each byte lives in.

                  Ingest                 Storage                  Query
                                                       (hot)
                                                       (warm)
                                                       (cold)
                                                       (deleted)
   bytes/sec  -->  ingest_rate  -->  monthly_bytes  -->  monthly_cost
        |              |                    |
        |              |                    +--> bytes retained in hot
        |              +---------------------+--> bytes moving to cold
        +-------------------------------------> query-side compute

For metrics the dominant lever is active series multiplied by samples per second multiplied by retention. Series count is the cardinality term; scrape interval is the sample-rate term.

For logs the dominant lever is lines per second multiplied by bytes per line multiplied by retention. Line length is the unpredictable part: a developer who adds debug logging that includes a JSON blob can triple log volume without anyone noticing.

For traces the dominant lever is spans per second multiplied by bytes per span multiplied by sampling rate multiplied by retention. Sampling is the only knob between request volume and ingest cost.

How to configure the budget

The right unit for a budget is per signal, per team, per lever. A minimal configuration that supports this looks like the following. It is not a software configuration; it is a tracking template a finance-aware operator maintains alongside the platform.

# File: cost-budget.yaml
# Reviewed: every quarter by platform and finance.
# Each block declares a budget envelope. Alert fires at 80 percent
# of the ceiling; the platform cuts off new ingest at 110 percent.

budgets:
  metrics:
    platform_core:
      ingest_rate_ceiling_per_sec: 250000  # active series / 15s scrape
      active_series_ceiling: 3750000      # 250k/s at 15s scrape
      retention_hot_days: 15              # TSDB blocks on local disk
      retention_cold_days: 0              # Prometheus stores locally
      bytes_per_sample_budget: 2.0        # TSDB on-disk budget
    per_team:
      team_alpha:   { series_ceiling: 250000, retention_hot_days: 15 }
      team_bravo:   { series_ceiling: 500000, retention_hot_days: 15 }
      team_charlie: { series_ceiling: 100000, retention_hot_days:  7 }
      default_ceiling: { series_ceiling: 50000, retention_hot_days: 7 }
      overage_action: drop                # not bill-back; cap and drop

  logs:
    platform_core:
      ingest_bytes_per_sec_ceiling: 200000000  # ~190 MB/s sustained
      retention_hot_days: 7                    # ingester + boltdb
      retention_cold_days: 90                 # S3 IA tier
      bytes_per_line_budget: 1024             # warning above this
    per_team:
      team_alpha:   { bytes_per_sec_ceiling: 60000000, retention_hot_days: 7 }
      team_bravo:   { bytes_per_sec_ceiling: 80000000, retention_hot_days: 7 }
      team_charlie: { bytes_per_sec_ceiling: 20000000, retention_hot_days: 3 }
      overage_action: rate_limit_per_tenant   # drop proportional to overshoot

  traces:
    platform_core:
      spans_per_sec_ceiling: 50000            # after edge sampler
      retention_hot_days: 7                   # Tempo block store warm
      retention_cold_days: 30                 # Tempo block store cold
      bytes_per_span_budget: 800              # warning above this
    per_team:
      team_alpha:   { spans_per_sec_ceiling: 20000, sampling_rate: 0.10 }
      team_bravo:   { spans_per_sec_ceiling: 15000, sampling_rate: 0.05 }
      team_charlie: { spans_per_sec_ceiling:  5000, sampling_rate: 0.01 }
      overage_action: drop_newest             # tail-based, not head

The four ceiling types — series, ingest bytes, span count and retention days — are the levers a sysadmin can move without changing application code. Everything else (label cardinality, line length, span attributes) is a developer hygiene problem.

How to validate the cost picture

Validation is the bridge between the cost model and the live platform. The right sequence is per signal, in increasing order of measurement cost.

# Severity: READ-ONLY
# Prometheus: confirm active series count and the top talkers.
curl -s 'http://prometheus:9090/api/v1/query?query=count%20by%20(job)(up%3D%3D1)' \
  | jq '.data.result | sort_by(-.value[1])[:10]'

A runnable output looks like:

[
  {"metric":{"job":"node-exporter"},"value":[1722240000,"3321"]},
  {"metric":{"job":"kube-state-metrics"},"value":[1722240000,"2904"]},
  {"metric":{"job":"app-checkout"},"value":[1722240000,"1188"]},
  ...
]

# Severity: READ-ONLY
# Prometheus: ingest rate from the WAL appended-samples counter.
curl -s 'http://prometheus:9090/api/v1/query?query=rate(prometheus_tsdb_head_samples_appended_total%5B5m%5D)' \
  | jq -r '.data.result[0].value[1]'

illustrative: 87412

# Severity: READ-ONLY
# Loki: per-tenant ingest bytes per second.
logcli --addr=http://loki:3100 instant-query \
  --query='sum by (tenant) (rate(loki_ingester_bytes_received_total[5m]))' \
  --since=1h

{team_alpha="30000000"} {team_bravo="11000000"} {team_charlie="4000000"}

# Severity: READ-ONLY
# Tempo: spans per second and bytes per span from ingester metrics.
tempo-cli query metrics \
  --addr=http://tempo:3100 \
  --query='sum by (service) (rate(tempo_ingester_spans_received_total[5m]))'

The numbers from these four commands should agree with the budget envelope in cost-budget.yaml to within roughly 10 percent. A larger gap is a sign that the budget was set from guesswork, not measurement.

How it can fail

Six shapes repeat in production. Each has a recognisable symptom.

1. Runaway ingest from a deployment. A new release changes the log format and adds a stack trace to every INFO line. Within 24 hours loki_ingester_bytes_received_total is five times the previous level; the S3 bill catches up at month end.
2. Cardinality explosion from a label. An engineer adds a request_id label to every Prometheus exemplar. The series count triples in 12 hours; prometheus_tsdb_head_series hits the memory limit and the WAL starts returning scrape errors.
3. Cross-tenant bleed. The platform team does not enforce per-tenant limits. One team is noisy. Its logging drives the ingester_streams_active for everyone. The platform becomes unfit for the tenants that were within budget.
4. Unbudgeted retention. Loki was sized for seven days. An engineer changed compactor.retention_enabled and set retention_period to 90 days. The hot ingester ran out of disk; cold S3 accumulated three months of unneeded data.
5. Mixed hot and cold in one query. A single Grafana panel spans 30 days. The query planner has to merge cold and hot results and the dashboard tab takes 30 seconds to load. Cost shows up as ingest budget overshoot, not storage.
6. The blank receipt. Finance forwards the vendor bill. The platform has no per-tenant labels anywhere; no team can be identified. The platform team owns the cost by default.

How to troubleshoot runaway cost

The right order is: find the largest contributor, decide whether to keep it, then apply the cheapest control that preserves the signal value.

Symptom (monthly bill grew)
   |
   +-- Per-signal sub-total: which signal grew?
   |     |
   |     +-- Metrics:   active series changed?  scrape interval changed?
   |     |              rules added?  exemplars on?
   |     |              federation in use?
   |     |
   |     +-- Logs:      lines/sec?  bytes per line?  retention?
   |     |              structured vs unstructured?
   |     |              tenant attribution?
   |     |
   |     +-- Traces:    spans/sec?  sampling rate?
   |                    attributes per span?  retention?
   |
   +-- Per-tenant sub-total: which tenant grew?
   |
   +-- Per-lever: ingest, retention, bytes per sample, query
   |
   Decision point
   |-- Stop the bleed (raise a quota, lower sampling)
   |-- Apply a control (drop rule, label relabel, sample rate)
   |-- Documented: cost platform change log
   |
Root cause

If the diagnostic stops at “the bill is bigger,” it has stopped too early. The reason the bill is bigger is one of the four levers on one of the three signals from one of the tenants. Refuse to plan a fix until the diagnostic reaches a tenant.

Security implications

The cost dashboard exposes per-team ingest rates. That is metadata about how busy each team is and what tooling they operate. Treat it as internal: do not publish raw per-tenant ingest volumes to a public dashboard or a vendor-managed multi-tenant dashboard. Authentication on the Prometheus, Loki and Tempo admin endpoints is mandatory for the same reason. Per-tenant limits are a security boundary as well as a cost boundary; a tenant can DoS the platform by design.

Performance implications

Cost is performance asked a different way. The cost equation interlocks with the platform resource budget in three places:

• Hot memory. Active series in Prometheus; chunk entries in the Loki ingester; trace-block metadata in Tempo. Each one is a RAM-resident data structure. Cost ceilings that exceed RAM become OOM incidents.
• WAL and disk write throughput. Each ingest path is a write to a local disk before replication. Cost ceilings that exceed disk throughput become dropped-tail errors and late alerts.
• Compactor and ingester CPU. The compactor rebuilds index blocks; the ingester serves queries. Cost ceilings that exceed CPU budget become slow dashboards. They do not look like a cost problem until someone correlates them with the daily curve.

A budget is therefore a forecast of expected resource use. Tune the platform first, then set the budget ceiling to roughly 70 percent of the measured headroom. The 30 percent gap is operational slack for incident-day ingest.

Production guidance

• Validate the cost model on a sandbox before quoting a ceiling. The numbers converge after two weeks of real traffic.
• Make the budget visible to the tenants whose ingest drives it. Showback — the team sees its number, nobody else does — is cheaper to operate than chargeback and catches the same growth in most organisations.
• Set per-signal sub-totals, not a blended total at the top. A rolled-up total at the bottom of the report is fine.
• Tag everything with a tenant. The platform cost is the sum of the tenant costs; without tenant tags, the platform team is the tenant.
• Re-derive the ceiling once per quarter from measured ingest, not from last year’s budget.

Verification

You should now be able to answer:

• How does the cost equation split into ingest rate, retention and bytes per sample, and which lever dominates each signal?
• Why is a blended monthly total a poor unit for a budget?
• What is the right diagnostic order when a monthly bill grows by 40 percent?
• What is the difference between a showback and a chargeback, and when is each right?
• What four ceilings must a self-hosted Mimir / Loki / Tempo budget declare for it to be enforceable?

Quiz