Cost Controls

The cost-discipline lessons so far have each explained a single lever: cardinality on metrics, line hygiene on logs, sampling on traces. Each lever is necessary but not sufficient. A platform team that controls only one of them is one noisy service away from its budget. This lesson assembles those levers into a coherent control surface: the per-tenant, per-signal, per-tier budget the operator actually runs.

What cost controls are

Cost controls are the operator-side knobs that bound ingest, storage and query cost without changing application code. They fall into four families, mapped onto the three signals.

• Ingest rate limit. Hard ceiling on samples per second (metrics), bytes per second (logs) or spans per second (traces) per tenant. Enforced at the distributor.
• Retention. Hard ceiling on how long data is kept. Hot and cold tiers. Enforced at the compactor and at the index store.
• Cardinality ceiling. Hard ceiling on active series per tenant (metrics) or stream count per tenant (logs). Enforced at the distributor and at the receiver.
• Query budget. Hard ceiling on query-side compute (read bytes, range, parallelism). Enforced at the query frontend.

The controls compose. A tenant’s ingestion is bounded by its ingest rate limit; its storage is bounded by retention times ingest; its query cost is bounded by its query budget. When one control fires, the others absorb the consequence: a tenant whose ingest limit fires does not consume storage; a tenant whose retention fires does not consume ingest.

Why a sysadmin cares

Three operational pains the control surface prevents.

1. The tier mismatch. A development team running a load test shares a Loki tenant with production services. The load test pegs 80 percent of the ingester memory. Production logs start to fail.
2. The silent retirement. An engineer lowers retention for a single tenant to chase a post-mortem. Three quarters later the retention override has not been reverted. Cold storage is 12 times larger than the budget.
3. The uneven budget. Five teams share one ingest budget; one team uses 80 percent. The other four have headroom they cannot use. Quota is the only fair shape.

In each case the same sequence applies: identify the tenant, identify the lever, apply the control at the right tier.

How the control surface works

The mental model is three concentric rings. The innermost ring is the platform-wide ceiling. The middle ring is the per-tenant quota. The outermost ring is the per-signal override.

   platform-wide ceiling  -- the absolute upper bound; enforced
   |                         even if every tenant wants more
   |
   +-- per-tenant quota   -- the per-signal envelope; the budget
   |                         the team is told it has
   |
   +-- per-signal override -- the per-stream or per-route
                              override; an exception for one team
                              or one workspace

A request to change a control flows outward from the centre: a team requests an override, the platform approves, the override is recorded, and the per-tenant quota is rebalanced. A failure flows inward: a noise incident triggers a per-tenant adjustment before the platform-wide ceiling is touched.

   per-signal override       (per stream, per route, per workspace)
        |
        v
   per-tenant quota          (per signal: ingest rate, retention)
        |
        v
   platform-wide ceiling     (the absolute bound)
        |
        v
   the actual signal         (collected; rate-limited; retention-applied)

How to configure cost controls

A working cost-control configuration has three tiers: dev, staging and production. Each tier declares its budget and the overage action. The example below is Mimir / Cortex + Grafana Loki + Grafana Tempo with per-tenant overrides.

# File: mimir-distributor.yaml
# Severity: CONFIGURATION (reload required)
# Mimir / Cortex distributor: per-tenant metric limits.

limits:
  # Default tenant (no header) ceiling.
  ingestion_rate: 10000
  ingestion_burst_size: 20000
  max_global_series_per_metric: 50000
  max_global_series_per_user: 500000

  # Per-tenant overrides. A team whose ingest is known to be
  # higher gets a higher ceiling without changing the default.
  per_tenant_limits:
    team_alpha:    { ingestion_rate: 30000, max_global_series_per_user: 3000000 }
    team_bravo:    { ingestion_rate: 20000, max_global_series_per_user: 2000000 }
    team_charlie:  { ingestion_rate:  5000, max_global_series_per_user:  500000 }

# File: loki-config.yaml
# Severity: CONFIGURATION (SIGHUP for limits_config; restart for schema_config)

limits_config:
  ingestion_rate_mb: 5                       # default per-tenant
  ingestion_burst_size_mb: 10
  reject_old_samples: true
  reject_old_samples_max_age: 168h           # 7 days
  retention_period: 604800                   # 7 days default (seconds)

compactor:
  retention_enabled: true
  working_directory: /loki/compactor
  delete_request_store: s3

# Per-tenant retention overrides: this is the per-stream shape
# Grafana recommends. Mimir / Loki 3.x supports the same on the
# ruler side via overrides in storage_config.
overrides:
  team_alpha:
    ingestion_rate_mb: 30
    retention_period: 2592000                # 30 days for team_alpha
  team_bravo:
    ingestion_rate_mb: 20
    retention_period: 1209600                # 14 days for team_bravo
  team_charlie:
    ingestion_rate_mb: 5
    retention_period: 432000                 # 5 days for team_charlie

schema_config:
  configs:
    - from: "2026-01-01"
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h

# File: tempo-config.yaml
# Severity: CONFIGURATION (reload required)

server:
  http_listen_port: 3100

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: tempo:4317

# Per-tenant overrides for span-per-second ingestion. Tempo 2.x
# applies these in the metrics-generator and distributor layers.
ingestion:
  max_request_bytes: 10485760           # 10 MiB request cap

overrides:
  per_tenant_overrides:
    team_alpha:   { ingestion_max_spans_per_second: 20000 }
    team_bravo:   { ingestion_max_spans_per_second: 15000 }
    team_charlie: { ingestion_max_spans_per_second:  5000 }

storage:
  trace:
    backend: s3
    s3:
      bucket: tempo-traces
    block:
      version: vparquet
      flush_size: 524288               # 512 KiB
      flush_age: 1h
  wal:
    path: /var/tempo/wal

# Per-tenant retention lives in the compactor on the storage layer.
compactor:
  compaction:
    block_retention: 168h              # 7 days hot
    compacted_block_retention: 720h    # 30 days cold

The configuration choices that do the work:

• per_tenant_overrides declares the budget envelope per team. Without it, every tenant shares the platform default.
• retention_period per tenant recognises that not every team needs a 30-day window. The default of 7 days applies to unmapped tenants; the override for team_alpha raises it.
• block_retention and compacted_block_retention are the Tempo equivalent. Hot and cold are independent ceilings.

The Prometheus-side cardinality ceiling is on the scrape, not on the distributor:

# File: /etc/prometheus/prometheus.yml
# Severity: CONFIGURATION (SIGHUP)

scrape_configs:
  - job_name: app
    sample_limit: 2000                  # refuse overflow per scrape
    target_limit: 200
    metric_relabel_configs:
      - regex: 'request_id|session_id'
        action: labeldrop
      - regex: 'node_.*'
        action: keep

sample_limit is a hard cardinality ceiling at the per-scrape level. Combined with metric_relabel_configs, it bounds the cardinality that ever reaches the head.

How to validate cost controls

Three queries together verify that the controls are enforcing as designed.

# Severity: READ-ONLY
# Per-tenant rejected samples (Mimir / Cortex).
curl -s 'http://mimir:9009/api/v1/status/limits' \
  | jq '.limits | to_entries | sort_by(-.value.ingestion_rate)[:10]'

illustrative: limits per tenant with ingestion_rate values, sorted.

# Severity: READ-ONLY
# Loki ingester rejections by tenant in the last hour.
logcli instant-query \
  --addr=http://loki:3100 \
  --query='topk(10, sum by (tenant) (rate(loki_distributor_requests_total{status_code="429"}[1h])))'

illustrative:
{team_alpha="12"} {team_bravo="3"} {team_charlie="0"}

# Severity: READ-ONLY
# Tempo: per-tenant span-cap rejections (the otel collector
# tail-sampler attributes each dropped span to its tenant).
curl -s 'http://otelcol:8888/metrics' \
  | grep -E '^otelcol_rejected_(spans|log_records)_total' \
  | grep -v "^#"

illustrative:
otelcol_rejected_spans_total{...}="43000"

The validation passes when the rejection counter is non-zero for the tenants who are over budget (the controls are firing) and zero for the tenants who are within budget (the controls are not firing unnecessarily).

# Severity: READ-ONLY
# Confirm that retention compactor is honoring per-tenant override.
curl -s 'http://loki:3100/config' \
  | jq '.limits_config.retention_period,
        (.overrides | to_entries
                     | sort_by(-.value.retention_period))'

illustrative:
604800
{team_alpha:2592000, team_bravo:1209600, team_charlie:432000}

How it can fail

Six shapes repeat.

1. The default tenant. A new pipeline stamps tenant="" for unmapped traffic. The default tenant has the platform default; lines that should have been on team_alpha end up on nobody’s budget.
2. The override that outlived its purpose. A 30-day retention override added for a post-mortem stays for two quarters. The budget report flags the tenant only when the override is read.
3. The ingest limit without a backoff. A tenant hits its ingestion_rate and the agent retries without backoff. The distributor spends more CPU refusing than serving. A single noisy tenant can starve the distributor for the rest.
4. The cardinality ceiling on the wrong job. A label-drop rule on node_exporter is fine. The same rule on a job whose labels an alert depends on turns into a “no data” alert.
5. The retention on the wrong stream. A compactor retention override is on the stream name; the actual stream writes under a different label. Compactor never matches; nothing expires.
6. The query budget without a timeout. A Grafana panel issues a multi-day range query. The query frontend has a query budget but the panel times out at the panel level before the backend reports the rejection. Cost shows up as a “panel timeout” instead of “rejection.”

How to troubleshoot cost controls

The diagnostic order is: observe, attribute, recalibrate.

Symptom (cost over budget; or data missing)
   |
   +-- Which tenant is over?  which is missing?
   |
   +-- Rejection counter: is the limit firing?
   |     |
   |     +-- Yes, with traffic: ceiling is correct; team needs
   |     |                     a quota raise
   |     +-- Yes, no traffic:  pipeline is mis-stamping tenant
   |     +-- No, with traffic: ceiling is too high; data is
   |                          being kept that should be dropped
   |
   +-- Retention compactor: is it running?
   |     |
   |     +-- Yes, with delay: per-stream override is wrong
   |     +-- No:  compactor is paused or storage full
   |
   +-- Cardinality ceiling: is sample_limit firing?
   +-- Query budget: is the panel timing out?
   |
   +-- Decide: tighten a control, raise an override, or fix the
   |          pipeline stamp
   |
   +-- Document: cost platform change log
   |
Root cause

The three controls (ingest, retention, ceiling) interact. A tenant whose ingest limit fires does not consume retention; a tenant whose retention fires does not consume ingest. Diagnose in that order.

Security implications

Per-tenant isolation is the security boundary. A tenant whose ingest limit is too high can deny service to other tenants. A tenant whose retention is too long can hold data the team should not retain. A tenant whose cardinality ceiling is too high can deny service to the metrics plane. Treat the per- tenant override YAML the way an RBAC layer is treated: every change is a privilege change, reviewed and recorded.

Authentication on the distributor and on the OTLP / Loki HTTP ingest path is mandatory; the cost-control surface has no meaning without it.

Performance implications

Controls are themselves a performance cost. The distributor evaluates the per-tenant token bucket on every ingest request. At 100 000 req/s with 200 tenants, the per-request overhead is visible. The query-frontend parallelism budget on Loki costs query latency. The cardinality ceiling on Prometheus is free until it fires; when it fires, the scrape returns errors that cost the agent’s retry budget. A control surface must be sized for the traffic shape of the platform, not just for the budget.

Tune the platform first, then set the controls to roughly 70 percent of measured headroom. The 30 percent gap is what makes the controls quiet on a calm day.

Production guidance

• Treat the cost-control YAML the way a code review treats permissions. Every change is reviewed and recorded. The control surface is the platform’s RBAC for telemetry.
• Declare per-tier budgets: dev / staging / production. The dev tier can have aggressive defaults; the production tier must enforce hard ceilings.
• Export per-tenant rejections to the cost dashboard. The whole point of a control is to fire visibly; firing invisibly is the failure mode.
• Re-derive the per-tenant quota once per quarter from measured ingest. Quotas derived six months ago are fictional.
• Record per-stream retention overrides in a runbook. An override without a runbook is a slow leak; an override with a runbook is a controlled exception.

Verification

You should now be able to answer:

• What are the four canonical control surfaces and which signal does each map to?
• Why is a per-tier cost policy (dev / staging / production) more durable than a single platform-wide budget?
• How does the Mimir distributor enforce per-tenant limits, and where does Loki enforce the same shape?
• How do you validate that a control is firing on the right tenant and on the right metric?
• What is the right diagnostic order when one tenant is over budget and another is missing data?

Quiz