Cost Monitoring

The platform team had a cost dashboard. It was six panels and a monthly review cadence. The cost grew 30 percent over the previous quarter and the dashboard showed it on day 28. The alert never fired. The team had the data; they did not have the cadence. Cost monitoring is the difference between a number on a panel and a discipline that catches drift before the invoice arrives.

What cost monitoring is

Cost monitoring is the discipline of measuring the cost-model inputs (ingest rate, retention, bytes per sample, rejection counters) on a dashboard that an operator looks at before the invoice arrives. It has three parts:

• Leading indicators. Per-signal per-tenant ingest counters and rejections. Updated every minute. Show drift a month before the invoice.
• Trend dashboards. Per-signal weekly ingest aggregated to the tenant. Show whether growth is structural or one-off.
• Finance review. A monthly report that totals per-signal per-tenant ingest against the budget envelope. Show whether the budget still holds.

The three parts feed each other. A leading indicator panel that has been climbing for three weeks is a forecast for the finance review. A finance review that broke the budget last month is the justification for tightening the leading-indicator thresholds.

Why a sysadmin cares

Three operational pains recur.

1. The dashboard exists, but no one looks. A cost dashboard was built at the last quarter’s review. Nobody owns it. Cost grows 20 percent; nobody notices; the invoice catches the team by surprise.
2. The alert that never fires. A leading indicator alert was configured at a threshold above any plausible reading. The alert has never fired in production. By the time cost growth is visible, it is already at the rejection threshold.
3. The tenant attribution gap. The finance team asks which team drove the cost growth. The dashboard has a single blended total. Nobody can point to a tenant without running a new query.

The discipline the lesson teaches: every cost dashboard has an owner, a cadence and an alert. The owner looks at the dashboard on the cadence. The alert fires before drift becomes rejection.

How cost monitoring works

The mental model is three dashboards and three cadences. Each dashboard feeds a different decision.

                 ingest counter      rejection counter
                 (leading)           (lagging)
                     |                    |
                     v                    v
   daily cadence --+----- cost dashboard --------+-- weekly trend
                     |                            |
                     v                            v
                  forecast to month end         structural or one-off?
                     |                            |
                     v                            v
                  finance review (monthly) ---> budget re-derive

The flow runs every day. The leading indicator is sampled at 60 s; the projection recalculates on the same cadence. The weekly trend rolls up the leading indicator to a week total. The monthly review compares the monthly total to the budget envelope.

   per-tenant ingest counter
        |
        v
   rate() -- 60s window -- 24h window -- 7d window
                          |
                          v
                          forecast to month end
                                 |
                                 v
                          alert fires? (80 percent of budget)
                                 |
                                 +-- yes: review quota, raise or
                                 |       reset the offending tenant
                                 +-- no:  log; carry forward

How to configure cost monitoring

A working cost-monitoring setup has three dashboards and three alerts. The dashboards are defined as Grafana JSON in source control; the alerts are Prometheus alerting rules.

# File: alerts/cost-mon.yaml
# Severity: CONFIGURATION (Prometheus reload required)
# Per-tenant and per-signal cost alerts. Three thresholds per
# tenant: leading, lagging, capacity.

groups:
  - name: cost-mon.metrics
    interval: 60s
    rules:
      - alert: MetricsTenantOverBudgetForecast
        expr: |
          ( sum by (tenant) (rate(cortex_distributor_samples_in_total[24h])) * 86400 * 30 )
            >
          label_replace(vector(0), 'tenant', '', '', '')
            * on() group_left() label_replace(
                ( cortex_tenant_limits{limit_name="ingestion_rate"} * 86400 * 30 ),
                'tenant', '$1', 'tenant', '(.*)'
              )
        for: 6h
        labels:
          severity: cost
        annotations:
          summary: 'Tenant {{ $labels.tenant }} projected to exceed metrics ingest budget'
          description: |
            Per-tenant sample ingest rate over the last 24h projects
            to more than the configured budget envelope. Raise the
            quota or inspect the largest contributor.

      - alert: MetricsTenantRejected
        expr: |
          sum by (tenant) (rate(cortex_distributor_samples_dropped_total[15m])) > 0
        for: 30m
        labels:
          severity: cost
        annotations:
          summary: 'Tenant {{ $labels.tenant }} exceeded the metrics ingest limit'
          description: |
            The metrics distributor rejected samples in the last 30
            minutes for tenant. The limit is firing; data is being
            lost. Investigate.

  - name: cost-mon.logs
    interval: 60s
    rules:
      - alert: LogsTenantOverBudgetForecast
        expr: |
          ( sum by (tenant) (rate(loki_ingester_bytes_received_total[24h])) * 86400 * 30 )
            > on(tenant) group_left
              ( loki_tenant_limits{limit_name="ingestion_rate_mb"} * 1048576 * 86400 * 30 )
        for: 6h
        labels:
          severity: cost
        annotations:
          summary: 'Tenant {{ $labels.tenant }} projected to exceed log ingest budget'
          description: |
            Per-tenant byte ingest over 24h projects to more than
            the configured envelope.

      - alert: LogsTenantRejected
        expr: |
          sum by (tenant) (rate(loki_distributor_requests_total{status_code="429"}[15m])) > 0
        for: 30m
        labels:
          severity: cost
        annotations:
          summary: 'Tenant {{ $labels.tenant }} rejected by Loki distributor'

  - name: cost-mon.traces
    interval: 60s
    rules:
      - alert: TracesTenantOverBudgetForecast
        expr: |
          ( sum by (tenant) (rate(otelcol_receiver_accepted_spans[24h])) * 86400 * 30 )
            > 30000000          # 30 M spans per day ceiling per tenant
        for: 6h
        labels:
          severity: cost
        annotations:
          summary: 'Tenant {{ $labels.tenant }} projected to exceed trace ingest budget'

The companion dashboard JSON has three rows.

{
  "title": "Telemetry Cost Dashboard",
  "uid": "telemetry-cost",
  "schemaVersion": 38,
  "rows": [
    {
      "title": "Leading indicators — per signal",
      "panels": [
        {
          "type": "timeseries",
          "title": "Metrics ingest rate (samples/sec) by tenant",
          "targets": [
            {"expr": "sum by (tenant) (rate(cortex_distributor_samples_in_total[5m]))"}
          ]
        },
        {
          "type": "timeseries",
          "title": "Log ingest rate (bytes/sec) by tenant",
          "targets": [
            {"expr": "sum by (tenant) (rate(loki_ingester_bytes_received_total[5m]))"}
          ]
        },
        {
          "type": "timeseries",
          "title": "Trace ingest rate (spans/sec) by tenant",
          "targets": [
            {"expr": "sum by (tenant) (rate(otelcol_receiver_accepted_spans[5m]))"}
          ]
        }
      ]
    },
    {
      "title": "Lagging indicators — limits firing",
      "panels": [
        {
          "type": "timeseries",
          "title": "Metrics rejections by tenant",
          "targets": [
            {"expr": "sum by (tenant) (rate(cortex_distributor_samples_dropped_total[5m]))"}
          ]
        },
        {
          "type": "timeseries",
          "title": "Log distributor 429 by tenant",
          "targets": [
            {"expr": "sum by (tenant) (rate(loki_distributor_requests_total{status_code=\"429\"}[5m]))"}
          ]
        }
      ]
    },
    {
      "title": "Trend — projected to month end",
      "panels": [
        {
          "type": "stat",
          "title": "Projected metrics ingest (30d)",
          "targets": [
            {"expr": "sum by (tenant) (rate(cortex_distributor_samples_in_total[24h])) * 86400 * 30"}
          ]
        },
        {
          "type": "stat",
          "title": "Projected log ingest bytes (30d)",
          "targets": [
            {"expr": "sum by (tenant) (rate(loki_ingester_bytes_received_total[24h])) * 86400 * 30"}
          ]
        }
      ]
    }
  ]
}

The shape is consistent: leading indicators in the first row, lagging in the second, projection in the third. Finance pulls the projection row on the monthly cadence; the on-call team looks at the leading-indicator row daily.

How to validate cost monitoring

# Severity: READ-ONLY
# Per-tenant samples ingested per second (the leading indicator).
curl -s 'http://mimir:9009/prometheus/api/v1/query?query=sum%20by%20(tenant)%20(rate(cortex_distributor_samples_in_total%5B5m%5D))' \
  | jq '.data.result'

illustrative:
[{"metric":{"tenant":"team_alpha"},"value":[1,"28000"]},
 {"metric":{"tenant":"team_bravo"},"value":[1,"12000"]}]

# Severity: READ-ONLY
# Confirm the alert rules loaded. A reload that failed to apply
# the rules is silent; the cost dashboard will not warn.
promtool check rules /etc/prometheus/rules/cost-mon.yaml

illustrative:
Checking /etc/prometheus/rules/cost-mon.yaml
  SUCCESS: 6 rules found

# Severity: READ-ONLY
# Confirm the dashboard JSON parses (Grafana refuses a panel
# with a typo; the dashboard is loaded with red error labels).
curl -s -X POST 'http://grafana:3000/api/dashboards/validate' \
  -u admin:admin \
  -H 'Content-Type: application/json' \
  -d @telemetry-cost.json \
  | jq .

illustrative:
{"message":"Dashboard is valid","status":"success"}

# Severity: READ-ONLY
# Reject counter for the largest tenant, in the last 1h.
logcli instant-query \
  --addr=http://loki:3100 \
  --query='topk(5, sum by (tenant) (rate(loki_distributor_requests_total{status_code="429"}[1h])))'

illustrative:
{team_alpha="0"} {team_bravo="2"}

The validation passes when the dashboard is loaded, the alert rules are present and the leading indicators are sampled at the right cadence.

How it can fail

Six shapes repeat.

1. The dashboard with no owner. A dashboard is built during a quarterly cost review. The team rotates; nobody owns the dashboard. Drift accumulates. The first sign is the invoice.
2. The alert with a too-high threshold. A leading-indicator alert was set at 200 percent of any plausible read. The alert has never fired. Drift runs past the rejection threshold before anyone notices.
3. The panel built from the wrong counter. A panel graphs received_total summed across the platform; the dropped_total counter is the lagging indicator and is not on the dashboard. The team sees ingest rising; rejections silently rise with it. Both surfaces are reachable from the same data; only one is in the dashboard.
4. The dashboard imported but not exported. A Grafana dashboard is imported via the API. The source of truth in Git is forgotten. Three quarters later the dashboard is out of date with the latest rule file.
5. The retention override without a corresponding panel. The retention override YAML grows from 5 lines to 50. No panel exposes the per-tenant retention. The finance report shows what the system does, not what it is configured to do.
6. The tenant label inconsistency. Mimir uses user as the label; Loki uses tenant; Tempo uses org_id. The dashboard builder has to map three label conventions. One joinquery later, the dashboard hides the data the team needs.

How to troubleshoot cost monitoring

The diagnostic order is: panel, alert, ownership.

Symptom (cost grew; dashboard missed it)
   |
   +-- Panel loaded?  alert wired?  owner assigned?
   |
   +-- Panel: is the leading indicator moving? at the right
   |          cadence?  with the right tenant label?
   |
   +-- Alert: is the threshold reachable?  did the alert rule
   |          fail to load after a Prometheus reload?
   |
   +-- Owner: does anyone look at this dashboard on cadence?
   |          if not, set up the cadence first, the panel
   |          second
   |
   +-- Decide: fix the panel, retune the alert, or reassign
   |          ownership
   |
   +-- Document: cost platform change log
   |
Root cause

The most common shape is “the dashboard exists; the alert does not fire; nobody owns it.” Treat ownership as a configuration value: every dashboard JSON has a team: label and a rotation calendar attached; every alert rule has an oncall_team: label.

Security implications

The cost dashboard exposes per-tenant ingest and retention. That is metadata about how busy each team is and what tooling they operate. Treat the dashboard as internal: do not publish raw per-tenant ingest volumes to a public URL or to a vendor- managed multi-tenant dashboard. Authentication on the Grafana endpoint is mandatory. RBAC on the dashboard matters: a developer who can see every team’s ingest can reverse-engineer their traffic patterns.

The Prometheus / Loki / Tempo admin endpoints that publish the counters must be authenticated; an unauthenticated /metrics endpoint can leak per-tenant ingest to anyone with network access.

Performance implications

The cost dashboard itself is a small consumer of the platform. Six PromQL queries against the metrics plane; three LogQL queries against Loki; one OTel metric query against the collector. Each query is cheap until the platform grows: at 50 tenants, the join between cortex_distributor_samples_in_total and cortex_tenant_limits becomes the slow part. The cost dashboard is fine until it is not.

PromQL rule evaluation is the second cost. The MetricsTenantOverBudgetForecast rule runs every 60 s; with 200 tenants the evaluation is roughly 5 000 series. A dashboard with double the cadence and double the tenants is 20 000 series per evaluation; tune the cadence down before adding tenants.

Production guidance

• Tie every cost dashboard to an owner. A dashboard without an owner is a dashboard without a review.
• Set the leading-indicator alert at 80 percent of the budget envelope, not at 100 percent. The 20 percent gap is the time the team has to react.
• Run the projection rule on a 24 h, not a 5 min, window. A 5 min projection is too noisy; a 24 h projection is the credible cost forecast.
• Review per-tenant ingest weekly. Catch the 10 percent drift before it is a 30 percent drift.
• Export the projection panel as a CSV monthly for finance. The CSV is the contract between the platform and the budget.

Verification

You should now be able to answer:

• What are the three parts of a cost monitoring discipline, and what cadence owns each?
• Why is a per-tenant ingest counter a leading indicator and a rejection counter a lagging indicator?
• Which Prometheus / Loki / OTel counters are the canonical sources for the leading-indicator panels?
• What is the most common shape of cost monitoring failure and how does it manifest?
• Why does an 80 percent alert threshold on the projected forecast leave the team time to act, where a 100 percent threshold does not?

Quiz