Tenant Cost Allocation · Observability

The observability platform bill arrives at 41,000 EUR for the month, up from 26,000 the month before. The finance business partner asks which team caused the increase. The platform team can answer “logs went up”, which is not an answer. Two weeks of investigation later the cause is found: one service began logging full HTTP request bodies at info level after a library upgrade, adding 380 GB per day at a 90-day retention.

Per-tenant cost allocation exists so that this question takes ninety seconds. It is not primarily a finance exercise. It is the fastest runaway-telemetry detector you will ever build, because the team that receives the bill is the team that can fix the log line.

What it is

Per-tenant cost allocation is the practice of attributing a measurable share of platform cost to each tenant, using the platform’s own telemetry as the meter. Two delivery models:

Showback - each team sees its cost; no money moves. Behaviour changes because the number is visible and attributable.
Chargeback - the cost is actually billed to the team’s cost centre. Stronger incentive, considerably more argument, and it requires the numbers to be defensible.

Start with showback. Chargeback requires a cost model teams trust, and trust is earned by six months of reports that reconcile against the invoice.

The unit of allocation is the tenant, which is why the tenant model from lesson one is a prerequisite rather than an aside. Cost you cannot attribute to a tenant is cost you cannot attribute to anyone.

Why a sysadmin cares

Runaway telemetry has no natural brake. A debug log line costs its author nothing and costs the platform 12,000 EUR a year. Limits cap the damage; cost visibility stops it recurring.
Capacity planning needs a per-tenant growth curve. An aggregate trend tells you to buy more. Per-tenant trends tell you which conversation to have.
Retention is negotiable when it has a price. “Ninety days” becomes “thirty-one days” remarkably quickly once the delta is on a report with a team name attached.
Federated and expensive queries are cost too. Query cost is usually 15-30% of an observability bill and is invisible without per-tenant query accounting.

How it works

Four cost drivers, each with a metric that meters it.

  DRIVER                 METER                             SHAPE
  ------------------------------------------------------------------
  Logs ingest      loki_distributor_bytes_received_total   GB/month
  Metrics series   cortex_ingester_memory_series           series-mo
  Traces ingest    tempo_distributor_bytes_received_total  GB/month
  Storage          bytes x retention                       GB-month
  Query            querier CPU seconds / bytes processed   share
  ------------------------------------------------------------------
                            |
                            v
        allocate shared platform cost (control plane,
        gateways, Grafana, on-call) pro rata by the
        weighted sum of the above
                            |
                            v
        reconcile total against the cloud invoice
        target: within 5%
                            |
                            v
        monthly report per tenant, with month-on-month delta

The important design decision is storage as GB-month, not GB. A tenant sending 10 GB/day at 31 days retention holds roughly 310 GB of compressed data at steady state; the same tenant at 90 days holds 900 GB. Charging on ingest alone makes long retention free, which is exactly the behaviour you do not want to subsidise.

Under the hood

The meters are ordinary Prometheus metrics exposed by the stack components, and they must be scraped into a tenant that is allowed to hold them - conventionally the platform’s own tenant. Each carries a tenant label, but the label name differs by component, which is the single most common source of broken cost dashboards:

Loki: tenant on loki_distributor_bytes_received_total
Mimir: user on cortex_discarded_samples_total and cortex_ingester_memory_series
Tempo: tenant on tempo_distributor_bytes_received_total

loki_distributor_bytes_received_total counts uncompressed bytes accepted at the distributor, before compression and before replication. Actual object-storage consumption is typically 8 to 12 times smaller after compression, then multiplied by the replication factor for the ingester WAL but not for long-term storage. If you bill on the raw counter without a compression coefficient, your derived total will exceed the invoice by roughly an order of magnitude - which destroys trust in the report immediately.

Derive the coefficient empirically rather than assuming it: divide actual bucket growth over a month by the counter’s increase over the same month, per tenant, because compression ratio depends heavily on log shape. JSON logs compress far better than base64 payloads.

cortex_ingester_memory_series is a gauge and is replicated, so divide by the replication factor before using it as a series count. Averaging that gauge over the month gives series-months, which is the correct unit for metrics storage in a block-based system where each series contributes to every two-hour block.

How to configure it

First, make sure the meters are collected. The platform scrapes its own components into the platform tenant:

# /etc/prometheus/prometheus.yml (excerpt) - the metering scrape
scrape_configs:
  - job_name: loki
    metrics_path: /metrics
    static_configs:
      - targets: ['loki-distributor-1:3100','loki-distributor-2:3100']

  - job_name: mimir
    static_configs:
      - targets: ['mimir-ingester-1:8080','mimir-ingester-2:8080']

remote_write:
  - url: https://metrics.example.internal/api/v1/push
    headers:
      X-Scope-OrgID: team-platform    # metering data belongs to the
                                      # platform tenant, never to a
                                      # customer tenant

Then pre-aggregate. Cost queries over a month are expensive and are run repeatedly; recording rules turn them into cheap lookups:

# /etc/prometheus/rules/tenant-cost.yml
groups:
  - name: tenant-cost
    interval: 5m
    rules:
      # uncompressed log bytes per second, per tenant
      - record: tenant:loki_ingest_bytes:rate5m
        expr: sum by (tenant) (rate(loki_distributor_bytes_received_total[5m]))

      # active series per tenant, corrected for replication factor 3
      - record: tenant:mimir_active_series:avg
        expr: sum by (user) (cortex_ingester_memory_series) / 3

      # trace bytes per second, per tenant
      - record: tenant:tempo_ingest_bytes:rate5m
        expr: sum by (tenant) (rate(tempo_distributor_bytes_received_total[5m]))

      # query load: bytes processed per tenant, a proxy for querier cost
      - record: tenant:loki_query_bytes:rate5m
        expr: sum by (tenant) (rate(loki_querier_bytes_processed_total[5m]))

The price book. Keep it in version control next to the tenant files, with the derivation of each number recorded:

# cost/price-book.yaml
currency: EUR
period: monthly
# derived from the 2026-07 invoice divided by measured volumes;
# review quarterly
rates:
  logs_ingest_per_gb_uncompressed: 0.021   # covers ingest compute
  logs_storage_per_gb_month:       0.024   # object storage + requests
  logs_compression_ratio:          9.4     # MEASURED, not assumed
  metrics_per_1k_series_month:     1.85
  traces_ingest_per_gb:            0.018
  traces_storage_per_gb_month:     0.024
  query_per_tb_processed:          0.90
shared_cost_month: 6400                    # control plane, gateways,
                                           # Grafana, platform on-call
shared_allocation: weighted                # pro rata by direct cost

And the report generator. This is deliberately a small, auditable script rather than a product:

#!/usr/bin/env bash
# READ-ONLY - monthly per-tenant cost report
set -euo pipefail

MIMIR=https://metrics.example.internal/prometheus
MONTH="${1:-$(date -d 'last month' +%Y-%m)}"
START="$(date -d "${MONTH}-01" +%s)"
END="$(date -d "${MONTH}-01 +1 month" +%s)"
SECONDS_IN_MONTH=$(( END - START ))

q() {
  curl -sG "${MIMIR}/api/v1/query" \
    -H 'X-Scope-OrgID: team-platform' \
    --data-urlencode "query=$1" \
    --data-urlencode "time=${END}" \
  | jq -r '.data.result[] | "\(.metric.tenant // .metric.user) \(.value[1])"'
}

# uncompressed GB ingested over the month, per tenant
q "avg_over_time(tenant:loki_ingest_bytes:rate5m[30d]) * ${SECONDS_IN_MONTH} / 1024^3" \
  | sort > /tmp/logs_gb.txt

# average active series over the month, per tenant
q "avg_over_time(tenant:mimir_active_series:avg[30d])" | sort > /tmp/series.txt

join /tmp/logs_gb.txt /tmp/series.txt

team-billing 361.4 41200
team-hr 88.9 9800
team-payments 11842.7 512400
team-platform 4471.2 1103000
team-search 2984.0 88100
team-storefront 8120.6 402700

How to validate it

Validation for a cost model means one thing: does the sum reconcile against the invoice? Everything else is arithmetic.

Compute the derived total:

awk '{ gb+=$2; series+=$3 } END {
  storage_gb = gb / 9.4 * (31/30)          # compressed, ~1 month held
  printf "ingest  %10.2f EUR\n", gb*0.021
  printf "storage %10.2f EUR\n", storage_gb*0.024
  printf "metrics %10.2f EUR\n", series/1000*1.85
}' /tmp/joined.txt

ingest     585.66 EUR
storage    1600.19 EUR
metrics    4136.44 EUR

Compare with the actual bucket size, which is the ground truth for storage:

aws cloudwatch get-metric-statistics \
  --namespace AWS/S3 --metric-name BucketSizeBytes \
  --dimensions Name=BucketName,Value=logs-prod-eu-west-1 \
               Name=StorageType,Value=StandardStorage \
  --start-time 2026-07-31T00:00:00Z --end-time 2026-08-01T00:00:00Z \
  --period 86400 --statistics Average \
  --query 'Datapoints[0].Average' --output text

67284419379200

67.3 TB. If your derived storage figure implies 6 TB, your compression coefficient is wrong by a factor of ten - fix the coefficient, do not adjust the price. Derive it properly per tenant:

for t in team-payments team-hr; do
  bytes=$(aws s3 ls --summarize --recursive \
    "s3://logs-prod-eu-west-1/${t}/" | awk '/Total Size/{print $3}')
  printf '%s stored_gb=%.1f\n' "$t" "$(bc -l <<<"$bytes/1024^3")"
done

team-payments stored_gb=38914.2
team-hr stored_gb=291.7

Divide the month’s uncompressed ingest by the storage delta for the same period to get the tenant’s real ratio.

Finally, sanity-check attribution coverage. Every euro must land on a tenant:

curl -sG https://metrics.example.internal/prometheus/api/v1/query \
  -H 'X-Scope-OrgID: team-platform' \
  --data-urlencode 'query=sum(tenant:loki_ingest_bytes:rate5m) / sum(rate(loki_distributor_bytes_received_total[5m]))' \
  | jq -r '.data.result[0].value[1]'

A value below 1 means some ingest carries no tenant label and is unattributed. Find it before publishing the report.

The monthly report

The report a team can act on is short, comparative, and names the change. Volume without a delta produces no behaviour change.

Observability showback - July 2026
Tenant: team-payments        Owner: payments     Cost centre: CC-2201

  Logs ingest       11,842 GB uncompressed        248.70 EUR
  Logs storage       4,125 GB-month (ratio 9.2)    99.00 EUR
  Metrics          512,400 series-month           948.00 EUR
  Traces               812 GB                      14.60 EUR
  Query                2.4 TB processed             2.16 EUR
  Shared platform  pro rata 18.4%                1,177.60 EUR
  --------------------------------------------------------------
  Total                                          2,490.06 EUR
  Previous month                                  1,905.40 EUR
  Change                                            +30.7%

  Largest contributor to change:
    stream {namespace="pay-prod", app="settlement"}
    +3,900 GB (+49%) from 2026-07-08

  Retention: 90 days (PCI). Reducing to 31 days would
  save approximately 66 EUR/month of storage.

Two properties make this actionable: the change is attributed to a specific stream and a date, and the retention trade-off is quantified. A report that says only “you spent 2,490 EUR” gets filed.

How it can fail

1. Billing on uncompressed bytes without a compression coefficient. Symptom: derived cost is 8-12x the invoice; teams reject the report and never look at it again. Cause: the distributor counter measures what arrived, not what is stored.

2. Wrong tenant label name. Symptom: metrics cost shows zero for every tenant, or one aggregate row with an empty label. Cause: Mimir uses user where Loki uses tenant; a copied dashboard sums by the wrong one.

3. Replication factor double-counting. Symptom: series counts are exactly three times what the tenant expects. Cause: cortex_ingester_memory_series is summed across replicas without dividing by the replication factor.

4. Storage charged as GB rather than GB-month. Symptom: a tenant with 90-day retention pays the same as one with 31-day retention; long retention proliferates because it is free. This is the most consequential modelling error, because it removes the only lever teams actually have.

5. Shared cost allocated per tenant equally. Symptom: team-hr at 89 GB/month receives the same 800 EUR platform share as team-payments at 11.8 TB, escalates, and is right to. Allocate shared cost pro rata by direct cost, and state the method on the report.

6. Counter resets treated as usage. Symptom: an implausible spike on the day a component was restarted or rescheduled. Cause: using increase() over a long range across restarts, or summing raw counters instead of rates. Use avg_over_time of a rate recording rule, as in the rules above.

A tenant disputes their bill

Assume they are right until you have reconciled. Pull the three independent numbers: the distributor counter for the month, the actual object-storage size under that tenant prefix, and the tenant’s own view of what it sent (its agent metrics). If all three agree, walk through the price book with them; if the agent metrics disagree with the distributor counter, you have a metering bug, not a billing dispute.

The most common legitimate dispute is double ingest: an agent misconfiguration sending the same logs twice, or two collectors scraping the same targets. The tenant is billed correctly for what arrived, but the underlying cause is a platform-visible configuration error and the fair resolution is usually to credit the month and fix the pipeline. Keeping the reconciliation script auditable is what makes that conversation short.

How to troubleshoot it

Confirm the meters are being scraped. A gap in the metering series silently produces a low bill:

curl -sG https://metrics.example.internal/prometheus/api/v1/query \
  -H 'X-Scope-OrgID: team-platform' \
  --data-urlencode 'query=count_over_time(tenant:loki_ingest_bytes:rate5m[30d])' \
  | jq -r '.data.result[] | "\(.metric.tenant) \(.value[1])"'

team-payments 8640
team-hr 8640
team-search 3102

8640 samples is a complete month at 5-minute resolution. 3102 means team-search metering was missing for two thirds of the month.

Check for unattributed volume. The coverage ratio above must be
1. If not, find the series without a tenant label.
Recompute the compression ratio for the outlier tenant. A tenant whose derived and actual storage diverge usually has an unusual log shape - binary payloads, or already-compressed content.
Attribute a spike to a stream, not just a tenant. This is the query that answers “which log line”:
```
logcli --org-id=team-payments series '{}' --since=720h \
  --analyze-labels | head -12
```
Then compare volume per stream selector across two weeks to find the one that changed.
Correlate the spike with a change. The date in the report is the input to the tenant’s deployment history. Most cost regressions are a library upgrade or a log-level change, and both are visible in a commit log.
Check for a limits interaction. A tenant that hit 429 for part of the month is billed less than it sent; explain that on the report rather than letting them discover it as an apparent saving.

Security implications

Cost data is not neutral. Three considerations:

Metering data belongs in the platform tenant. Ingest rates and series counts per tenant describe other tenants’ workloads. A tenant should see its own figures, not the whole table - use per-tenant dashboards with a template variable pinned by datasource, not one shared dashboard.
Volume is a side channel. A sharp rise in a security team’s log volume can reveal that an investigation is under way. If that matters in your organisation, exclude sensitive tenants from any broadly visible cost dashboard.
The price book is a policy artefact. Changes to rates change what teams are charged. Review them like code, with the derivation recorded, so nobody has to take a rate on trust.

Performance implications

Cost queries are expensive queries. A month-long avg_over_time over per-tenant rates across fourteen tenants can be one of the heaviest queries on the platform. Pre-aggregate with recording rules and run the report once monthly, off-peak.
Do not run the report against the tenant’s own datasource. Use the platform tenant so that report generation does not consume the query budget of the tenant being measured.
Bucket listing is slow and charged per request. s3 ls --recursive over a 39 TB prefix issues a large number of LIST calls. Prefer CloudWatch BucketSizeBytes or storage inventory reports for routine figures, and reserve recursive listing for reconciliation.

Verification

You should now be able to answer:

Why must loki_distributor_bytes_received_total be divided by a measured compression ratio before it can be priced?
Why is storage charged in GB-month rather than GB, and what behaviour does that change?
Which label carries the tenant on Mimir metrics, and why does it trip up copied dashboards?
How do you correct cortex_ingester_memory_series for the replication factor?
What two elements make a monthly report actionable rather than filed?