Shared vs Dedicated Platform

A 200-engineer engineering organisation runs four Prometheus deployments, six Grafana instances, three Loki clusters, and two Tempo backends. Every team provisioned its own stack two years ago because “shared is risky.” Today, every team has a different label schema, four incompatible dashboard formats, and no way to follow a trace across team boundaries. The reliability of each stack is similar; the coherence of the observability programme is poor. A user-visible failure that spans three teams takes three hours to diagnose because no one can correlate metrics, logs, and traces across the boundaries.

This is what the shared-vs-dedicated decision exists to prevent. The right answer for most organisations is shared platform with multi-tenancy; the wrong answer for almost every organisation is every team gets its own stack. The discipline is to know the trade-offs and to know when to graduate a workload off the shared platform.

What “shared vs dedicated” means

A shared observability platform is a single deployment of Prometheus (or Cortex/Mimir), Loki, Tempo, Grafana, and the OpenTelemetry Collector / Alloy, used by every team. The boundary between teams is logical (tenant ID, RBAC, label filter), enforced by the platform.

A dedicated platform is a separate deployment per team (or per business unit). The boundary between teams is physical (different hosts, different clusters, different storage). Each team owns its stack end-to-end.

The third shape, which appears in practice, is shared infrastructure, dedicated namespaces: the platform team runs the cluster, but each team has its own Grafana, its own recording rules, and its own alertmanager receiver, with no cross-team query path.

Why a sysadmin cares

A shared platform amortises the cost of running, upgrading, and patching Prometheus, Loki, Tempo, and Grafana. One team operates it; everyone benefits. The cost is contention: one team’s high-cardinality scrape can degrade query latency for everyone. The dedicated platform inverts the trade-off: no contention, no amortisation.

The decision is rarely “shared for everyone.” The decision is “shared by default, with a documented exit ramp.” The exit ramp exists for three reasons: compliance (PCI, HIPAA scopes), noisy-neighbour (a single team whose cardinality dominates), and isolation (a tier-0 service that must not share a blast radius with anything else).

How it works

The shared platform is a multi-tenant deployment with three isolation layers:

              +-------------------------------------+
              |       Shared Cluster               |
              |   (one Prometheus / Loki / Tempo)   |
              +-------------------------------------+
                          |       |       |
              +-----------+       |       +-----------+
              |                   |                   |
              v                   v                   v
        +----------+        +----------+        +----------+
        | tenant:  |        | tenant:  |        | tenant:  |
        | payments |        | platform |        | growth   |
        +----------+        +----------+        +----------+
              |                   |                   |
        +-----+-----+       +-----+-----+      +------+------+
        |   |   |   |       |   |   |   |      |   |   |    |
        v   v   v   v       v   v   v   v      v   v   v    v
      API  Logs  Trace     hosts collectors  scrape jobs ...
        |   |   |   |       |   |   |   |      |   |   |
        +<--+---+---+<------+---+---+---+<-----+---+---+-> cross-tenant
                                                       query (gated by RBAC)

Three layers matter:

• Tenant identity. Every metric, log stream, and trace carries a tenant label (the Loki X-Scope-OrgID, the Prometheus external_labels, the OTel resource attribute service.namespace).
• Per-tenant limits. Loki has per-tenant ingestion_rate_mb, max_query_parallelism, and retention_period. Prometheus has per-job scrape limits. Tempo has per-tenant search limits.
• RBAC. Grafana folders, data source permissions, and Alertmanager receivers all map to tenants.

The dedicated platform is the same stack but with a hard boundary: no shared cluster, no shared storage. The two deployment shapes are the same software in different configurations.

The right approach for most organisations is the shared shape with multi-tenancy and the exit ramp documented.

Under the hood: how multi-tenancy is enforced

How to configure it

Loki: enable multi-tenancy and per-tenant limits

# /etc/loki/loki-config.yaml
# CONFIGURATION: enables auth and defines per-tenant overrides.
auth_enabled: true

# Common distributor and querier settings.
distributor:
  receivers:
    otlp:
      protocols:
        grpc: {}

# Per-tenant overrides. The key is the tenant ID as it appears
# in the X-Scope-OrgID header.
limits_config:
  retention_period: 30d
  ingestion_rate_mb: 10
  ingestion_burst_size_mb: 20
  max_query_parallelism: 32
  reject_old_samples: true
  reject_old_samples_max_age: 168h
  creation_grace_period: 10m
  # Per-tenant overrides win over the global defaults above.
  per_tenant_override_config: /etc/loki/overrides.yaml
  # The override file is hot-reloaded; changes take effect
  # without a Loki restart.

# /etc/loki/overrides.yaml
# CONFIGURATION: per-tenant rate limits. The platform team
# tunes this file when a tenant's needs differ from the
# global defaults.
overrides:
  payments:
    ingestion_rate_mb: 50      # payments ships the most logs
    retention_period: 90d
    max_query_parallelism: 64
  platform:
    ingestion_rate_mb: 20
    retention_period: 30d
  growth:
    ingestion_rate_mb: 10      # noisy-neighbour probation
    retention_period: 14d

The per-tenant overrides win. The discipline is: defaults go in limits_config, exceptions go in per_tenant_override_config. A team that needs a higher rate opens a ticket; the platform team edits the override file with a reviewable diff.

Grafana: data source per team

# /etc/grafana/provisioning/datasources/payments.yaml
# CONFIGURATION: each team has its own data sources. The Loki
# datasource sends X-Scope-OrgID: payments on every request,
# isolating the team's queries from other tenants.
apiVersion: 1
datasources:
  - name: 'Loki-payments'
    type: 'loki'
    uid: 'loki-payments'
    url: 'http://loki-gateway:3100'
    access: 'proxy'
    isDefault: false
    jsonData:
      httpHeaderName1: 'X-Scope-OrgID'
    secureJsonData:
      httpHeaderValue1: 'payments'
    # Read-only data source for cross-team dashboards; the
    # payments team sees other tenants only through the
    # cross-tenant gateway, which is logged.
  - name: 'Prometheus-payments'
    type: 'prometheus'
    uid: 'prom-payments'
    url: 'http://prometheus:9090'
    access: 'proxy'
    jsonData:
      httpMethod: 'POST'

The X-Scope-OrgID value is the tenant identity. A team that needs to query another team’s logs does so via a cross-tenant gateway that logs every query; the gateway is a separate Grafana data source with Viewer permission only.

OpenTelemetry Collector / Alloy: pass tenant in resource

# /etc/alloy/config.alloy
# CONFIGURATION: Alloy stamps the tenant onto every span and
# log line before export.
livedebugging { }

otelcol.receiver.otlp "default" {
  grpc { endpoint = "0.0.0.0:4317" }
  http { endpoint = "0.0.0.0:4318" }

  output {
    metrics = [otelcol.processor.batch.default.input]
    logs    = [otelcol.processor.batch.default.input]
    traces  = [otelcol.processor.batch.default.input]
  }
}

otelcol.processor.attributes "tenant" {
  # Inject the tenant from the incoming service.namespace; if
  # missing, route to the default tenant.
  actions = [
    {
      key            = "loki.tenant"
      from_attribute = "service.namespace"
      action         = "insert"
    },
  ]

  output {
    metrics = [otelcol.exporter.otlphttp.grafana.input]
    logs    = [otelcol.exporter.loki.grafana.input]
    traces  = [otelcol.exporter.otlphttp.tempo.input]
  }
}

otelcol.exporter.loki "grafana" {
  forward {
    endpoint {
      url = "http://loki-gateway:3100/loki/api/v1/push"
      headers = {
        "X-Scope-OrgID" = "{ .resource.attributes.loki.tenant }",
      }
    }
  }
}

The collector is the trust boundary. A misconfigured collector that sends all tenants to fake collapses isolation. The discipline is to stamp the tenant at the receiver and never trust application code to label correctly.

How to validate it

Validation is isolation: confirm one tenant cannot read another tenant’s data.

# READ-ONLY. Confirm Loki returns only the tenant's logs.
# The query for tenant=payments must not include logs from
# other tenants even if the label selector matches.
curl -s -H 'X-Scope-OrgID: payments' \
  'http://loki-gateway:3100/loki/api/v1/query?query={service="checkout"}' \
  | jq '.data.result | length'

# READ-ONLY. Confirm the same query from tenant=growth returns
# a different (or empty) set. The two outputs should not
# intersect.
curl -s -H 'X-Scope-OrgID: growth' \
  'http://loki-gateway:3100/loki/api/v1/query?query={service="checkout"}' \
  | jq '.data.result | length'

# READ-ONLY. Confirm per-tenant limits are active. The
# ingestion_rate_mb from the overrides file should appear in
# the /config response.
curl -s -H 'X-Scope-OrgID: payments' \
  http://loki-gateway:3100/config | jq '.limits_config'

# READ-ONLY. Confirm the Grafana folder permissions match the
# tenant. A misconfigured folder permission is a leak.
curl -s -u admin:$GRAFANA_PASS \
  http://grafana:3000/api/folders/team-payments/permissions \
  | jq '.[] | {role, permission}'

Illustrative output of the cross-tenant query test:

$ curl -s -H 'X-Scope-OrgID: payments' \
    'http://loki-gateway:3100/loki/api/v1/query?query={service="checkout"}' \
    | jq '.data.result | length'
1842

$ curl -s -H 'X-Scope-OrgID: growth' \
    'http://loki-gateway:3100/loki/api/v1/query?query={service="checkout"}' \
    | jq '.data.result | length'
0

Tenant payments sees 1,842 log lines for checkout; tenant growth sees zero. The boundary holds.

How it can fail

Five failure shapes appear repeatedly in shared platforms:

1. auth_enabled: false. Loki runs without auth. Every request is treated as tenant fake. Cross-tenant query is trivial. Symptom: one team reads another’s logs by accident.
2. Collector misroutes the tenant. The OTel collector stamps loki.tenant=fake because the application did not send service.namespace. All logs land in the default tenant. Symptom: dashboards return mixed-tenant data.
3. Per-tenant overrides drift from defaults. The override file is edited by hand; the global limits_config is edited by ConfigMap. They diverge; new tenants inherit the older defaults. Symptom: a new team gets a 14-day retention when the policy is 30 days.
4. Noisy neighbour. One team ships 90% of the bytes. Their ingestion pattern saturates the distributor. Other teams see rate-limit errors. Symptom: dashboards show gaps; the loud team sees no degradation.
5. Cross-tenant query path without audit. A Grafana data source with Editor permission on every folder breaks the boundary. Symptom: a Grafana user from team A queries team B’s logs without anyone noticing.

How to troubleshoot it

The diagnostic order for “is the shared platform sharing the right things?”:

1. Confirm auth is enabled. Curl the gateway’s /config endpoint and verify auth_enabled: true. If false, every query below this step is meaningless.
2. Confirm the tenant ID arrives. Curl /loki/api/v1/labels with the tenant header; the response should be the tenant’s labels. If empty, the collector is not stamping.
3. Confirm per-tenant limits apply. Curl /config with the tenant header; the response should include the overrides. If absent, the override file is not loaded.
4. Confirm Grafana permission isolation. Curl the folder permissions; verify the team has the expected role and no others.
5. Confirm cross-tenant query is gated. The cross-tenant gateway should log every query. A query from team A to team B’s logs that does not appear in the gateway log is a leak.
6. Form the diagnosis. Auth off, or collector misroutes, or overrides not loaded, or Grafana RBAC drift, or cross-tenant query without audit. Each is a separate fix.

Security implications

The shared platform concentrates risk. A misconfigured tenant header is a cross-tenant data leak. A misconfigured Grafana permission is an exposure of every team’s dashboards. The discipline:

• Tenant header — set by the gateway, never by the client. The data source provisioning sets the header server- side; the user cannot override it.
• Audit log — the gateway logs every query with the requesting user and the target tenant. A daily review confirms no cross-tenant queries are happening without a ticket.
• Cross-tenant gateway — a separate Grafana data source with Viewer permission and audit logging. Editor permission on cross-tenant is forbidden.
• Retention — per-tenant retention prevents a noisy team from filling the shared object store.

Performance implications

The shared platform has predictable performance: ingestion rate per tenant, query parallelism per tenant, retention per tenant. The dedicated platform has independent performance: one team’s cardinality does not affect another. The trade-off:

Most organisations run shared for ~80% of teams, dedicated for the remaining 20% (compliance, tier-0, noisy). The exit ramp is a documented process, not a one-off project.

Production guidance

• Default to shared. The cost of every team running its own stack is paid forever in operators, upgrades, and incoherence.
• Configure per-tenant limits before adding the second team. Adding limits after the third noisy neighbour has shipped is an emergency.
• Validate isolation monthly. A cross-tenant query test that returns zero for one tenant and non-zero for another is the canonical validation.
• Document the exit ramp. The conditions for graduating a team to dedicated (cardinality threshold, compliance scope, tier-0 classification) belong in the platform policy, not in a one-off email.
• Audit Grafana folder permissions quarterly. The permission drift between Grafana and Alertmanager is the most common boundary leak.

Verification

You should now be able to answer:

• What is the difference between shared multi-tenant and per-team dedicated observability?
• Where does the tenant boundary live in Loki, Prometheus, Tempo, Grafana, and the OTel Collector?
• What is the failure shape when auth_enabled is false?
• When should a workload graduate from shared to dedicated?
• How do you validate that one tenant cannot read another’s data?

Quiz