Grafana HA Cost

A team stands up two Grafana replicas behind a load balancer. Six months later, the on-call engineer is paged for a Grafana outage. The database is unreachable. Every replica is down. The team has spent $250/month on the second replica and $200/month on the load balancer. The private Postgres bill is $150/month. The total monthly cost is $600. The total downtime prevented in the last six months is roughly 11 minutes (one kernel upgrade reboot). The cost-per-minute-of-prevented-downtime is $54. The team did not run a failover test. The HA was a label, not a property. The lesson is that HA is not free, and the cost matters more than the marketing.

The cost of HA is the operational overlay on top of the Grafana core. The cost includes the second (or more) compute instance, the shared database, the load balancer, the operational complexity of keeping N replicas in sync, and the human cost of more things to upgrade. The right tier is the cheapest shape that meets the actual need. The wrong tier is the most expensive shape the team can afford.

What it is

The cost of Grafana HA is the sum of the resources required to run the chosen tier. The three common tiers:

• Tier 1: single Grafana + sqlite. One replica. sqlite database. No load balancer. Cost: one Grafana image, one PersistentVolume for the database. Acceptable for small teams and development environments.
• Tier 2: two Grafana replicas + shared Postgres. Two replicas behind a layer-7 load balancer. Postgres in the same VPC. Cost: two Grafana compute, one load balancer, one Postgres (managed or self-hosted). The right answer for most production observability stacks.
• Tier 3: N Grafana replicas + HA Postgres (Multi-AZ). N replicas behind a load balancer with autoscaling. HA Postgres with replicas. Cost: substantial. Reserved for regulated industries or multi-region deployments.

The cost is not the surface number. The cost includes the operational complexity of keeping N replicas in sync, the postgres HA story, the load balancer HA story, the upgrade plumbing, the test procedure, and the on-call rotation that knows how to recover.

   Tier 1                      Tier 2                      Tier 3
   +-----------+               +-------+-------+           +-----------+
   | Grafana   |               | g1   |   g2  |           | g1 g2 g3  |
   | + sqlite  |               +-------+-------+           +-----+-----+
   +-----------+               | shared Postgres|          | HA Postgres|
                               +----------------+          +------------+
                                                                     active-active
   cost: ~$30/mo               cost: ~$250/mo               cost: ~$1500+/mo
   recovery: 30 min            recovery: 5 min              recovery: 1 min
   ops: 1 person               ops: 1 person                ops: 2 persons

Why a sysadmin cares

The four operational pains that disappear once the tier is correctly sized:

1. Premature HA. A team stands up Tier 3 because the manager read a Gartner report. The cost is $1500/month for a stack that serves 30 users. The complexity is large for the operational maturity of the team.
2. Insufficient HA. A team keeps Tier 1 for a stack that serves 500 users with production alert rules. The first time the host reboots for a kernel upgrade, the on-call engineer is paged for the alert gap. The cost is one incident per quarter.
3. HA load balancer becomes the new single point of failure. The team stands up Tier 2 with one HAProxy. The HAProxy dies. Every replica is unreachable. The cost is the HAProxy HA story.
4. The database is the bottleneck nobody measured. Tier 2 uses one Postgres. The dashboards are slow at 14:00 every day. The Grafana replicas are at 30% CPU. The Postgres is at 95% CPU. The cost is the assessment that Grafana HA needs more replicas, when the actual fix is a Postgres upgrade.

How it works

The cost scales with the tier. The right tier is the one that matches the operational reality of the team’s load and maturity, not the one that matches the most ambitious design.

A useful heuristic for the right tier:

• One Prometheus data source, one Loki data source, one Tempo data source, fewer than 50 users, less than 10 alerts is Tier 1 territory. A single Grafana on a 4 GB VPS with sqlite handles this comfortably.
• Multiple data sources, hundreds of users, more than 10 alerts, production alert rules is Tier 2 territory. Two replicas behind a load balancer with a managed Postgres is the right answer.
• Multi-region, multi-tenant, hundreds of users, regulated alerting requirements is Tier 3 territory. Active-active with database replicas and a CDN is the right answer.

The cost calculation is small. A Grafana process is 1 CPU, 1 GB RAM. Two replicas is 2 CPU, 2 GB RAM. The shared Postgres is the line item that grows: a 4-vCPU managed RDS Postgres is roughly $200/month, and the Multi-AZ version is roughly $400/month. The load balancer is another $20-50/month depending on the provider.

How to configure it

The reference Tier 2 deployment in Kubernetes:

# Tier 2 reference: two Grafana replicas, shared Postgres
apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana
  namespace: observability
spec:
  replicas: 2
  selector:
    matchLabels:
      app: grafana
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app: grafana
    spec:
      containers:
        - name: grafana
          image: grafana/grafana:11.2.0
          ports:
            - containerPort: 3000
          env:
            - name: GF_DATABASE_TYPE
              value: postgres
            - name: GF_DATABASE_HOST
              value: postgres.observability.svc:5432
            - name: GF_DATABASE_NAME
              value: grafana
            - name: GF_DATABASE_USER
              value: grafana
            - name: GF_DATABASE_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: grafana-db
                  key: password
            - name: GF_DATABASE_SSL_MODE
              value: require
            - name: GF_DATABASE_MAX_CONN
              value: "20"
            - name: GF_SECURITY_SECRET_KEY
              valueFrom:
                secretKeyRef:
                  name: grafana-secret
                  key: secret_key
          resources:
            requests:
              cpu: 200m
              memory: 256Mi
            limits:
              cpu: 1000m
              memory: 1Gi
          readinessProbe:
            httpGet:
              path: /api/health
              port: 3000
            initialDelaySeconds: 10
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /api/health
              port: 3000
            initialDelaySeconds: 30
            periodSeconds: 10
---
apiVersion: v1
kind: Service
metadata:
  name: grafana
  namespace: observability
spec:
  selector:
    app: grafana
  ports:
    - name: http
      port: 80
      targetPort: 3000
  type: ClusterIP

The key sizing decisions:

• replicas: 2 — two replicas for N+1 redundancy.
• maxSurge: 1, maxUnavailable: 0 — the rollout never reduces the available replica count. The new replica passes readiness before the old one stops.
• GF_DATABASE_MAX_CONN: 20 — per-replica pool. With two replicas, the database sees up to 40 connections. The Postgres max_connections must be at least 60.
• readinessProbe: /api/health — the load balancer only routes to ready replicas. A replica in crash-loop does not receive traffic.
• resources.requests — set to the steady-state CPU and memory. The HPA scales on these.
• maxSurge: 1, maxUnavailable: 0 — the rollout preserves availability. The new replica comes up before the old one goes down.

The shared Postgres is a managed service (RDS / Cloud SQL / Aurora) with Multi-AZ. The Postgres is the single stateful dependency; the cost of HA is the cost of the Postgres.

How to validate it

Confirm the tier matches the load, the failover works, and the cost is documented.

# READ-ONLY
# Confirm the replica count.
kubectl -n observability get deploy grafana -o jsonpath='{.spec.replicas}'

2

Two replicas is the Tier 2 target.

# READ-ONLY
# Confirm the database is reachable from every replica.
for pod in $(kubectl -n observability get pod -l app=grafana -o name); do
  echo "--- $pod ---"
  kubectl -n observability exec $pod -- wget -q -O - http://localhost:3000/api/health | jq '.database'
done

--- pod/grafana-7c5d8b6f-abcde ---
"ok"
--- pod/grafana-7c5d8b6f-fghij ---
"ok"

Both replicas report database: ok. The load balancer is routing traffic to both; the database is the single point of failure.

# CONFIGURATION + SERVICE-IMPACT
# Run the failover test. Delete one replica and confirm the
# other takes the load.
kubectl -n observability delete pod -l app=grafana --field-selector=status.phase=Running

pod "grafana-7c5d8b6f-abcde" deleted

The deployment creates a new replica. The other replica continues to serve. The dashboard renders are uninterrupted (more or less — the rollout is observable as a brief blip).

# READ-ONLY
# Confirm the cost is documented.
kubectl -n observability describe resourcequota

The resource quota documents the budget. The team owns the cost.

How to fail

Six failure modes hit tier sizing in production.

1. Tier 1 outage. Single Grafana dies. Every user loses dashboards. Symptom: a 30-minute outage on the first host reboot after a kernel upgrade. The fix is to upgrade to Tier 2.
2. Tier 2 with sqlite. The “two replicas” are parallel silos. Symptom: dashboards created on one replica are missing on the other. The fix is to share the database.
3. Postgres is the bottleneck. Tier 2 is correctly configured but the Postgres is under-provisioned. Symptom: dashboards are slow at peak. The Grafana replicas are at 30% CPU. The Postgres is at 95% CPU. The fix is to upgrade the Postgres, not the Grafana.
4. Premature HA. Tier 3 is stood up for a Tier 1 load. Symptom: the team is paying $1500/month for a stack that serves 30 users. The fix is to downgrade to Tier 1 and spend the savings on something useful.
5. Cost of drift. One replica is upgraded, the other is not. Symptom: the runtime versions differ. Dashboards load on one replica, fail on the other. The fix is to roll out the same image to every replica.
6. HA load balancer is the new single point of failure. Tier 2 has one HAProxy. The HAProxy dies. Symptom: every replica is unreachable. The fix is to run HAProxy in HA mode (VRRP, keepalived, or a cloud LB).

How to troubleshoot it

Diagnose from the cost inward.

1. Which tier are you actually in? Count the replicas, the database, the load balancer. The classification is straightforward.
2. Did the failover test succeed? The test result is the ground truth. Run it quarterly.
3. Is the database the bottleneck? pg_stat_statements is the canonical tool. The longest queries are the candidates.
4. Is the load balancer the bottleneck? The load balancer logs are the source. Look for connection resets.
5. Is the cost reasonable? The bill is the source. The cost-per-user-per-month is the metric.

Distinguish “is the system up?” (the process is running) from “is the system highly available?” (the failover test passed). The label is not the property.

Security implications

The tier multiplies the attack surface. The security implications of upgrading the tier:

• More replicas, more entry points. Every replica is a potential attack target. Keep them identical. Patch them on the same schedule.
• The shared database is the credential store. TLS, vault, audit. The database is the dependency.
• The load balancer is the TLS terminator. The certificate rotation, the cipher profile, the HSTS configuration.
• Compliance. A regulated industry mandates HA. The cost is a non-choice. The right tier is the cheapest compliant tier.

Performance implications

The tier is the shape of the latency floor. The performance characteristics:

• Tier 1. Single replica. The latency is bounded by the single database round-trip.
• Tier 2. Two replicas. The latency is bounded by the database round-trip plus potential contention between the two replicas.
• Tier 3. N replicas. The latency is bounded by the database round-trip plus the load balancer overhead plus the contention between N replicas.

Adding replicas does not reduce latency. It increases the concurrent requests the database sees. The right configuration is the one that minimises the database contention per request, not the one that maximises the replica count.

Production guidance

The right approach is to match the tier to the operational reality of the team, not to the marketing slide.

• Tier 1 is the right answer for small teams and development environments. Resist the urge to over-engineer.
• Tier 2 is the sweet spot for most production observability stacks. Two replicas, shared Postgres, layer-7 load balancer.
• Tier 3 is for regulated industries or multi-region deployments. The cost is justified by the compliance or the multi-region story.
• Test the failover path quarterly. The test is the ground truth.
• Document the tier. The next on-call engineer needs to know what they are operating.
• Monitor the cost. The bill is the source. The cost per user per month is the metric.

Verification

You should now be able to answer:

• What are the three common tiers of Grafana HA, and what is the cost of each?
• Why is the Postgres database the dominant cost line in a Grafana HA deployment?
• What is the failure mode of a tier chosen above the actual need?
• What is the failure mode of a tier chosen below the actual need?
• Why is running the failover test quarterly the operational discipline, not the design?

Quiz