Two Instances Is Not HA

A Grafana dashboard says “Grafana HA enabled, 2/2 instances healthy.” The two instances are Prometheus 2.55.x running in single-binary mode on two different VMs, each scraping the same targets. Both report up == 1. The dashboard is green. Six months later a host failure takes down Prometheus A. Prometheus B is still running and still reports up == 1 for the same targets. The user-visible behaviour is: alerts continue to fire, dashboards continue to render. But the metrics for the last six months on Prometheus A are gone, because Prometheus B never had them. Prometheus A and Prometheus B are two independent TSDBs with two independent WALs and two divergent time series. The “high availability” is an illusion made up by a panel that checks processes, not data.

This is the most expensive HA mistake in the observability stack. It is also the most common.

What “two instances” looks like

The shape is simple and almost always accidental:

+-- VM A -- prometheus-2.55 -- TSDB A -- scrape targets
                                                ^
+-- VM B -- prometheus-2.55 -- TSDB B -- scrape targets

Variations:

• Two Loki single-binary processes, each with its own chunks directory, each receiving the same log stream from two Promtail clients (one configured to send to A, one to B).
• Two Tempo processes, each with its own WAL and trace storage, each receiving from two OTel Collectors.
• Two Mimir ingesters started with replication_factor: 1 and a non-shared ring (the “two ingester” anti-pattern).
• A “primary/standby” pair with no automatic failover, no shared WAL, and a heartbeat that pages someone to “fail over” by hand.

The defining characteristic is no shared state. Each process is a complete silo. The data on process A does not exist on process B and never has.

Why this pattern appears

It appears because:

1. It looks like HA. Two processes are running. Two processes are healthy. Two failure domains exist. The manager asks “is this HA?” and the engineer says “yes, two boxes.”
2. It is the path of least resistance. No KV store is required. No replication factor is required. No Thanos or Mimir is required. Just run the binary twice.
3. The failure mode is silent. When one box dies, the other keeps running. Nothing errors. Nothing alerts. The data on the dead box is gone, but the dashboards that should show “data is missing” are themselves driven by data that is missing.

What the silent data loss actually looks like

Take the two-Prometheus example. Both scrapes hit the same targets. Each receives its own copy of every sample. Each builds its own TSDB. The two TSDBs are independent.

When VM A dies:

            Before                          After
+---------------------+         +---------------------+
| Prometheus A  (live)|         | Prometheus A  (DEAD)|
| Prometheus B  (live)|         | Prometheus B  (live)|
+---------------------+         +---------------------+
       Both have full              B has its own copy only.
       6-month history.            A's 6-month history is
                                   gone. The "HA" survives.
                                   The data does not.

The recovery path is:

1. Replace VM A.
2. Reinstall Prometheus on the new VM.
3. Start scraping.
4. Watch the new TSDB rebuild itself from now forward. The last 6 months are gone forever.

The alert that should have fired (“TSDB retention gap detected”) never existed. The panels that show data continuity never existed. The recovery procedure was a manual rebuild, not a failover, because there was nothing to fail over to.

The same failure shape in Loki and Tempo

Loki single-binary

Loki in mode: simple is also a single-binary TSDB. Two “simple” mode Loki processes receiving the same log stream do not replicate the stream. Each process has its own chunks directory, its own index, and its own retention window. When one dies, the data on it is gone.

The correct replacement is Loki in mode: microservices with ingester.lifecycler.ring.replication_factor: 3. Then the ingester ring handles replication and the two-process anti-pattern is not the default.

Tempo

Tempo’s local trace storage (backend: local) is the same shape. Two Tempo processes with backend: local are two parallel silos. The replacement is backend: s3 (or GCS, Azure Blob) plus distributor.receivers.otlp.replication_factor: 3 on the ingester ring. The traces live in shared object storage; the ingesters are coordinated by a ring.

Grafana

Grafana 11.x is the exception that proves the rule. Two Grafana replicas pointing at the same external database are HA, because the data (dashboards, users, data sources) lives in the database, not in the Grafana process. The “two instances” pattern is correct only when the state is external to both instances.

How to detect the anti-pattern

Five checks confirm whether a “two instance” deployment is real HA or a parallel silo:

# READ-ONLY
# 1. Are both instances writing to the same data store?
# Prometheus:
curl -s http://prom-a:9090/api/v1/status/runtimeinfo | \
  jq '.data.storageRetention'
# Loki (boltdb-shipper):
ls -la /loki/chunks/ /loki/index/
# If A and B have separate directories, the data is not shared.

# READ-ONLY
# 2. Is there a shared object store?
# Loki / Tempo / Mimir / Thanos all converge on this.
kubectl get configmap loki-config -o yaml | grep -E 's3|gcs|azure'
# If the answer is "local" or "filesystem," the system is
# using local-disk state and the two-instance pattern does
# not provide HA.

# READ-ONLY
# 3. Does the ring show the expected replication factor?
curl -s http://loki-distributor:3100/ring | \
  jq '.shards[].replication_factor' | sort -u
# expected: a single value matching the configured R

# READ-ONLY
# 4. Is there a leader-election backend for singletons?
# (compactor, ruler)
kubectl get pods -l app=loki-compactor -o name | wc -l
# expected: 1 (active) + 1 (standby). More than 1 active is
# split brain. Zero means the compactor is down.

# READ-ONLY
# 5. Do the dashboards distinguish "process is up" from
# "data is intact"? If a panel exists that says "data continuity
# OK," the system is HA. If not, it is two processes.

Replacing the anti-pattern

The correct replacement depends on the stack:

The replacement is more complex than “run two boxes.” The complexity is the price of real HA. The benefit is that when one box dies, the surviving system already has the data and the recovery is automatic.

Production guidance

Verification

You should now be able to answer:

• What is the parallel-silo anti-pattern?
• Why does running two Prometheus processes not provide HA?
• What is the silent data loss window in a parallel-silo Loki or Tempo deployment?
• What is the correct replacement for each of the four parallel-silo shapes?

Quiz