Compatibility

Grafana 11.3.0 ships a new Loki datasource plugin that uses the Loki 3.x /loki/api/v1/query response shape. The production Loki fleet is at 3.1.0. The new datasource works against staging. The operator rolls Grafana to 11.3.0 first; the canary Grafana returns data: null for every panel backed by Loki; the engineer diagnoses, rolls Grafana back, and a post-mortem records the cause as “skipped compatibility check.”

The failure shape is not a bug. Grafana 11.3.0 advertised support for Loki 3.x in its release note. The team happened to be on a Loki minor that exposed a regression the new datasource took advantage of. Compatibility is not a single answer; it is a table.

What compatibility is

Compatibility is the property that two adjacent components in the observability stack can speak the same protocol, with the same payload shapes, at the same version of any behavioural contract. The stack has six logical tiers and every tier is connected to its neighbours by a wire protocol or a query language:

• Grafana speaks to Prometheus over the Prometheus HTTP API.
• Grafana speaks to Loki over the Loki query API and LogQL.
• Grafana speaks to Tempo over the Tempo query API and TraceQL.
• The OTel Collector and Alloy receive OTLP, Prometheus remote-write, and a dozen other receiver formats.
• Prometheus scrapes exporters that speak the Prometheus text exposition format.
• Alertmanager receives alerts from Prometheus over the Alertmanager HTTP API.

Every one of these wires has a versioned contract. The operator’s job is to keep the two sides of each wire in agreement, or at least within the project’s documented compatibility window.

Why a sysadmin cares

The stack is heterogeneous. There is no single “version” to pin. Each component moves on its own cadence, and the release notes only document the supported pairings up to a point. Three failure shapes recur:

• Newer Grafana, older datasource. Grafana 11.3 expects Loki 3.x. Loki 2.9 responds in the older shape. The datasource plugin parses the response and silently returns no data.
• Older Alloy, newer OTel Collector protocol. A collector running 0.110.x emits OTLP 1.5.0; an Alloy running 0.4x reads OTLP 1.3.0. The collector’s batch processor drops attributes the Alloy does not understand.
• Newer Prometheus, older exporter. Prometheus 2.55 expects certain client_golang semantics. An exporter built against client_golang 1.18 still emits the older shape; Prometheus accepts the data but interprets some histograms incorrectly.

In each case, the components are individually healthy. The combination is not.

How it works: a compatibility matrix

The matrix is a table. Each row is one component; each column is the neighbour it talks to; each cell is the documented compatibility window.

                       Prometheus  Loki    Tempo    OTel     Alert-
                       2.55.x      3.x     current  Coll.    manager
                                                  0.110.x  0.27.x
+----------------------+-----------+--------+--------+--------+--------+
| Grafana 11.3         |   yes*    |   yes* |  yes*  |   -    |   -    |
+----------------------+-----------+--------+--------+--------+--------+
| Prometheus 2.55.x    |    -      |   -    |   -    |   yes  |  yes   |
+----------------------+-----------+--------+--------+--------+--------+
| Loki 3.x             |    -      |   -    |   -    |   yes  |   -    |
+----------------------+-----------+--------+--------+--------+--------+
| Tempo current        |    -      |   -    |   -    |   yes  |   -    |
+----------------------+-----------+--------+--------+--------+--------+
| OTel Collector 0.110 |   yes     |   yes  |  yes   |   -    |   -    |
+----------------------+-----------+--------+--------+--------+--------+
| Alertmanager 0.27.x  |   yes     |    -   |    -   |    -   |   -    |
+----------------------+-----------+--------+--------+--------+--------+

(*) Grafana's compatibility window for each datasource is
    documented in the datasource plugin's release note. A
    "yes*" means the pairing is documented as supported for
    the current minor of both sides.

What the matrix hides:

• N-1 support. Most components support the previous minor of their neighbour. Prometheus 2.55 supports the Alertmanager 0.26.x API as well as 0.27.x. The supported range is narrower at the leaf than at the root.
• Plugin API skew. Grafana 11.3 may not load a datasource plugin built for 11.2. The plugin has its own semver inside the Grafana instance. The matrix needs a second axis for plugin versions.
• Contrib-receiver skew. The OTel Collector’s contrib distribution is a moving target. A contrib receiver built yesterday may emit metrics Prometheus does not understand tomorrow. The matrix needs a date axis.

How to configure it: the per-tier upgrade order

The rule of thumb is leaves first, root last. The collector and exporters are the leaves — they touch the external world and change shape most often. Grafana is the root — it consumes the other tiers’ output and is the last to learn the new shape.

   Tier 1 (leaves)         Tier 2/3 (middle)        Tier 5 (root)
+---------------------+   +--------------------+    +--------------+
| OTel Collector      |   | Prometheus         |    | Grafana      |
| Alloy               |   | Loki               |    |              |
| Exporters           |   | Tempo              |    |              |
+---------------------+   +--------------------+    +--------------+
         |                          |                       ^
         +---> upgrade first -------+                       |
                    |                                        |
                    +-------------- upgrade last -----------+

A safe per-tier order:

1. Exporters and leaf agents. Anything that scrapes or pushes the wire format the rest of the stack expects.
2. OTel Collector / Alloy. The aggregator and processor.
3. Storage backends. Prometheus, Loki, Tempo. Schema work lives here.
4. Alerting and routing. Alertmanager. It speaks only to Prometheus.
5. Grafana and presentation. The last to learn the new shape.

The rule is not absolute. A Grafana upgrade may need to precede a Loki upgrade if the new Grafana datasource plugin expects the old Loki API. The matrix is the decision tool; the rule is the default.

How to validate it

The compatibility check is the cross-product of the individual component validations from lesson 01:

# READ-ONLY: confirm the running Prometheus version and
# the Alertmanager version it is talking to.
prometheus --version 2>&1 | head -1
curl -fsS http://alertmanager:9093/-/ready
curl -fsS http://alertmanager:9093/api/v2/status \
    | jq '.versionInfo.version'

# READ-ONLY: confirm the Grafana datasource plugin
# version and the upstream it expects.
grafana cli admin data-source list 2>/dev/null \
    | jq '.[] | {name, type, version}'

# READ-ONLY: confirm the OTel Collector and Alloy are on
# the same OTLP protocol version.
otelcol --version
otelcol components 2>/dev/null \
    | grep -E 'otlp|receiver'
alloy --version
alloy fmt --check /etc/alloy/config.alloy

# READ-ONLY: verify that Grafana can actually talk to
# every datasource it claims to talk to.
for ds in $(grafana cli admin data-source list 2>/dev/null \
              | jq -r '.[].uid'); do
    grafana cli admin data-source test "$ds"
done

The last command — data-source test — is the one that catches the most compatibility regressions. It runs the actual query and reports the failure.

How it can fail

Five failure modes recur around cross-component compatibility.

1. Grafana ahead of Loki. A new Grafana datasource plugin uses a Loki 3.x-only response shape; the production Loki is at 2.9. Symptom: data: null on every Loki panel; no error in the Grafana log beyond parse error.
2. Collector ahead of exporters. The OTel Collector 0.110.x emits OTLP 1.5.0; the Alloy pipeline still reads OTLP 1.3.0. Symptom: attributes dropped at the Alloy receiver; no error in the Collector log.
3. Alertmanager API skew. Prometheus 2.55.x expects Alertmanager 0.27.x API; the production Alertmanager is at 0.26.x. Symptom: unsupported API version in the Prometheus rule evaluator log; alerts are computed but not delivered.
4. Plugin version skew. Grafana 11.3 refuses to load a datasource plugin built against 11.2’s plugin API. Symptom: the datasource shows up as “unhealthy” in the Grafana UI; panels show “Failed to load datasource”.
5. Protocol default change. A new default for the Prometheus remote-write retry policy interacts badly with the receiver’s connection limit. Symptom: intermittent connection refused errors at the receiver; metric series appear in the receiver only sporadically.

How to troubleshoot it

When a cross-component regression appears, the diagnostic order is from the user-visible symptom back to the wire contract:

1. What does the user see? An empty panel, a missing alert, a dropped trace.
2. What does the consumer log say? Grafana’s plugin log, Prometheus’s rule evaluator log, Alloy’s receiver log.
3. What does the producer log say? The exporter’s metric log, the Collector’s batch processor log, Loki’s ingester log.
4. What version is each side? Check the matrix. Is the pairing inside the supported window?
5. Form a hypothesis. Which recent change moved a side out of the supported window?
6. Test the hypothesis. Roll the most-recently-changed side back. Does the symptom clear?
7. Validate the fix. Update the matrix with the observed pairing; capture the failure shape in the post-mortem.

Security implications

Cross-version skew is a security surface in two specific ways:

• Authentication protocol skew. A newer collector may default to mTLS while an older receiver expects a bearer token. The pairing silently downgrades to no authentication on the next retry. Symptom: data flows but the operator cannot tell whether it was authenticated.
• Vulnerable pairings. The project tracks CVEs against a specific pairing. A collector at version N and a receiver at version N-2 may both be individually patched against the CVE but the combined path may not be.

Performance implications

Cross-version skew produces two specific performance shapes:

• Round-trip amplification. A newer client retries on errors an older server returns as success. Symptom: latency rises, throughput halves.
• Cache invalidation. A new query engine invalidates caches the old query engine populated. Symptom: cache hit rate drops to zero for one window after each component upgrade.

Production guidance

• Maintain the matrix as code. A spreadsheet is lost; a checked-in YAML is reviewed.
• Upgrade leaves first. The collector and exporters land before the root.
• Run the data-source test on every Grafana upgrade. It is the cheapest cross-component check.
• Capture the observed pairing in the matrix, not the documented one. The matrix is a record of what is actually running.

Verification

You should now be able to answer:

• Why does the upgrade order go from leaves to root rather than the reverse?
• What is the difference between a documented pairing and an observed pairing in the compatibility matrix?
• Which CLI command catches the most Grafana-to-Loki regressions in production?
• What does a Grafana-ahead-of-Loki skew look like in the panels?

Quiz