Upgrade Order

A platform team runs a Prometheus 2.51 with an Alertmanager 0.26 on the same release cadence. The team approves a joint upgrade to Prometheus 2.55 and Alertmanager 0.27. The team’s deployment script tags Alertmanager first because “the Alertmanager is the simpler one.” By 03:15 the Alertmanager 0.27 is up. The Prometheus 2.55 rollout begins. By 03:33 the first Prometheus replica has been restarted. The alertmanager logs start showing nflog: unknown version on gossip and gossip clusters are split across the two Alertmanager versions. The team halts the Prometheus rollout and rolls back the alertmanager cluster; the cluster takes 20 minutes to re-converge.

The error was the order. Alertmanager first was right; stopping mid-rollout was right. The mistake was running the Alertmanager upgrade before reading the Alertmanager release notes, and the cost was visible in the gossip layer.

This lesson is about ordering: which side of the alerting chain to upgrade first, why, and the failure shape when the order is wrong.

What it is

The alerting chain has five moving parts. Each lives in a separate process tree with its own release cadence, and each has a binary contract with the components it talks to.

  exporters / instrumentation
        |  scrape (OpenMetrics)
        v
  Prometheus                ----remote-write----> long-term backend
        |  HTTP API (alert states)
        v
  Alertmanager
        |  webhook / email / etc.
        v
  receivers (PagerDuty, Slack, ...)

The upgrade order is the sequence in which the operator upgrades each component in a coordinated way, such that at no point does the chain reach an inconsistent state where one side speaks protocol v2 and the other side speaks protocol v1.

  +-----+------------+------------------------------+
  | step| component   | rationale                    |
  +-----+------------+------------------------------+
  |  1  | receivers   | destination first; they are  |
  |     |             | least coupled to the chain   |
  |  2  | Alertmanager| newest AM is the one that    |
  |     |             | knows the newest protocol    |
  |  3  | Prometheus  | next; needs the new AM API   |
  |  4  | exporters   | they produce what Prometheus |
  |     |             | consumes; OpenMetrics is     |
  |     |             | backwards-compatible         |
  |  5  | dashboards  | Grafana and UI last; read    |
  |     |             | what Prometheus has already  |
  |     |             | accepted                     |
  +-----+------------+------------------------------+

Why a sysadmin cares

A wrongly-ordered upgrade is the most expensive alerting incident you can run. Three shapes recur:

1. The split gossip cluster. Two Alertmanager versions on the same gossip ring that disagree on the wire format causes the cluster to split-brain. Alerts that should deduplicate arrive twice; alerts that should not be suppressed are silenced. Recovery is halting the rollout and rolling back to a single version before resuming.
2. The unparseable AM API. Prometheus 2.55 expects Alertmanager 0.27’s API for some features. An older Alertmanager refuses the request. Prometheus logs the error per evaluation interval. Recovery is upgrading Alertmanager first.
3. The silent exporter regression. An exporter is upgraded to a version that returns metrics in a slightly different shape (a unit change, a label rename). Prometheus parses cleanly; recording rules consume the new shape; dashboards shift. Recovery is the rule test suite, which is the lesson-04 discipline.

How it works

The ordering decision comes from the wire contract at each boundary. The operator reads each release note for the upgrade window and identifies which boundaries changed.

  Prometheus  ----HTTP API---->  Grafana
       |
       | HTTP API
       v
  Alertmanager  ----webhook----> receivers
       ^
       | gossip
       |
  Alertmanager (peers)

The boundaries:

• Prometheus → Alertmanager uses the v1/v2 alert state protocol. New Prometheus versions may emit new state fields (a new label, a new annotation). The new Alertmanager knows the new fields.
• Alertmanager ↔ Alertmanager uses the gossip protocol on top of the cluster. Within a major AM version the gossip is stable across minors; the cluster survives a rolling upgrade cleanly.
• Prometheus → Grafana uses the Prometheus HTTP API. The API is stable across Prometheus minors; Grafana is largely compatible.
• Exporters → Prometheus uses the OpenMetrics / Prometheus exposition format. The format is stable within a major version. New exporters may add labels that older Prometheus ignore (forward compatibility); older exporters may use older constructs that newer Prometheus still parse (backward compatibility).

The order matches the dependency direction: upgrade the side that receives the new protocol before the side that emits it. Receiver first, emitter last.

How to configure it

The order is enforced by the deployment manifest and the upgrade runbook.

Deployment manifest with per-component image pins

# monitoring/kustomization.yaml — version pins per workload
resources:
  - alertmanager/base.yaml
  - prometheus/base.yaml
  - exporters/base.yaml
images:
  - name: prom/alertmanager
    newTag: v0.27.0
  - name: prom/prometheus
    newTag: v2.55.1
  - name: prom/node-exporter
    newTag: v1.8.2

Each component has its own pin. The upgrade PR bumps the relevant tag and links to the corresponding release note. The order in which the PR merges and the order in which the operator rolls out each component are documented in the runbook.

Rollout runbook

#!/usr/bin/env bash
# Upgrade runbook. The order matters; do not parallelise.
set -euo pipefail

# 1. Receivers first.
echo "[1/5] Confirm receivers accept the upcoming webhook contract."
# (manual: read the receiver destination-side release notes)

# 2. Alertmanager next.
echo "[2/5] Rolling Alertmanager to v0.27.0."
kubectl -n monitoring set image statefulset/alertmanager \
  alertmanager=prom/alertmanager:v0.27.0
kubectl -n monitoring rollout status statefulset/alertmanager --timeout=600s

# 3. Prometheus after Alertmanager.
echo "[3/5] Rolling Prometheus to v2.55.1."
kubectl -n monitoring set image statefulset/prometheus \
  prometheus=prom/prometheus:v2.55.1
kubectl -n monitoring rollout status statefulset/prometheus --timeout=1200s

# 4. Exporters last.
echo "[4/5] Rolling node-exporter daemonset to v1.8.2."
kubectl -n monitoring set image daemonset/node-exporter \
  node-exporter=prom/node-exporter:v1.8.2
kubectl -n monitoring rollout status daemonset/node-exporter --timeout=600s

# 5. Validation.
echo "[5/5] Run validation pass (see lesson 06)."

The runbook commits the operator to the order. A PR that reorders the steps (Prometheus before Alertmanager) is a reviewable change that requires an explicit justification.

HA Alertmanager gossip configuration

# alertmanager.yml — HA gossip and clustering
global:
  resolve_timeout: 5m

route:
  receiver: 'default'
  group_by: ['alertname', 'cluster']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

receivers:
  - name: 'default'
    pagerduty_configs:
      - routing_key: ${PAGERDUTY_KEY}
        severity: 'critical'

cluster:
  gossip_transport: udp
  protocol: gossip
  listen_address: '0.0.0.0:9094'
  peers:
    - 'alertmanager-0.alertmanager:9094'
    - 'alertmanager-1.alertmanager:9094'

The gossip configuration is the contract between the two Alertmanager replicas. The two replicas must speak the same gossip protocol version. A rolling upgrade that crosses a gossip-incompatible boundary will split-brain; the cluster log will show nflog: unknown version on gossip and alerts will appear twice.

How to validate it

Five mechanical checks confirm the order produces a consistent chain.

# READ-ONLY: confirm the running Alertmanager reports the new version.
curl -fsS http://alertmanager-0.alertmanager:9093/-/ready
curl -fsS http://alertmanager-0.alertmanager:9093/api/v2/status \
  | jq '.data.versionInfo.version'
# "0.27.0"

# READ-ONLY: confirm Alertmanager cluster is healthy.
curl -fsS http://alertmanager-0.alertmanager:9093/api/v2/status \
  | jq '.data.cluster.status'
# "ready"

# READ-ONLY: confirm Prometheus reports the new version.
curl -fsS http://prometheus.monitoring:9090/api/v1/status/runtimeinfo \
  | jq '.data.version'
# "2.55.1"

# READ-ONLY: confirm Prometheus ships alerts to the new AM API.
curl -fsS http://prometheus.monitoring:9090/api/v1/alertmanagers \
  | jq '.data.activeAlertmanagers[].url'
# ["http://alertmanager-0:9093/api/v2", "http://alertmanager-1:9093/api/v2"]

# READ-ONLY: confirm exporters are reachable.
curl -fsS http://prometheus.monitoring:9090/api/v1/targets \
  | jq '[.data.activeTargets[] | select(.health != "up")] | length'
# 0

A clean validation: Alertmanager reports the new version and a ready cluster status; Prometheus reports the new version and ships alerts to the v2 endpoint; every exporter target is up.

How it can fail

Six shapes recur.

1. AM before receivers are confirmed. Alertmanager upgrades before the receiving system has accepted the new webhook shape. Receivers reject the payload; alerts pile up in the AM nflog. The fix is to confirm the receiver contract before rolling the AM cluster.
2. Prometheus before Alertmanager. Prometheus 2.55 expects Alertmanager 0.27’s API. Alertmanager 0.26 returns 404 on /api/v2. Prometheus logs the error per evaluation. The fix is to upgrade AM first.
3. Alertmanager gossip mixed. Two replicas on different versions gossip in incompatible formats. Cluster splits. Fix: halting the rollout; restoring a single version; waiting for convergence.
4. Exporter rolled before Prometheus accepts new shape. Prometheus parses cleanly anyway, but recording rules consume the new shape and downstream dashboards shift. Fix: rule test suite catches it if maintained.
5. Remote-write target not informed. Prometheus ships samples to a remote backend (Thanos Receiver, Mimir, Cortex) that expects the older remote-write protocol. The backend drops or rejects samples. Fix: upgrade remote backend first or coordinate.
6. Grafana data source stale. Grafana still queries the Prometheus endpoint on the old API. The new Prometheus may have shifted a deprecated endpoint. The fix is to verify the Grafana data source settings match the new Prometheus API expectations.

How to troubleshoot it

The diagnostic order when the upgrade has produced an inconsistent chain:

1. What does the Alertmanager log say? kubectl logs ... alertmanager-0 | grep -E 'gossip|nflog|version'. The first line about an unknown or incompatible version identifies the boundary.
2. What does Prometheus log say? kubectl logs ... prometheus-0 | grep -E 'alertmanager|alertmanagers'. The first line about a 404 or a 500 from Alertmanager identifies the boundary direction.
3. What does the alertmanager cluster status say? /api/v2/status reports cluster.status: ready, cluster.peers, and gossip state. A peers count inconsistent with the deployment replica count indicates the gossip ring is not yet stable.
4. What does the Prometheus /api/v1/alertmanagers endpoint say? A list with one entry when the deployment has two replicas indicates one AM replica is not yet reachable from Prometheus.
5. Is the upgrade half-done? Check the running image tag against the operator’s plan. A mix of two versions is the diagnosis; revert to a single version is the fix.

Security implications

The order has one direct security touchpoint:

• Webhook payload shape. A new Alertmanager may change the webhook payload to a new schema. Receivers that rely on the old shape may reject the new payload or, worse, accept a malformed payload that triggers downstream security behaviour. The upgrade order (receivers first) prevents this.

Performance implications

The performance cost of a coordinated upgrade is bounded by the slowest component to converge. Alertmanager gossip converges in seconds. Prometheus convergence is on the order of the rule-evaluation interval. Exporters converge as fast as the daemonset rollout. The total window is minutes, not hours.

The cost of an out-of-order upgrade is unbounded; it scales with the time-to-detection of the inconsistent chain.

Production guidance

• Keep the per-component image pins in a single file (kustomize images: block or equivalent). The diff for one upgrade is one file.
• Write the order down. Operators change; the runbook persists.
• Test the order on staging before production. The staging cluster catches gossip-format mismatches.
• Roll one component at a time. Do not parallelise AM and Prometheus rollouts on the same window.
• Validate the chain after each component, not just at the end. An inconsistent chain that has been running for an hour has produced duplicate pages.
• Coordinate with remote-write and remote-read backends. The order extends to those components too.

Verification

You should now be able to answer:

• Why is Alertmanager upgraded before Prometheus?
• Which component holds the most expensive version-coupled contracts in the chain?
• What is the failure shape of a mixed-version Alertmanager gossip ring?
• How does the order extend to remote-write and remote-read backends?

Quiz