Upgrade Basics

At 02:14 an on-call engineer runs apt upgrade on a Prometheus host during a quiet window. The package version moves from 2.55.0 to 2.55.1. Twenty minutes later, the rule evaluator silently drops a recording rule that referenced a label the new release no longer accepts. Dashboards go grey. By 02:50 the team is reverse-rolling a single-host change that was meant to be a no-op.

The change was technically a patch release. The operator assumed patch meant “no thinking required.” It almost was. The recording-rule syntax tightened in 2.55.x and the project’s own release note called it out under “[BUGFIX] Stricter PromQL label matching”. Nobody on the rotation had read the release notes.

That gap — assuming an upgrade is routine without checking — is the failure shape this lesson exists to close.

What upgrade discipline is

Upgrade discipline is the set of habits that turn a version bump into a deliberate operation rather than a reflex. It has four parts:

1. Read the version. The semver string encodes the maintenance risk. A major bump is a different shape of work from a patch bump.
2. Read the release note. Every line is a clue about what changed, what broke, what was deprecated, and what was fixed.
3. Pick the tier. Different parts of the observability stack tolerate upgrades differently. The collector is not the same kind of dependency as Grafana.
4. Pick the window. Upgrades have to land at a time when their failure shape is detectable, attributable, and reversible.

The discipline is the same whether the operator is moving from 2.54.0 to 2.55.1 or from 11.2.0 to 11.3.0. The contents of the release note change; the steps do not.

Why a sysadmin cares

The observability stack is unusual: it observes itself. When it breaks, the team’s ability to detect and diagnose the breakage falls at the same time. The cost of a bad upgrade is therefore amplified.

Three operational pains the discipline prevents:

• Telemetry loss during incidents. If the rule evaluator breaks during a peak-traffic event, the alert that should have fired does not fire. The on-call engineer arrives blind.
• Schema migrations that go sideways. Loki’s index gateway and compactor, Prometheus’s TSDB on-disk format, Tempo’s block layout — all of these can require explicit migration steps that are not undoable without a backup.
• Cross-component drift. Grafana 11.x will not query a Prometheus that is running an unreleased internal API. Loki 3.x has dropped a remote-write receiver that some exporters relied on. The version skew is invisible until a panel breaks.

The cost of doing the discipline well is about thirty minutes per upgrade. The cost of doing it badly is measured in lost dashboards during an incident.

How it works: semver as a contract

The semver string MAJOR.MINOR.PATCH is a contract about what the new version is allowed to have done to behaviour:

    MAJOR      .    MINOR     .    PATCH
      |              |              |
      |              |              +--- Backwards-compatible
      |              |                  bug fixes only.
      |              +------------------ New features, no
      |                                   breakage, deprecations
      |                                   may appear.
      +--------------------------------- Breaking changes
                                          permitted. API may
                                          be removed.

The Grafana, Prometheus, Loki, Tempo, OpenTelemetry Collector and Alloy projects follow this contract with varying strictness. Prometheus in particular is conservative: a major bump is rare and indicates a real change. Grafana and Loki are more liberal — minor releases can include breaking configuration changes (renamed flags, removed receivers). The release note is the source of truth, not the version number.

What each tier of semver implies for the operator:

• Patch. Re-read the bug-fix list. Assume the upgrade is safe. Validate anyway.
• Minor. Re-read the deprecation list, the new-feature list, and any “changed” entries. Configuration drift is the main risk.
• Major. Read the migration guide. Plan for schema work, API removals, and behaviour change. The release note is the start of the plan, not the plan itself.

How it works: the per-tier approach

The observability stack has six logical tiers. Each tier has a different blast radius and a different tolerance for downtime:

+----------------------------------------------------+
| Tier 1: Collection  (Alloy, OTel Collector,         |
|          exporters, scrape configs)                 |
+----------------------------------------------------+
| Tier 2: Storage backends (Prometheus TSDB,          |
|          Loki store, Tempo blocks, object storage)  |
+----------------------------------------------------+
| Tier 3: Query / process engines (Prometheus,         |
|          Loki queriers, Tempo queriers)             |
+----------------------------------------------------+
| Tier 4: Alerting and routing (Alertmanager,          |
|          notification receivers)                    |
+----------------------------------------------------+
| Tier 5: Presentation (Grafana, dashboards,          |
|          datasources, alerting UI)                  |
+----------------------------------------------------+
| Tier 6: Cross-cutting (config repo, IaC, backups,   |
|          runbooks, on-call rotations)               |
+----------------------------------------------------+

The rule of thumb:

• Tier 1 first. The collector and exporters change shape most often. Upgrade them early in the cycle so the rest of the stack sees the new shape first.
• Tier 2 second. Storage schema changes are the riskiest. They require downtime or rolling-restart tolerance.
• Tier 3 third. Queriers and rule evaluators follow storage.
• Tier 4 fourth. Alertmanager is independent enough that it can move out of band.
• Tier 5 last. Grafana can read data from older components; an old Grafana cannot read data from a newer querier that has changed a response shape.
• Tier 6 is continuous. Config as code, IaC, backups and runbooks have to move with every tier.

How to configure it: an upgrade plan template

The plan is not a wiki page. It is a checked-in file with a predictable shape. One file per upgrade, in the same repository as the stack’s configuration:

# upgrades/2026-09-prometheus-2.55.1.yaml
upgrade:
  id: 'prom-2.55.1-2026-09-12'
  component: 'prometheus'
  from_version: '2.55.0'
  to_version: '2.55.1'
  tier: 3
  window: 'low-traffic'
  owner: 'observability-on-call'
  rollback:
    snapshot: 's3://prom-snapshots/2026-09-11-pre/'
    procedure: 'runbooks/prometheus-rollback.md'
  release_note_reviewed_by: 'alice'
  release_note_reviewed_at: '2026-09-10T15:00:00Z'
  risk:
    semver_tier: 'patch'
    breaking_changes: 'none-known'
    deprecations: 'none-known'
    security: 'none-known'
    schema_change: 'none-known'
  validation:
    - 'promtool check config /etc/prometheus/prometheus.yml'
    - 'curl -fsS http://prom-1:9090/-/ready'
    - 'curl -fsS http://prom-1:9090/api/v1/query?query=up'
  canary:
    host: 'prom-canary-01'
    duration_minutes: 60
    exit_criteria: 'no error budget burn > 0.1%'
  go_no_go:
    required: ['release-note-reviewed', 'snapshot-taken',
               'canary-passed']

The fields encode the discipline. A reviewer who reads the plan sees the version, the tier, the window, the rollback, the validation, and the canary — without having to ask.

How to validate it

The minimum validation set for any patch upgrade on Prometheus, Loki, Tempo, Alloy or the OTel Collector:

# READ-ONLY: confirm the package was unpacked and the binary
# runs.
prometheus --version 2>&1 | head -1
# Prometheus version 2.55.1

# READ-ONLY: validate the configuration before restart.
promtool check config /etc/prometheus/prometheus.yml
# SUCCESS: /etc/prometheus/prometheus.yml is valid

# READ-ONLY: validate the rule files.
promtool check rules /etc/prometheus/rules/*.yml
# SUCCESS: 27 rules found

# CONFIGURATION: reload the running service so it picks up the
# new rule files without dropping scrapes.
curl -fsS -X POST http://prom-1:9090/-/reload
# (empty body on success)

# READ-ONLY: confirm readiness and the metric pipeline is alive.
curl -fsS http://prom-1:9090/-/ready
# ready
curl -fsSG --data-urlencode 'query=up' \
    http://prom-1:9090/api/v1/query | jq '.data.result | length'
# 412

The same shape applies to the other components:

# Loki
loki -version | head -1
loki -config.file=/etc/loki/loki.yaml -verify-config

# Tempo
tempo -version | head -1
tempo -config.file=/etc/tempo/tempo.yaml -verify-config

# Grafana
grafana-server -v | head -1
grafana cli admin data-source list

# OTel Collector (binary)
otelcol --version
otelcol validate --config=/etc/otelcol/config.yaml

# Grafana Alloy
alloy --version
alloy fmt --check /etc/alloy/config.alloy

The -verify-config flag (Loki, Tempo) and validate subcommand (Collector) are dry-runs: they parse the config and exit non-zero on error without starting the service. They exist precisely so the upgrade can be tested before restart.

How it can fail

Five failure modes recur in observability-stack upgrades.

1. Assumed patch is not a patch. A maintainer backports a behaviour change into a patch release under a CVE justification. The rule evaluator starts treating a label match as strict where it was loose. Symptom: a previously green recording rule stops producing samples; dashboards show no data for that series.
2. Simultaneous multi-component upgrade. Three components are bumped in the same change window. When the symptom appears, the team cannot tell which component caused it. Symptom: a partial outage with no attributable change.
3. Schema migration run against live writers. Prometheus’s TSDB format change, Loki’s index gateway cutover, or Tempo’s block-layout change is applied while writers are still pushing the old shape. Symptom: errors on ingest; in the worst case, a corrupted TSDB head.
4. No upgrade window discipline. The upgrade lands at the busiest time of the business week. Symptom: the upgrade itself is fine, but the time to detect the regression is measured in hours because nobody is watching the dashboards.
5. Ignored deprecation. A configuration flag the operator has been using was deprecated two minors ago and is now removed. Symptom: the service refuses to start with the existing config; the operator has no fallback because the flag was already gone in the previous release they did not read about.

How to troubleshoot it

When an upgrade goes wrong, the diagnostic order matters. Start at the service view and move toward the platform view. Do not start by restarting the upgraded process — that destroys the evidence you need to attribute the failure.

1. Was it working before? Compare up{} for the upgraded component against the pre-upgrade snapshot.
2. What does the service view say? Read the upgraded process’s stdout / stderr. Look for “config validation failed”, “out of range”, “deprecated flag”.
3. What does the platform view say? Look at the platform’s own metrics for the upgraded component. Did up{} flip? Did prometheus_tsdb_head_series drop? Did loki_request_duration_seconds spike?
4. Form a hypothesis. Pin the failure to a single line of the release note or to a single config change.
5. Find evidence. Compare the running config against the config that was tested. Compare the running rule files against the rule files that were validated.
6. Test the hypothesis. Roll back to the previous version on a single canary host. Does the symptom clear?
7. Validate the fix. Replay the release-note review with the symptom in mind. Confirm the rollback is the right next step.

The diagnostic order is “did the change go in correctly before asking whether the change is correct.”

Security implications

Two security implications are specific to upgrade work:

• CVEs land in patch releases. The Grafana, Prometheus and Loki projects publish security advisories. A Grafana CVE rated high or critical can require an out-of-cycle upgrade outside the normal window. The discipline of reading the release note catches this; the discipline of patching on a known cadence prevents it from accumulating.
• Credential rotation. Some upgrades change how the component reads its secrets (filesystem to env var, vault path to short-lived token). The upgrade window is also the moment to rotate the credential the component is using.

Performance implications

Performance implications of an upgrade are not symmetric with the upgrade’s risk:

• Storage upgrades can change block layout, compaction patterns, and index size. The TSDB head can grow after a Prometheus major bump because of changed series defaults.
• Query latency can regress because a query engine now applies a stricter planner. Read the release note’s “Performance” section.
• Ingest latency can regress if a collector changes its batching defaults. A batch size reduction can double the number of outbound requests to Loki or Tempo.

The release note will not always call out the performance shape. The validation step is where it shows up.

Production guidance

• Read the release note before opening the upgrade PR. The release note is the first artefact the operator produces comments about, not the last.
• Take a snapshot before the upgrade. The snapshot is the rollback artefact. It is useless if it is taken after the bad state has been written.
• Canary first, then fleet. One host takes the upgrade and proves the change. The fleet follows.
• Validate the canary. The canary is not the upgrade; the validated canary is the upgrade.
• Document the rollback in the same change. The rollback is part of the change, not a separate retrospective.

Verification

You should now be able to answer:

• What does a semver patch bump imply versus a minor bump versus a major bump?
• Which tier of the observability stack should be upgraded first, and why?
• What four signals in a release note should change the shape of an upgrade plan?
• Why is the snapshot taken before the upgrade and not after?

Quiz