Prometheus Upgrade Basics

A 02:30 page lands: the on-call engineer accepts the PR that bumps Prometheus from 2.51 to 2.55. The Helm rollout proceeds. By 02:47 the UI loads, the targets page shows green, and the engineer files the ticket closed. By 03:15 the recording rules have stopped evaluating; by 04:00 the on-call engineer is back, this time looking at a TSDB in a format the previous version cannot read.

A Prometheus upgrade is not “swap a container and reload.” The discipline exists because three layers change on every release: the on-disk TSDB layout, the configuration schema, and the rule / alerting semantics. This lesson is the umbrella. The four lessons after it each cover one of those layers in detail.

What it is

A Prometheus upgrade is the act of moving a running Prometheus binary from one version to another while preserving operational correctness: the same metrics on the targets page, the same recording-rule outputs, the same alerting evaluation, the same remote-write traffic, and the same on-disk TSDB that can still be read on rollback. The version surface inside a Prometheus release falls into four classes:

  +------------------+----------------------------------------+
  | class            | upgrade posture                        |
  +------------------+----------------------------------------+
  | patch (x.y.Z)    | drop-in. binary swap, no plan needed.  |
  | minor (x.Y)      | in-place safe. read release notes.     |
  | major (X)        | may require migration. plan required.  |
  | preview / RC     | production-blocked. staging only.      |
  +------------------+----------------------------------------+

The numbering follows SemVer strictly. Minor releases are guaranteed backwards-compatible at the wire, scrape, and rules level; major releases are the points where the project’s authors may change on-disk formats, deprecate configuration fields, or amend PromQL semantics. In the 2.x line the project has shipped one major-version bump in the recent history (TSDB v2 → v3 across certain versions), and the on-disk format flag --storage.tsdb.wal-compression was the relevant migration switch in release 2.55.

Why a sysadmin cares

A hasty upgrade is the single most expensive Prometheus outage you can run. Three production-grade incidents recur:

1. The unknown format change. An operator runs a major-version upgrade in place. The new binary writes a TSDB block in a layout the previous binary cannot read. Rollback is blocked until the on-disk directory is restored from backup. Mean time to recovery is measured in hours, not minutes.
2. The dropped rule. A recording rule uses a deprecated aggregator that has been removed. The new Prometheus logs a parse error per evaluation interval and stops emitting the series. Downstream dashboards and alerts that consumed the series silently empty out.
3. The flipped default. A default value changes between versions (a quota, a scrape timeout, a sample limit). The operator did not specify the value explicitly, so the new default takes effect, and behaviour shifts without an obvious cause.

The discipline this lesson describes prevents each of those three.

How it works

The six-step discipline is the same shape regardless of the size of the jump. Run it once for 2.51 → 2.55. Run it again for 2.55.0 → 2.55.2. The cost is thirty minutes per step; the cost of skipping any of them is unbounded.

   1. Read release notes                (READ-ONLY)
   2. Snapshot TSDB and config          (DATA-LOSS-RISK: backup)
   3. Validate config and rules         (READ-ONLY: promtool)
   4. Test in non-production            (SERVICE-IMPACT: staging)
   5. Canary one replica                (SERVICE-IMPACT: subset)
   6. Roll production + validate        (SERVICE-IMPACT: prod)

Step 1 turns you into a person who knows what is changing, not a person who is finding out. Step 2 makes rollback possible — without a snapshot there is no rollback, only forward. Step 3 catches the silent mistakes before they reach a process. Step 4 exercises the upgrade in an environment where the failure cost is bounded. Step 5 narrows the blast radius on production to a single replica so that the operator sees the failure shape on one host before it reaches all hosts. Step 6 is the actual change, followed immediately by the validation in lesson 6.

How to configure it

The upgrade discipline does not live in prometheus.yml. It lives in three external places, all of which the operator owns:

• A snapshot before the upgrade. The single most important mechanical step. Stop the process, snapshot the data directory to a safe location, and only then proceed.
• A version-pinned image. prom/prometheus:v2.55.1 rather than prom/prometheus:latest. Pinning the patch level is the difference between a known upgrade and a surprise one.
• A peer-reviewable change request. The image tag bump and any flag changes commit as a PR with a link to the upstream release notes.

Version-pinned container spec

# Kubernetes Deployment excerpt — pinned image, explicit flags
spec:
  containers:
    - name: prometheus
      image: prom/prometheus:v2.55.1
      imagePullPolicy: IfNotPresent
      args:
        - --config.file=/etc/prometheus/prometheus.yml
        - --storage.tsdb.path=/prometheus
        - --storage.tsdb.retention.time=30d
        - --web.enable-lifecycle
        - --web.console.libraries=/usr/share/prometheus/console_libraries
        - --web.console.templates=/usr/share/prometheus/consoles
      ports:
        - name: http
          containerPort: 9090
      readinessProbe:
        httpGet:
          path: /-/ready
          port: http
        periodSeconds: 5
      livenessProbe:
        httpGet:
          path: /-/healthy
          port: http
        periodSeconds: 30
        failureThreshold: 5

The two probes turn the upgrade from “the process started” into “the process is ready.” The lifecycle endpoint enables a graceful POST /-/reload on configuration changes without a container restart, which matters once the cluster runs a hot-reload-aware config manager.

Rollout strategy annotation

# PodDisruptionBudget anchors the canary quota.
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: prometheus
  namespace: monitoring
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: prometheus

With two Prometheus replicas and maxUnavailable: 1, the rolling upgrade proceeds one replica at a time. The first replica terminates only after the new replica is Ready. The operator still validates both replicas at the end; the PDB only guarantees the order.

How to validate it

The validation that lives closest to the upgrade is mechanical and runs from the operator’s workstation against the production HTTP API.

# READ-ONLY: confirm the new binary reports the expected version.
curl -fsS http://prometheus.internal:9090/api/v1/status/runtimeinfo \
  | jq '.data.version, .data.startTime, .data.storageRetention'
# "2.55.1"
# "2026-08-14T09:30:14.234Z"
# "30d"

# READ-ONLY: confirm TSDB head is healthy (no truncation in flight).
curl -fsS http://prometheus.internal:9090/api/v1/status/tsdb \
  | jq '.data'
# {"headStats":{"numSeries":42118,"minTime":"...","maxTime":"..."},
#  "seriesCountByMetricName":{...}, ...}

# READ-ONLY: confirm the scrape and rule workers are running.
curl -fsS http://prometheus.internal:9090/api/v1/status/config \
  | jq '.data.yaml | lines | length'
# (matches the line count of the expected config on disk)

# CONFIGURATION: confirm that rule evaluation is producing output.
curl -fsS 'http://prometheus.internal:9090/api/v1/query?query=up' \
  | jq '.data.result | length'
# 187
# (non-zero value, matches the expected active target count)

# SERVICE-IMPACT: reload without restarting the process.
curl -X POST http://prometheus.internal:9090/-/reload

The five commands turn “the binary is up” into a measurable claim: the right version is running, the TSDB is healthy, the config has the expected line count, the targets are responding, and the rule engine is producing output. Lesson 6 deepens each of these checks.

How it can fail

Six shapes recur. Recognise the symptom; the fix follows.

1. Pinned to :latest. The image tag in the deployment is prom/prometheus:latest. The actual running version drifts with the upstream Docker Hub release. The symptom is “Prometheus upgraded itself; we do not know which version is running.” Check with curl /api/v1/status/runtimeinfo.
2. Snapshot taken after the upgrade. The snapshot script runs on a cron and the upgrade happened outside the schedule. The backup is of the new binary’s TSDB, not the old one’s. The symptom is “we have a backup, but the backup is unreadable on the previous binary.” Restore is possible only by replaying remote-write data.
3. Config not validated against the new binary. A field was renamed. The new binary accepts the config with a log warning; the operator does not see the warning. Behaviour silently differs. The fix is promtool check config against the new binary image.
4. Rules tested only in staging. Recording rules reference metrics that the staging export set does not produce. The staging run evaluates cleanly; production has a permanent vector cannot contain metrics with the same name-set error. The fix is a staging scrape config that mirrors the production target catalogue.
5. Canary skipped. All replicas upgraded at once. The first sign of trouble is at the whole-cluster level. The fix is maxUnavailable: 1 and an explicit canary step.
6. Reload endpoint disabled. --web.enable-lifecycle is not set. Every config change requires a container restart. The symptom is a long rollout on every rule change. The fix is to add the flag.

How to troubleshoot it

If the upgrade has already gone wrong, the diagnostic order is:

1. What is the running version? curl /api/v1/status/runtimeinfo. If it does not match the intended image tag, the rollout did not finish; investigate the deployment, not the binary.
2. Is the process up? curl /-/healthy and curl /-/ready. The two endpoints answer different questions. -/healthy means the process is alive. -/ready means TSDB initialised and the configuration is valid.
3. What does the log say? kubectl logs -l app=prometheus --tail =200 for the same time window as the upgrade. Look for level=error lines about parse error, unknown field, out of order, and migration required.
4. What does the TSDB say? curl /api/v1/status/tsdb reports the head stats and the number of series. A series count that is dramatically lower than before the upgrade indicates that some targets stopped scraping.
5. What does the targets page say? curl /api/v1/targets for the health state of each target. Targets in up = 0 after the upgrade indicate a scrape-config regression.
6. Form a hypothesis. Pick the layer (TSDB, config, rules) that the symptoms point at and follow the matching lesson.

Security implications

An upgrade introduces three security touchpoints:

• The new binary’s known CVEs. A patch release often includes a security fix. The release note flags which CVE. The operator must check the changelog against the host’s CVE register, not just the binary version, because a skipped patch release may have been the one carrying the fix.
• Web configuration changes. The web.yml syntax for TLS and basic-auth has been adjusted across versions. An upgrade on top of an older web.yml may silently load with the old defaults and miss a hardening directive.
• Remote-write credentials rotation. A new version may rotate the credential it sends to the remote; verify the remote backend still accepts the new value. Some operators find out about an upstream rotation only after the upgrade fails the remote-write handshake.

Performance implications

The performance cost of an upgrade is twofold:

• First-compaction cost. A new TSDB format writes its first block differently. The first compaction after an upgrade is visibly slower than steady-state. Schedule upgrades outside peak hours.
• Initial scrape fan-out. All replicas re-evaluate their rules from scratch after restart. A large rule group may take several minutes to come up. The readiness probe holds the new replica out of the service rotation until the rule engine has warmed. minTime = maxTime indicates that the evaluation is at a single instant and has not yet collected a window; alert on this until both diverge.

Production guidance

• Pin the image tag to the exact patch level (v2.55.1, not v2.55). Patch-level changes are infrequent and almost always silently behavioural.
• Take a snapshot before each upgrade, even on patch versions. The cost is five minutes of downtime; the alternative is hours of recovery.
• Run promtool check config and promtool check rules against the new binary, not the one currently running. The promtool shipped with the upgrade image is the one that validates the upgrade image.
• Configure a scrape-interval-graded canary with maxUnavailable: 1. One replica at a time.
• Read the operational release notes, not only the headline bullet. The headline is what changed; the operational notes are what the operator must do.
• Document the upgrade window, the operator on call, and the rollback path in the same PR that ships the tag bump.

Verification

You should now be able to answer:

• What is the difference between a patch, minor, and major Prometheus release from the operator’s point of view?
• Why is a snapshot non-optional, even for a patch upgrade?
• Which steps of the six-step discipline can a CI pipeline run automatically, and which require a human?
• What symptoms follow a hasty upgrade?

Quiz