Canary Upgrade

A Loki 3.x upgrade is rolled out canary-first. The canary ingester logs look clean. The canary querier responds in under 200ms on the synthetic test. The fleet rollout proceeds. Twelve hours later, the compactor begins backfilling a query backlog that the canary never saw because the canary was idle. The backlog grows until queries time out. The team rolls back the fleet, learns the canary did not exercise the compactor path, and writes a new validation gate.

The canary passed. The fleet failed. The difference is the validation gates.

What a canary upgrade is

A canary upgrade is the discipline of performing the upgrade on a single representative instance, validating that instance against a defined set of gates, and only then performing the upgrade on the rest of the fleet. The four parts are:

1. Canary host selection. One instance whose workload shape matches the fleet’s. The canary is not the least important host; it is the most representative.
2. Validation gates. Quantitative checks the canary must pass before the fleet rollout. The gates map to the risks identified in the release-note review.
3. Hold window. A defined duration the canary must remain healthy. The hold window is long enough to exercise the slow paths (compaction, replay, GC).
4. Fleet rollout. The fleet upgrade proceeds only after the canary has cleared the gates for the hold window. The canary is not the upgrade; the validated canary is the upgrade.

Why a sysadmin cares

The canary is the cheapest insurance the operator can buy against a bad fleet rollout. Three failure shapes the canary prevents:

• Fleet-wide crash on startup. A bad configuration crashes the upgraded component within seconds. The canary surfaces it; the fleet never sees it.
• Slow path regression. A workload that only manifests under sustained traffic (compaction, replay, GC) is invisible to a synthetic test on a single host. The canary, running real traffic, surfaces it.
• Cross-component skew. The canary is the first place the new component meets the existing fleet. If the pairing is broken, the canary is the first place the error appears.

The cost of the canary is one host’s worth of disruption-plus-monitoring for the duration of the hold window. The cost of skipping the canary is the entire fleet’s disruption at the same moment.

How it works: the canary flow

  Upgrade plan approved
          |
          v
  +-------------------+   No   +-------------------+
  | Can host match?    |------>| Re-scope           |
  +-------------------+       +-------------------+
          | Yes
          v
  +-------------------+
  | Upgrade canary    |
  +-------------------+
          |
          v
  +-------------------+   No   +-------------------+
  | Gates pass?        |------>| Roll back canary   |
  +-------------------+       +-------------------+
          | Yes
          v
  +-------------------+   No   +-------------------+
  | Hold window clean? |------>| Roll back canary   |
  +-------------------+       +-------------------+
          | Yes
          v
  +-------------------+
  | Roll fleet        |
  +-------------------+
          |
          v
  +-------------------+   No   +-------------------+
  | Fleet gates pass?  |------>| Roll back fleet    |
  +-------------------+       +-------------------+
          | Yes
          v
  +-------------------+
  | Close change      |
  +-------------------+

The flow has four decision points. The most-skipped one is “hold window clean?” — the operator declares success because the gates passed once, and the slow path surfaces during the hold window after the fleet rollout.

How to configure it: a canary plan

The canary plan is part of the upgrade plan. It is written before the upgrade begins:

# upgrades/2026-09-loki-3.1.0.yaml (canary section)
canary:
  host: 'loki-canary-01'
  selection_rationale: 'Same zone as prod, ingester-only,
    receives 5% of prod traffic.'
  gates:
    - name: 'startup-time'
      command: 'systemctl show loki -p ActiveEnterTimestamp'
      pass: '< 30s'
    - name: 'ingest-throughput'
      command: |
        curl -fsSG --data-urlencode \
          'query=rate(loki_request_duration_seconds_count{job="loki-canary-01"}[5m])' \
          http://mimir:9009/api/v1/query
      pass: '> 1000 samples/sec'
    - name: 'query-latency-p99'
      command: |
        curl -fsSG --data-urlencode \
          'query=histogram_quantile(0.99, sum(rate(loki_request_duration_seconds_bucket{job="loki-canary-01"}[5m])) by (le))' \
          http://mimir:9009/api/v1/query
      pass: '< 500ms'
    - name: 'chunk-write-errors'
      command: |
        curl -fsSG --data-urlencode \
          'query=rate(loki_chunk_store_writes_failures_total[5m])' \
          http://mimir:9009/api/v1/query
      pass: '= 0'
    - name: 'rule-evaluation-success'
      command: 'promtool check rules /etc/loki/rules/*.yml'
      pass: 'exit 0'
  hold_window_minutes: 60
  exit_criteria: 'all gates pass for full hold window'
fleet:
  rollout_strategy: 'rolling, one zone at a time'
  fleet_gates:
    - 'ingest-throughput within 5% of pre-upgrade'
    - 'query-latency-p99 within 10% of pre-upgrade'
    - 'no new error log patterns'

The gates map to the release-note risks. A release note that flags a query-engine regression has a query-latency-p99 gate. A release note that flags a config-validation tightening has a promtool check rules gate. The gates are not generic; they are specific.

How to validate it

The canary validation commands are the same shape as the upgrade-validation commands from lesson 01, applied to one host:

# READ-ONLY: confirm the canary host is running the new
# version.
ssh loki-canary-01 'loki -version' | head -1
# loki version 3.1.0

# READ-ONLY: confirm the canary is receiving traffic.
ssh loki-canary-01 'curl -fsS http://localhost:3100/ready'
# ready

# READ-ONLY: confirm the canary's ingest rate matches the
# expectation.
curl -fsSG --data-urlencode \
    'query=rate(loki_request_duration_seconds_count{instance="loki-canary-01"}[5m])' \
    http://mimir:9009/api/v1/query | jq '.data.result[0].value[1]'

# READ-ONLY: confirm the canary's chunk-store writes are
# succeeding.
ssh loki-canary-01 'journalctl -u loki --since "10 minutes ago"' \
    | grep -E 'chunk_store_writes_failures|write failed' \
    | wc -l
# 0

# READ-ONLY: confirm the rule files still validate against
# the new binary.
promtool check rules /etc/loki/rules/*.yml
# SUCCESS: 18 rules found

The validation must run for the full hold window. A single five-minute check is not the canary; the full hold window is the canary.

How it can fail

Five failure modes recur around canary upgrades.

1. Canary is not representative. The canary host receives the quietest traffic in the fleet. The canary passes; the fleet fails under real load. Symptom: fleet rollout triggers the regression the canary was supposed to catch.
2. Canary passes once. The gates are run at minute five; the slow path (compaction, replay) surfaces at minute ninety. Symptom: the fleet rollout begins; the regression appears two hours later.
3. No hold window. The operator declares canary success immediately after the gates pass. Symptom: the canary was never exercised under sustained load.
4. Fleet ignores the canary signal. The canary fails a gate; the operator overrides the gate to “warning” and proceeds with the fleet. Symptom: the same failure the canary surfaced appears in the fleet.
5. Canary is a different shape. The canary is a single ingester; the fleet is an ingester plus compactor. Symptom: the canary does not exercise the compactor path; the compactor fails in the fleet.

How to troubleshoot it

When the fleet rollout fails after a passed canary, the diagnostic order is from the canary back to the gate definitions:

1. What does the fleet symptom say? Compare the fleet’s metric shape against the canary’s metric shape at the same moment.
2. What does the canary show now? The canary is still running; it may have surfaced the same regression in the time between the gate check and the fleet symptom.
3. What gate was missed? Map the fleet symptom back to a gate that would have caught it. The gate either did not exist, was not run for the full hold window, or was overridden.
4. Form a hypothesis. The canary was not representative, the hold window was too short, or the gate definitions were incomplete.
5. Test the hypothesis. Re-run the gate with realistic fleet traffic on the canary. Did the gate catch the regression now?
6. Validate the fix. Add the new gate to the upgrade plan; rerun the canary with the new gate.

Security implications

Two security implications are specific to canary upgrades:

• Canary authentication. A canary host with the fleet’s credentials is a host an attacker can pivot through if it is compromised. The canary must be on the same authentication posture as the fleet.
• Canary traffic isolation. A canary receiving production traffic is a canary that can leak. The canary should sit behind the same ingress controls as the fleet.

Performance implications

The canary has a small performance cost during the hold window:

• Reduced fleet capacity. One host is on the new version; the fleet has one fewer replica until the canary completes. For a fleet of three, that is a 33% capacity reduction during the canary.
• Asymmetric load. The fleet has to absorb the canary’s traffic during the canary. The fleet’s per-host load increases by 1/N.

Production guidance

• The canary is not the upgrade. The validated canary is the upgrade.
• The hold window is the canary. A single check is not a hold window.
• The gates map to the release note. A gate that does not map to a release-note risk is busywork.
• The canary host is documented. The selection rationale lives next to the upgrade plan.

Verification

You should now be able to answer:

• Why is the canary host selected for representativeness rather than for low importance?
• What is the difference between a passed canary and a passed hold window?
• What does a canary that passes once but fails the hold window look like in the fleet metrics?
• Which component’s slow path is hardest to exercise on a canary, and why?

Quiz