Loki Upgrade Validation

A team has just rolled Loki from 2.9 to 3.4. The team runs a smoke test: pushes a log line, queries it back, sees the result. The smoke test passes. The team marks the upgrade complete. The on-call engineer walks away. Two hours later, an operator opens a Grafana dashboard and queries a stream from six months ago. The dashboard is empty. The team investigates. The smoke test only validated the new schema. The old schema was silently broken by the schema_config update. The dashboard shows the old time range.

Validation is not a smoke test. A smoke test confirms the thing that was just deployed. Validation confirms the thing that was deployed against the thing that was already there. The synthetic log line is the foundation. The loki-canary is the first signal. The validation is the checklist that proves the upgrade is complete.

What it is

Validation is the post-upgrade process of confirming that the running cluster is serving the same data as the cluster before the upgrade. Validation is a checklist, not a single test. The checklist has four parts:

1. Config validation. The new binary loaded the new config. The schema_config list is what the plan specified. The runtime config file is loaded. The per-tenant overrides are applied.
2. Schema validation. The new schema is being written. The old schema is still being read. A query that straddles the migration date returns results from both schemas.
3. Storage validation. The read path is serving from the new bucket. The canary metric is reporting success. The query latency is within the pre-upgrade baseline.
4. End-to-end validation. A synthetic log line is written every minute. The line is queried back. The result matches the expected response. The check runs continuously and alerts on failure.

The four parts run in order. A config validation failure is a hard block. A schema validation failure is a hard block. A storage validation failure is a hard block. An end-to-end validation failure is a soft block (the upgrade may be complete; the validation may be wrong).

The synthetic log line is the foundation of the end-to-end validation. The line is written by loki-canary (or a custom agent) and queried back by the same. The canary reports four metrics:

• loki_canary_last_success — the timestamp of the last successful round trip.
• loki_canary_read_success_total — the count of successful queries.
• loki_canary_write_success_total — the count of successful writes.
• loki_canary_query_duration_seconds — the histogram of query latency.

A non-zero value for the last success metric is the first signal that the cluster is serving data. A query latency within the pre-upgrade baseline is the second.

Why a sysadmin cares

Validation is the difference between an upgrade that ships and an upgrade that wakes the on-call engineer at 02:00. Three production scenarios apply:

1. The schema_config list was not validated. The new schema entry was added. The read target was not updated. The cluster reads from the old schema for the new time range. The dashboard shows the old time range. The team discovers the regression when a user reports an empty query.
2. The migration was not validated. The boltdb-shipper to TSDB migration was run. The verification step was skipped. The cleanup deleted the source files. The TSDB files are incomplete. The dashboard shows the migrated time range.
3. The synthetic log line was not continuous. The canary was not running during the upgrade. The first signal of a regression is the user report. The team has no way to distinguish a regression from a transient.

The cost of a missing validation is the days of post-incident analysis while the team reconstructs what the cluster was doing before the upgrade. The cost of a present validation is the five minutes of checklist execution that confirms the upgrade is complete.

How to configure it

The validation is a set of commands, not a config file. The commands are the checklist. The checklist is run after every upgrade.

# /etc/loki/loki-canary.yaml
# The canary config. The canary writes a synthetic log line every
# minute and queries it back. The metrics are the foundation of
# the end-to-end validation.
loki-canary:
  base_url: http://loki-read-0:3100
  push_url: http://loki-write-0:3100
  query_timeout: 30s
  read_timeout: 30s
  max_entries: 100
  wal_enabled: true
  wal_dir: /var/lib/loki-canary
  interval: 1m

# The validation checklist. The commands are run in order. A
# failure on any command is a hard block.

# PART 1: Config validation
# Confirm the running binary sees the expected schema_config.
curl -s http://loki-write-0:3100/config | jq '.schema_config.configs'
# expected: the list with both v11 and v13 entries.

# PART 2: Schema validation
# Confirm the new schema is being written.
curl -s http://loki-write-0:3100/metrics | grep 'loki_tsdb_index_writes_total'
# expected: a non-zero count.

# PART 3: Storage validation
# Confirm the canary metric is reporting success.
curl -s http://loki-canary-0:3100/metrics | grep 'loki_canary_last_success'
# expected: a timestamp within the last 60 seconds.

# PART 4: End-to-end validation
# Query a stream that straddles the migration date.
logcli query --addr=http://loki-read-0:3100 \
  '{cluster="prod"} |= "synthetic-upgrade-test"' \
  --since=2026-08-25T00:00:00Z --until=2026-09-05T00:00:00Z
# expected: log lines from both sides of the migration date.

# The validation alert. The alert fires when the canary metric
# is stale for more than 5 minutes.
groups:
  - name: loki-canary
    rules:
      - alert: LokiCanaryLastSuccessStale
        expr: time() - loki_canary_last_success > 300
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: 'Loki canary has not reported success in over 5 minutes'
          description: 'The loki-canary metric loki_canary_last_success is older than 5 minutes. The cluster is not serving data.'

How to validate it

Five commands that confirm the validation is complete and the upgrade is healthy.

# READ-ONLY: confirm the canary is writing the synthetic line.
curl -s http://loki-canary-0:3100/metrics | grep 'loki_canary_write_success_total'
# expected: a non-zero count that increases by 1 every minute.

# READ-ONLY: confirm the canary is reading the synthetic line
# back.
curl -s http://loki-canary-0:3100/metrics | grep 'loki_canary_read_success_total'
# expected: a non-zero count that increases by 1 every minute.
# A read success but no write success means the line was written
# before the upgrade and is still being read.

# READ-ONLY: confirm the canary query latency is within the
# baseline.
curl -s http://loki-canary-0:3100/metrics | grep 'loki_canary_query_duration_seconds_sum'
# expected: the sum divided by the count is within 2x of the
# pre-upgrade baseline.

# READ-ONLY: confirm the schema_config is what the plan specified.
curl -s http://loki-write-0:3100/config | jq '.schema_config.configs[].schema'
# expected: the list of schema versions in the plan.

# READ-ONLY: confirm the version the running binary reports.
curl -s http://loki-write-0:3100/metrics | grep loki_build_info
# expected: loki_build_info{version="3.x.y", ...} 1

How it can fail

Six failure modes cover the most common production incidents tied to validation after a Loki upgrade.

1. The canary was not running during the upgrade. The first signal of a regression is the user report. The team has no way to distinguish a regression from a transient. Symptom: the loki_canary_last_success metric is stale by hours.
2. The synthetic line was not timestamped. The canary writes a line with no timestamp. The line is not findable by the time-range query. Symptom: the loki_canary_query_duration_seconds histogram shows the query but the result is empty.
3. The schema_config list was not validated. The new schema entry was added. The read target was not updated. The cluster reads from the old schema for the new time range. Symptom: queries for the new time range return empty.
4. The query that straddles the migration date was not run. The single-schema validation passes. The historical time range is empty. Symptom: the dashboard shows the pre-migration time range as empty.
5. The canary metric was not alerted on. The canary is writing and reading successfully. The canary metric is stale. The alert does not fire. Symptom: the team discovers the regression from a user report.
6. The validation was run only on the canary pod. The canary pod is healthy. The production pod is unhealthy. Symptom: the canary reports success but the Grafana dashboard is empty.

How to troubleshoot it

The diagnostic order for a validation failure:

1. Is the canary running? The loki_canary_last_success metric shows the last successful round trip. A stale value means the canary is failing.
2. Is the canary writing? The loki_canary_write_success_total metric shows the write count. A flat count means the write path is failing.
3. Is the canary reading? The loki_canary_read_success_total metric shows the read count. A flat count means the read path is failing.
4. Is the schema_config what the plan specified? curl /config | jq .schema_config.configs shows the running config. The list must match the plan.
5. Is the query that straddles the migration date returning results? logcli query with a time range that spans the migration date. An empty result for the pre-migration period is the symptom of a missing schema entry.
6. Is the canary metric being alerted on? The Prometheus alerting rules show whether the alert is configured. The alert must be configured before the upgrade is marked complete.

Security implications

The canary is a privileged client. The canary writes to the cluster with the same credentials as a production client. The credentials must be rotated after the upgrade. The canary service account must have the same IAM policy as a production client.

The synthetic log line is a real log line. The line must not contain sensitive data. The line must be marked with a unique identifier that makes it filterable. The line must be labelled in a way that distinguishes it from production traffic.

Performance implications

The canary is a workload on the cluster. The canary writes one line per minute. The canary reads one line per minute. The workload is small but not zero. The canary’s query lands on the same querier fleet as a production query. The loki_request_duration_seconds histogram on the canary query must be monitored.

The canary query latency is a leading indicator of the production query latency. A spike in the canary latency is a leading indicator of a regression. The histogram should be monitored alongside the production histograms.

Production guidance

• Run the canary continuously. The canary is the first signal of a regression. A canary that is stopped for the upgrade is a gap in the validation.
• Alert on the canary metric. The alert must fire before the user report. The standard alert is time() - loki_canary_last_success > 300.
• Run the validation checklist after every upgrade. The checklist is a set of commands in the upgrade runbook. The checklist must be run before the upgrade is marked complete.
• Validate the schema_config list against the plan. The list is the source of truth for the read path. The list must match the plan.
• Run the query that straddles the migration date. The query must return results from both sides of the date. A missing side is the symptom of a missing schema entry.
• Document the rollback path. The path is the inverse of the upgrade. The cluster is rolled back to the previous version. The previous config is restored. The dual-write window is the safety net.

Verification

You should now be able to answer:

• What are the four parts of the validation checklist, and why must they run in order?
• Why is the synthetic log line the foundation of the end-to-end validation, and what is the failure mode of a silent canary?
• What is the query that straddles the migration date, and why must it return results from both sides?
• What is the alert that proves the canary is stale, and what is the threshold?
• What is the rollback path if the validation fails after the upgrade is marked complete?

Quiz