Version skew: what must be true to roll at all · Ansible

Every rolling deployment makes a claim, and almost nobody writes it down:

The old version and the new version can run at the same time, serving the same users, against the same data.

That is not a nice-to-have. It is the defining property of the pattern. serial guarantees that for the duration of the rollout — minutes to hours — some hosts run the old code and some run the new, and both are in the load balancer taking live requests.

If that is not survivable, serial has not made the deployment safer. It has converted a short outage into a long one and given you a fleet you cannot easily return to a consistent state.

Four places skew bites

Database schema

The most common and the most damaging, because the data outlives the rollout.

A migration that runs at the start of the deploy changes the schema under the old version, which is still running on most of the fleet.

Change	Old version during rollout
Add a nullable column	fine — it ignores it
Add a column with a `NOT NULL` constraint and no default	breaks — its `INSERT` omits the column
Rename a column	breaks — it selects a column that no longer exists
Drop a column	breaks if it still reads it
Add an index	fine, though it may lock during creation
Change a column type narrowing the range	breaks — it writes values that no longer fit

The rename row is the classic. ALTER TABLE users RENAME COLUMN email TO email_address is one statement, runs in milliseconds, and every host still on the old version starts throwing errors on the next request.

Skew has to work in both directions:

Old code, new schema — during the rollout, before a host is updated.
New code, old schema — if the migration runs after the code, or if you roll back.

A change that only satisfies one direction is a change you cannot reverse.

API and message formats

Where services talk to each other, or to queues.

A queue is the sharp case. A new version publishes messages in a new format; consumers still on the old version cannot parse them. Unlike an HTTP call, a message persists — a malformed message sits in the queue being retried, failing, and possibly blocking everything behind it long after the rollout finished.

Removed API fields break consumers you did not deploy. Adding a field is usually safe; removing or renaming one is not, and the consumer may be a different team’s service on a different release cycle.

Session affinity and shared session state

A user whose session was created by the old version and whose next request lands on a new-version host.

If sessions live in a shared store — Redis, a database, signed cookies — the two versions must agree on the format. A new version that adds a required field to the session object will reject sessions created by the old one, and the user is silently logged out mid-checkout.

Sticky sessions reduce the exposure and do not remove it: your rollout restarts the host the user was stuck to.

Shared caches

Two versions writing different structures under the same cache keys is a mutual corruption problem. The old version reads a new-format entry and misinterprets it, or crashes on it.

Cache key namespacing by release version solves it cleanly and costs a cold cache at each deploy — which is itself a capacity event worth planning for on a heavily cached service.

Skew failures do not look like deployment failures

This is why they are dangerous.

Your health check passes. The host is serving the new version correctly, which is exactly what it was asked to confirm. The errors are happening on other hosts — the ones still running the old version against the new schema — and those hosts are not in the current batch, were not health-checked by this batch, and look fine to every check you wrote.

The failure surfaces as a rise in the application error rate with no correlated failure in the deployment log. The rollout continues, and it looks healthy the entire time, because from its point of view it is.

A health check on the host being deployed cannot detect a skew failure. Detecting it needs a fleet-wide error-rate check between batches, which is a monitoring integration rather than an Ansible task — and it is the strongest argument for the soak interval from the previous lesson, because a soak with somebody watching the error dashboard is a cheap version of the same thing.

Expand and contract

The pattern that makes a breaking change rollable. It converts one incompatible change into three compatible deployments.

Renaming email to email_address:

flowchart TD
  A["Release 1 — EXPAND<br/>Add email_address.<br/>Write to both columns,<br/>read from email."]
  B["Backfill<br/>Copy email into email_address<br/>for existing rows."]
  C["Release 2 — MIGRATE<br/>Read from email_address.<br/>Still write to both."]
  D["Release 3 — CONTRACT<br/>Stop writing email.<br/>Drop the column."]
  A --> B --> C --> D
  A -.- N1["safe: the pre-release code<br/>still reads email"]
  C -.- N2["safe: rolling back to<br/>Release 1 still works"]
  D -.- N3["safe only once every host<br/>runs Release 2"]

At every point, the version before and the version after can both run. That is the property that makes each of the three deployments a normal rolling deploy with no special handling.

The cost is honest and worth stating: three deployment cycles instead of one, a period where the application writes the same data twice, and the discipline to actually perform the contract step rather than leaving the expand state in place forever. Repositories accumulate half-finished expand-and-contract migrations, and each one is a column nobody dares touch.

The alternative to paying that cost is a maintenance window with the service stopped — which is a legitimate engineering choice and should be made deliberately rather than discovered at 03:00.

Deciding before you start

The question belongs in review, not in the incident.

Read-only / Safemaking the precondition explicit in the playbook— Read-only: an assertion that refuses to run unless somebody has declared the change rollable. It cannot verify compatibility — only that a human answered the question.

- name: Confirm this release is safe to roll
hosts: localhost
gather_facts: false
tasks:
  - name: Refuse to roll a release that has not been assessed for skew
    ansible.builtin.assert:
      that:
        - skew_assessed | default(false) | bool
        - skew_compatible | default(false) | bool
      fail_msg: >-
        Release {{ release_version }} has not been declared skew-compatible.
        During a rolling deploy the previous release runs alongside it.
        If they cannot coexist, use a maintenance window instead.
        Set skew_assessed and skew_compatible in the release vars once
        the schema, API, session and cache compatibility have been checked.
      success_msg: >-
        Release {{ release_version }} declared skew-compatible; rolling.

When to say no

Sometimes the answer is that a rolling deployment is the wrong pattern, and the lesson is to say so rather than to roll carefully and hope.

The versions genuinely cannot coexist, and expand-and-contract is not available — a third-party component, a binary format change, a protocol with no versioning. A maintenance window with the service stopped is shorter, more predictable, and reversible.

The migration is irreversible. If you cannot go back, rolling gains you nothing: the ability to stop halfway is the whole value of the pattern, and stopping halfway through an irreversible change leaves you in the worst state available.

Consistency matters more than availability. For a clustered system where members must agree — a quorum configuration, a replication topology — a split fleet is not degraded, it is broken. That is what any_errors_fatal is for, and it usually pairs with a window rather than a rollout.

Under the hood

Ansible has no view of this and cannot acquire one.

serial slices a host list. It has no model of the application running on those hosts, no knowledge that they share a database, and no way to know that a migration task in an earlier play changed a schema the un-deployed hosts depend on. Every safety mechanism in the previous two parts operates on task results per host — and a skew failure produces no failing task on any host in the current batch.

Trace it through the machinery deliberately:

max_fail_percentage counts failed hosts in the batch. The batch is succeeding; the errors are on hosts that are not in it.

any_errors_fatal triggers on a task returning an error. No task is returning an error.

The health check probes the host just deployed. That host is correct.

The recap reports failed=0 throughout, and the exit code is 0.

Which means the entire failure-detection apparatus this course has built is structurally incapable of seeing this class of failure, and no configuration of it will help. The detection has to come from outside — an error-rate check between batches, a synthetic transaction, a human watching a dashboard during the soak.

That is not a gap to be apologetic about; it is a boundary worth knowing. Ansible reports on what it did. Whether what it did was a good idea for the system as a whole is a question that has to be answered before the run, by a person, which is why the assertion above records a decision rather than measuring a property.

What happens if the migration runs as a run_once task inside the rolling play?

Two failures at once, and the second is the one Part XXXI warned about.

It runs once per batch, not once. run_once executes on the first host of the current batch — verified in Part XXXI, where a run_once task under serial: 2 over six hosts ran three times. A migration written that way runs once per batch, and the second run fails on a duplicate key or, worse, succeeds and applies the change twice.

It puts the schema change inside the rollout. Even if it ran exactly once, it would change the schema partway through, at a moment when most of the fleet is still on the old version. That is the skew failure deliberately engineered.

Migrations belong in a separate play before the rolling play, with hosts: targeting one host or localhost, and they belong to the expand phase — the schema change must already be compatible with the version currently running, because it will be running alongside it for the length of the rollout.

If a migration cannot be made compatible with the currently deployed version, it is not a migration that can be rolled, and this lesson has already given the answer for that case.

Knowledge check

Why this matters

This is the lesson that decides whether the rest of the part applies.

Everything else — the batch shape, the drain, the health check, the soak, the recovery procedure — assumes the two versions can coexist. If they cannot, that machinery is elaborately staging a failure. The rollout will look healthy at every step, because every step is measuring the host in front of it, and the damage will be happening on the hosts it already passed and the hosts it has not reached.

The question takes ten minutes to answer and belongs in the change review: can the previous release and this one run at the same time, against the same data, serving the same users? Schema in both directions, message formats, session state, cache keys.

If the answer is yes, roll. If the answer is no, either restructure the change with expand-and-contract so the answer becomes yes, or take a window and say so.

The answer you must never accept is “we did not check”. That is the one that turns a routine Tuesday deployment into an incident whose cause nobody can find, because nothing failed — the system simply stopped agreeing with itself, one batch at a time, with a clean run log throughout.