A backup you have not restored is a hypothesis

“A backup file exists” is not “a tested rollback”.

This course keeps returning to that distinction because the gap between the two is where change plans fail. backup: true on a module is trivially easy to add, it produces a real file with a real timestamp, and it appears in the change plan as evidence that rollback is covered. In most estates nobody has ever restored from one.

What backup: true actually gives you

It is a genuine facility, and being precise about its scope is what makes the rest of the lesson usable.

copy and template both document the option as: “Create a backup file including the timestamp information so you can get the original file back if you somehow clobbered it incorrectly.” The path comes back in backup_file, documented as returned “changed and if backup=yes”, with a sample of /path/to/file.txt.2015-02-12@22:09~.

So it gives you, precisely:

• one file
• on the host being changed
• as it was immediately before this task wrote to it
• with a path you can only know if you captured the return value

Everything about that list is a limitation, and the four limitations are the four ways the pattern fails in practice.

It is one file

The change touched a config file, a unit file, a package version and a sysctl. backup: true captured the config file. The reverse procedure needs all four.

It lives on the host being changed

If the change is the kind that might make the host unusable — a kernel upgrade, a network reconfiguration, a filesystem change — the backup is inside the blast radius. A backup that shares a failure domain with the thing it protects is a backup for the failures that do not matter.

It is only the state immediately before this task

Run the play twice and the second run backs up the file the first run wrote. The original is now two backups deep, and after five runs during a debugging session it is somewhere in a directory of near-identical files with timestamps a minute apart.

The path is only knowable if you captured it

This is the one that turns a backup into an archaeology exercise at 02:50.

- name: Deploy the configuration
ansible.builtin.template:
  src: app.conf.j2
  dest: /etc/app/app.conf
  owner: root
  group: app
  mode: '0640'
  backup: true
register: config_write
notify: reload app

- name: Record which backup belongs to this change
ansible.builtin.copy:
  content: |
    change_id: {{ change_window_id }}
    backup_file: {{ config_write.backup_file }}
    taken: {{ ansible_date_time.iso8601 }}
  dest: "/var/lib/app-change/{{ change_window_id }}.yml"
  owner: root
  group: root
  mode: '0600'
when: config_write.backup_file is defined

- name: Keep a copy off the host as well
ansible.builtin.fetch:
  src: "{{ config_write.backup_file }}"
  dest: "artifacts/{{ change_window_id }}/{{ inventory_hostname }}/"
  flat: true
when: config_write.backup_file is defined

What to capture before a change

The rule of thumb: capture whatever the reverse procedure reads. If the reverse procedure says “restore the previous config”, the previous config is a required artefact and the change does not start without it.

Where a backup has to live

Two rules, both learned the same way.

Not only on the host being changed. Same failure domain. If the change bricks the host, the backup went with it.

Not only on the controller. The controller is a single point of failure, is frequently a virtual machine nobody backs up, and — as Part L covers — is a thing that can be permanently lost. A change artefact that exists only in /tmp on the controller exists until the next reboot.

The practical answer is both: backup: true on the host for speed, and fetch to a controller path that is itself backed up, or to an artefact store. For anything larger than config files — database dumps, snapshots — the answer is the backup system, and the change plan references it rather than inventing a parallel one.

Asserting freshness in the pre-flight

A backup that exists and is nine days old is not a rollback for a change made today. This is the pre-check assertion from the maintenance-window lesson, and it is worth seeing what produces the number.

- name: Locate the most recent backup for this service
ansible.builtin.find:
  paths: /var/backups/app
  patterns: 'app-*.dump'
  age: -7d
register: recent_backups

- name: A backup exists at all
ansible.builtin.assert:
  that:
    - recent_backups.files | length > 0
  fail_msg: "No backup for app on {{ inventory_hostname }} in the last 7 days"
  quiet: true

- name: The newest backup is inside the freshness limit
ansible.builtin.assert:
  that:
    - >-
      (ansible_date_time.epoch | int)
      - (recent_backups.files | map(attribute='mtime') | max | int)
      < (max_backup_age_hours | int * 3600)
  fail_msg: >-
    Newest backup on {{ inventory_hostname }} exceeds the
    {{ max_backup_age_hours }}h limit
  quiet: true

Note what the assertion does not establish: that the backup contains what you think, that it is not truncated, that it is readable, or that anyone knows how to restore it. It establishes that a file matching a pattern exists and is recent. That is a necessary condition and it is not the interesting one.

Restoring one, inside the window

The step that converts the hypothesis into evidence, and the step that is almost always skipped.

Inside the window, before the change, restore one sample to a scratch location and check it. Not the production path — a temporary one. What this catches, in rough order of frequency:

1. The backup is zero bytes because the job that writes it has been failing silently for three weeks.
2. The backup is truncated because the disk filled halfway through.
3. The backup is encrypted with a key nobody present has.
4. The restore procedure references a tool that is not installed on the host that would need it.
5. The restore takes 40 minutes and the window has 20.

Item 5 is the one that changes the plan rather than the window. A restore duration measured once is worth more in the plan than any number of assertions that a file exists.

Knowledge check