Writing user-data - payload formats, module frequency, and the test loop

The previous lesson covered what cloud-init is and how to diagnose it. This one is about the thing you actually write: the user-data payload. Most cloud-init failures in production are not bugs in cloud-init. They are a payload in the wrong format, work placed in the wrong stage, or work placed at the wrong frequency - and every one of those failures is quiet.

The payload formats

cloud-init decides what a payload is by looking at its first line. There is no content negotiation and no fallback: an unrecognised first line means the payload is discarded and the instance boots as a stock image.

The list on your own host is authoritative, and cloud-init will print it:

$ cloud-init devel make-mime --list-types

cloud-boothook
cloud-config
cloud-config-archive
cloud-config-jsonp
jinja2
part-handler
x-include-once-url
x-include-url
x-shellscript
x-shellscript-per-boot
x-shellscript-per-instance
x-shellscript-per-once

Those last three are the interesting ones. A shell script attached as x-shellscript-per-boot runs on every boot; the same script attached as x-shellscript runs once for the life of the instance. The content type, not the script, decides.

Combining a cloud-config with a script

You often want both: declarative configuration for users, files and packages, and a script for the one thing no module covers. You cannot concatenate them - the first line decides the type for the whole payload. Build a MIME message instead:

cloud-init devel make-mime \
  -a base.yaml:cloud-config \
  -a bootstrap.sh:x-shellscript-per-instance \
  > user-data.mime

head -1 user-data.mime

Pass user-data.mime as the instance’s user-data. Each part is handled by the handler for its declared content type, at that handler’s normal point in the boot.

Where work belongs

A #cloud-config is not a script with YAML syntax. Each key is consumed by a named module, and each module runs in a fixed stage. The stage decides what is available.

#cloud-config

# init stage - before most services, network is up
write_files:
  - path: /etc/myapp/config.yaml
    permissions: '0640'
    owner: root:root
    content: |
      listen: 0.0.0.0:8080
      upstream: https://api.example.com

users:
  - name: appsvc
    system: true
    shell: /usr/sbin/nologin

# config stage
apt:
  sources:
    example:
      source: 'deb [signed-by=$KEY_FILE] https://apt.example.com stable main'
      keyid: REPLACE_WITH_REAL_KEY_ID

packages:
  - nginx
  - jq

# config stage, after packages
runcmd:
  - [ systemctl, enable, --now, nginx ]

# final stage
final_message: "provisioned after $UPTIME seconds"

The module list for a distribution is in /etc/cloud/cloud.cfg, split into cloud_init_modules, cloud_config_modules and cloud_final_modules. Reading it is the fastest way to answer “does my thing run before or after theirs”:

$ grep -A6 '^cloud_config_modules' /etc/cloud/cloud.cfg

cloud_config_modules:
- wireguard
- snap
- ubuntu_autoinstall
- ssh_import_id
- keyboard
- locale

Two placement rules cover most mistakes:

• bootcmd runs early and on every boot; runcmd runs late and once per instance. If you reach for runcmd to write a file, use write_files - it runs in the init stage, hundreds of module-seconds earlier, and it is declarative.
• runcmd is near the end of boot. A runcmd entry that starts a service which another runcmd entry then queries will race. Use systemctl enable --now and let systemd order it, rather than backgrounding something and sleeping.

Frequency: the thing that surprises people

Every cloud-init module has a frequency, and it is the single most consequential property in the system.

The script directories mirror this exactly. Anything dropped in them by an image build or a configuration management run is executed by the matching scripts_per_* module:

$ ls /var/lib/cloud/scripts/

per-boot
per-instance
per-once
vendor

/var/lib/cloud/scripts/per-boot/ is the correct answer to “this must run on every boot but I do not want to write a systemd unit”. It is also, usually, the wrong answer: a systemd unit with an explicit After= and a restart policy is better than a script the boot runs blind. Use per-boot for provisioning-adjacent work that genuinely belongs to the platform - re-registering with an inventory, re-fetching a short-lived token - and a unit for anything the application depends on.

The test loop

Almost all of this can be checked without launching anything.

# 1. Is the YAML valid, and are the keys ones cloud-init knows?
cloud-init schema --config-file base.yaml

# 2. Annotated errors, pointing at the offending line
cloud-init schema --config-file base.yaml --annotate

# 3. If the payload uses jinja templating, render it first
cloud-init devel render user-data.yaml

# 4. On an already-provisioned test instance: what was delivered?
sudo cloud-init query userdata
cloud-init query --list-keys

cloud-init schema catches the common defects - a key that does not exist, a string where a list belongs, an indentation error - in about a second, with no cloud account involved. Make it a CI step on the repository that holds your user-data.

$ cloud-init schema --config-file base.yaml

Valid schema base.yaml

What it does not catch: whether your runcmd works, whether the package exists, or whether the repository is reachable. For that, launch a disposable instance and read the output log:

cloud-init status --wait --long
sudo tail -40 /var/log/cloud-init-output.log

To re-test one module on a scratch instance without rebuilding it, run just that module at an overridden frequency:

sudo cloud-init single --name write_files --frequency always \
  --file /tmp/base.yaml

Scripts that fail silently

A user script attached as x-shellscript runs with its stdout and stderr captured into /var/log/cloud-init-output.log. Its exit code is logged, and then boot continues regardless. A failing script does not fail the instance, does not fail the health check, and does not fail the deploy.

Write the script so the failure is visible:

#!/bin/bash
set -euo pipefail

LOG=/var/log/provision.log
exec > >(tee -a "$LOG") 2>&1

trap 'echo "provision FAILED at line $LINENO" >&2; \
      systemd-notify --status="provision failed" 2>/dev/null || true' ERR

apt-get update
apt-get install -y --no-install-recommends nginx
install -m 0644 -o root -g root /tmp/site.conf /etc/nginx/conf.d/site.conf
nginx -t
systemctl enable --now nginx

set -euo pipefail stops the script at the first failure instead of running the remaining twenty lines against a broken machine. The trap gives the log a line an operator can grep for. nginx -t validates before the restart, so a bad config fails the provision rather than the service.

Then make something outside the instance notice. phone_home is cloud-init’s own mechanism and it runs in the final stage, so it fires only if the stages before it completed:

#cloud-config
phone_home:
  url: https://deploy.example.com/hooks/provisioned/$INSTANCE_ID
  post:
    - instance_id
    - hostname
  tries: 3

An instance that never phones home is an instance that never finished provisioning - which is a signal you can alert on, unlike an exit code in a log file nobody reads.

The root filesystem that did not grow

One specific failure is common enough to name. You launch from an image whose root volume is 8 GB, ask the platform for 100 GB, and df reports 8 GB.

lsblk
df -h /
sudo growpart /dev/nvme0n1 1     # grow the PARTITION
sudo resize2fs /dev/nvme0n1p1    # then grow the FILESYSTEM
# XFS instead: sudo xfs_growfs /

Normally cloud-init does both for you: the growpart and resizefs modules run in the init stage. They do not run if cloud-init was disabled, if the image ships a growpart: mode: off in /etc/cloud/cloud.cfg.d/, or if the root device is not the one cloud-init detected. The fix is two commands; the value is recognising the symptom as a provisioning gap rather than a platform bug.

Knowledge check