Cloud-init, ansible-pull, and other VM bootstrap patterns

A VM is just bytes until it’s configured. The question is how to get from “VM exists” to “VM runs our application correctly” reliably, at scale, without manual intervention. This lesson covers the standard patterns.

Cloud-init for first-boot configuration

Cloud-init runs at first boot and configures the VM based on user data. PVE integrates cloud-init directly:

# Attach cloud-init drive to a VM (one-time, on template creation)
qm set 9000 --ide2 local-zfs:cloudinit

# Set user data, meta data, network config
qm set 9000 --ciuser debian
qm set 9000 --cipassword <hashed-password>
qm set 9000 --sshkeys ~/.ssh/id_rsa.pub
qm set 9000 --ipconfig0 ip=10.0.0.100/24,gw=10.0.0.1
qm set 9000 --nameserver 10.0.0.5
qm set 9000 --searchdomain cluster.example.com

# Custom user data via cloud-init
qm cloudinit dump 9000 user > /var/lib/vz/snippets/user-config.yaml
qm set 9000 --cicustom "user=local:snippets/user-config.yaml"

The user-config.yaml is a standard cloud-init config:

#cloud-config
package_update: true
package_upgrade: true
packages:
  - nginx
  - prometheus-node-exporter
  - fail2ban

runcmd:
  - systemctl enable --now nginx
  - systemctl enable --now prometheus-node-exporter
  - ufw allow OpenSSH
  - ufw enable
  - ufw default deny incoming

write_files:
  - path: /etc/nginx/sites-available/default
    permissions: '0644'
    owner: root:root
    content: |
      server {
        listen 80 default_server;
        server_name _;
        location / {
          proxy_pass http://127.0.0.1:8080;
        }
      }

users:
  - name: deploy
    groups: sudo
    shell: /bin/bash
    sudo: ALL=(ALL) NOPASSWD:ALL
    ssh_authorized_keys:
      - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAQQD... deploy@cluster

Cloud-init runs once on first boot. Subsequent reboots don’t re-run it (unless you explicitly tell it to). This is the right behavior for a baseline configuration; for ongoing configuration, use a config management tool.

ansible-pull for self-managing VMs

ansible-pull inverts the normal Ansible model. Instead of a control host pushing configuration to managed nodes, each node pulls its configuration from git:

# Install ansible on the VM
apt install -y ansible git

# Pull and apply a playbook
ansible-pull -U https://github.com/example/cluster-config.git \
  -C main \
  playbooks/web.yml \
  --tags nginx

This runs on a cron schedule (every 15 min) and applies whatever config is in the latest commit. The VM self-manages.

Why ansible-pull?

• Works without a control host. Each VM pulls its own config. No central server to fail.
• Works at the edge. VMs in restricted networks (no inbound SSH) can still pull from a public git repo over HTTPS.
• GitOps-friendly. Configuration lives in git. PRs, reviews, history — all standard git workflow.

Why not ansible-pull?

• Slower at scale. Each VM pulls individually; a fleet of 1000 VMs hammers the git server.
• Eventual consistency. VMs apply config on their own schedule. If you need all VMs updated at once, push (Ansible) is faster.
• No orchestration. ansible-pull can’t sequence multi-host operations (rolling restart, leader election, etc.).

Recommended pattern: pull for base, push for changes

For a typical cluster:

• Base configuration (packages, users, monitoring agent): pull via ansible-pull on every VM
• Application configuration (rolling deploys, schema migrations): push via Ansible or another orchestrator

This combines the resilience of pull with the control of push.

Other bootstrap patterns

Ignition (for Fedora CoreOS, Flatcar)

Ignition runs at first boot, before systemd. It can partition disks, format filesystems, write files, and start services — all declaratively in JSON.

Use Ignition for container-optimised operating systems (Flatcar, Fedora CoreOS, RHEL CoreOS) where you want minimal OS and declarative configuration.

cloud-init with custom datasource

PVE’s cloud-init integration uses the nocloud datasource (config on a CD-ROM-like virtual drive). For multi-cloud portability, use the configdrive or OpenStack datasource with a config drive.

Custom first-boot script

For simple VMs:

# In cloud-init user-data
#cloud-config
runcmd:
  - |
    curl -s https://bootstrap.example.com/setup.sh | bash

The bootstrap script can do anything: install packages, register with consul, send a “ready” message to the orchestrator. This is fragile but useful for one-off VMs.

Bootstrap anti-patterns

Long cloud-init configs

Cloud-init runs synchronously at boot. A 30-minute cloud-init means 30 minutes until the VM is ready. Long configs should be async (a script that runs in the background) or moved to a config management tool.

Passwords in user data

# NEVER do this
ssh_pwauth: true
chpasswd:
  list: |
    root:mysecretpassword

Use SSH keys or a secrets manager (HashiCorp Vault, AWS Secrets Manager). Passwords in user data are visible to anyone who can read the VM’s config or the cloud-init datasource.

Insecure first-boot scripts

# DANGEROUS
runcmd:
  - curl -s https://example.com/setup.sh | bash

If the URL is hijacked, your VM is compromised. Verify checksums:

runcmd:
  - curl -s https://bootstrap.example.com/setup.sh -o /tmp/setup.sh
  - echo "<sha256-hash>  /tmp/setup.sh" | sha256sum -c
  - bash /tmp/setup.sh

Stateful bootstrap

Cloud-init should produce a deterministic state. If the same user-data produces different VMs on different runs, debugging is nightmarish.

Production considerations

• Templates vs cloud-init. Use a base template + cloud-init for most VMs. Use full clones for VMs that need a specific state at creation (e.g., a database with preloaded data).
• Bootstrap time budget. Aim for first-boot to ready-for-traffic in under 5 minutes. Longer than that, and orchestration becomes painful.
• Re-bootstrap. Some teams rebuild VMs from scratch regularly (immutable infrastructure). This requires automation for data restoration from PBS or similar.
• Bootstrap as code. The cloud-init config, ansible-pull playbook, and bootstrap scripts should all be in version control.

Common mistakes

• Cloud-init on every boot. Cloud-init by default only runs once. If you want config to be re-applied, use ansible-pull or another config management tool.
• Storing state in cloud-init. Cloud-init is for declarative configuration, not stateful data. State belongs in databases, not user-data.
• Ignoring failures. A cloud-init that fails silently produces broken VMs. Verify with cloud-init status after boot.

Key takeaways

• Use cloud-init for first-boot configuration.
• Use ansible-pull for ongoing configuration that should be self-managing.
• Never store passwords in user-data.
• Bootstrap should be fast (<5 minutes) and deterministic.

Knowledge check