cloud-init and the instance metadata service

A cloud image is generic. The instance that boots from it has a hostname, an SSH key, a network configuration and often an application. cloud-init is the mechanism that turns one into the other, on first boot, from data supplied by the platform.

Understanding it matters for two reasons. It is where “the instance came up but the SSH key is not there” is diagnosed, and it runs on top of the metadata service, which is the single most security-sensitive endpoint a cloud instance can reach.

Boot stages

cloud-init is not one service. It is four systemd units that run at different points, and knowing which one your configuration lands in tells you what is available to it.

The consequence that catches people: anything needing the network cannot run in the local stage, and runcmd is close to the end of boot, not the beginning. A runcmd entry that expects a service started by a later unit will fail intermittently depending on timing.

systemd-analyze blame | grep cloud
systemctl status cloud-final.service

Debugging cloud-init

Three commands answer almost every question.

# Did it finish, and where did the time go?
cloud-init status --long
cloud-init analyze blame | head

# Is the user-data even valid? Validate BEFORE you launch.
cloud-init schema --system
cloud-init schema --config-file user-data.yaml

# What actually happened
sudo less /var/log/cloud-init.log        # cloud-init's own trace
sudo less /var/log/cloud-init-output.log # stdout/stderr of runcmd and scripts

The state it reports matters:

• status: done - completed. It does not mean every runcmd succeeded; check cloud-init-output.log.
• status: error - a module raised. The --long output names it.
• status: running - still going, or hung waiting for a datasource that is not reachable.
• status: disabled - something created /etc/cloud/cloud-init.disabled, or the kernel command line contains cloud-init=disabled.

A common failure worth recognising: YAML that is valid but wrong. #cloud-config must be the very first line, with no leading blank line and no leading whitespace. Without it the file is treated as an unrecognised type and silently ignored - the instance boots perfectly with none of your configuration applied.

The instance metadata service

Every major cloud exposes a link-local HTTP endpoint that tells an instance about itself, and hands out temporary credentials for the role attached to it. The address and the calling convention differ per provider.

AWS - IMDSv2, token required:

TOKEN=$(curl -sX PUT "http://169.254.169.254/latest/api/token" \
  -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")

curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
  http://169.254.169.254/latest/meta-data/instance-id

curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
  http://169.254.169.254/latest/meta-data/iam/security-credentials/

GCP - a header, not a token:

curl -s -H "Metadata-Flavor: Google" \
  http://metadata.google.internal/computeMetadata/v1/instance/id

curl -s -H "Metadata-Flavor: Google" \
  http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token

Azure - a header and an explicit API version:

curl -s -H "Metadata: true" \
  "http://169.254.169.254/metadata/instance?api-version=2021-02-01"

The header requirements are not decoration. They exist so that a plain GET triggered by a server-side request forgery cannot reach the endpoint: an attacker who can make your application fetch an arbitrary URL usually cannot make it set an arbitrary header, and on AWS cannot make it issue a PUT first.

Where cloud-init and metadata meet

cloud-init reads its configuration from the metadata service, so a metadata problem presents as a cloud-init problem:

# Which datasource was detected?
cloud-init query --all | head -20
cat /run/cloud-init/ds-identify.log

# The raw user-data as delivered
sudo cloud-init query userdata

An instance stuck at status: running with nothing in the logs is usually failing to reach the datasource - a security group, a route, or a firewall rule blocking 169.254.169.254.

Knowledge check