Verifying the Migration

Verification is the part of the migration that proves the migration worked. It is not the apply, which only proves the migration ran. It is not the smoke test in CI, which proves the configuration compiles. It is the cross-check that the declared configuration, the recorded state, and the running infrastructure all agree, and that the surrounding system (dashboards, alerts, CI/CD, runbooks) still observes the estate as it did before.

The lesson is short because the procedure is short. The discipline is the part that takes the time, because every verification step has a temptation to skip it.

What verification is not

  Not verification                         Verification
  -----------------------------------     ---------------------------
  terraform apply returned 0              terraform plan returns 0
  The CI pipeline is green                The CI pipeline reaches the
                                          migrated resources through
                                          the provider API
  The state file exists                   The state file produces an
                                          empty plan for one full
                                          change window
  The team can run terraform plan         The on-call can run terraform
                                          plan from any host against
                                          the production backend

Each row on the right is a separate verification step. None of them implies the others.

The four verification gates

The migration is verified when all four gates pass. The gates are run in order, and any failure aborts the migration.

  Gate 1             Gate 2             Gate 3             Gate 4
  Empty plan         API reachability   Observability      Sign-off
  (state matches     (provider agrees   (dashboards,       (artefact is
  configuration)    with state)        alerts, CI still    committed and
                                       see the estate)     reviewable)
     |                  |                  |                  |
     v                  v                  v                  v
  plan -detailed-    describe calls    alert evaluation    git tag, change
  exitcode returns   succeed for       is unchanged;       log entry, runbook
  0 against the      every resource    dashboard panels    update, on-call
  migrated state     type in the       show the same       handoff
                     batch             data

Gate 1: empty plan

# READ-ONLY: confirm the migrated state matches the live
# configuration. The -detailed-exitcode flag is the canonical
# signal: 0 means no changes, 2 means changes were found.
terraform plan -no-color -detailed-exitcode
echo "exit=$?"

No changes. Your infrastructure matches the configuration.
exit=0

The empty plan must hold for at least one full change window before the migration is considered verified. A plan that is empty on Monday but non-empty on Friday means the migration introduced a race that the weekend did not catch.

Gate 2: API reachability per resource type

For each resource type that was imported, query the provider API directly and confirm the resource is reachable:

# READ-ONLY: cross-check the state against the provider API.
# For an EC2 instance migration:
aws ec2 describe-instances \
    --instance-ids i-0123456789abcdef0 \
    --query 'Reservations[0].Instances[0].State.Name' \
    --output text

running

# READ-ONLY: cross-check an S3 bucket.
aws s3api head-bucket --bucket acme-logs-2024
echo "exit=$?"

exit=0

# READ-ONLY: cross-check an IAM role.
aws iam get-role \
    --role-name terraform-deployer \
    --query 'Role.Arn' \
    --output text

arn:aws:iam::123456789012:role/terraform-deployer

The cross-check exists because Terraform’s read function can succeed against a cached state even when the API would reject the request. The cross-check is the proof that the resource is reachable through the operator’s credentials, which is the path the next apply will use.

Gate 3: observability intact

The migration changes the record of the estate. It does not change the estate’s observability — but it can. A few common ways this breaks:

• The CloudTrail log was pointed at a different bucket by accident during the import.
• The Prometheus scrape config references tags that the new configuration no longer applies.
• The dashboards filter by IAM role ARN, and the import created a different role ARN than the one the dashboards expect.

# READ-ONLY: confirm CloudTrail is still writing.
aws cloudtrail describe-trails \
    --query 'trailList[?Name==`acme-audit`].[Name,S3BucketName,IsMultiRegionTrail]' \
    --output text

acme-audit    acme-cloudtrail-logs    True

# READ-ONLY: confirm a Prometheus target is still scraping.
curl -fsS http://prometheus.acme.internal/api/v1/targets | \
    jq '.data.activeTargets[] | select(.labels.job=="web") | .health'

up
up
up

The observability check is the one most teams skip because the dashboards still load. The dashboards load because the data source has not changed; whether the data source is actually receiving data is a different question.

Gate 4: sign-off

The sign-off artefact is a git tag on the configuration that corresponds to the empty plan, plus a change log entry that identifies the migration batch, the rollback target, and the on-call handoff.

# CONFIGURATION: tag the empty-plan configuration.
git tag -a migration-batch-007-empty-plan \
    -m "Migration batch 7 verified: plan empty against \
state v4-7a2d, API cross-check passed, observability intact."

git push origin --tags

Enumerating objects: 1, done.
Counting objects: 100% (1/1), done.
Writing objects: 100% (1/1), 218 bytes | 218.00 KiB/s
To git@github.com:acme/infrastructure.git
 * [new tag]         migration-batch-007-empty-plan ->
   migration-batch-007-empty-plan

The tag is the input to the rollback. It identifies the configuration that corresponds to the verified state. The change log entry is the audit record.

When a non-empty plan is acceptable

Not every non-empty plan after a migration is a bug. The cases where it is acceptable:

  Acceptable                       Why
  ------------------------------   ---------------------------------
  Drift detected that the          The migration is the opportunity
  configuration was wrong about    to fix the drift. Document in
                                   the change log.
  A new attribute was added        The provider added an attribute
  by the provider; the            that the old configuration did
  configuration needs the          not declare. The new attribute
  new attribute                    is non-destructive.
  A moved block refactored         The plan shows the move as a
  the address                      no-op; the plan output says
                                   (no action required).

The cases where it is not acceptable:

  Not acceptable                   Why
  ------------------------------   ---------------------------------
  Any line containing destroy      The migration was supposed to be
                                   non-destructive. Stop.
  Any update to a stateful         Databases, object storage with
  resource                         data, KMS keys: these updates
                                   are high-risk.
  An update to IAM or KMS          A non-destructive-looking update
  that changes the policy          can lock the team out.
  document

The bar is asymmetric. False positives (treating drift as a bug) cost time. False negatives (treating a bug as drift) cost data.

Production failure modes

1. Verification stops at Gate 1. The team accepts the empty plan and declares the migration done. The IAM role change broke the dashboards, but no one checked. Symptom: silent observability loss that surfaces as a missed incident weeks later.

2. The API cross-check uses the wrong credentials. The operator’s IAM role has access; the CI runner’s role does not. Symptom: Gate 2 passes for the operator and fails for the next apply from CI.

3. The git tag is pushed without the configuration that produced it. The tag references a commit hash, but the hash is not in the main branch. Symptom: a rollback that checks out the tag finds an empty repository.

4. The sign-off is announced in chat but not recorded in the change log. The next migration’s planner cannot find the verification record. Symptom: the same checks are repeated because no one knows they were done.

5. The non-empty plan is accepted without a documented reason. A future auditor asks why a stateful resource was updated and the team has no answer. Symptom: an audit finding that the migration lacked governance.

6. The observability check is skipped because the dashboards “look fine”. The dashboards cache data; the underlying scrape or log ship has been broken since the migration. Symptom: dashboards blank when the cache expires and the alerting has been silent for a week.

What to do in production

• Run all four gates for every batch. Gates 1 and 2 are the minimum. Gates 3 and 4 are what the on-call relies on.
• Tag every batch with a git tag that corresponds to the empty plan. The tag is the rollback input.
• Record the sign-off in the change log, not in chat. Chat is searchable for a week; the change log is searchable for the lifetime of the estate.
• Cross-check the API from the CI runner’s IAM role, not just the operator’s. The next apply runs from CI.
• Treat the observability check as a first-class step. The dashboards and alerts are part of the estate Terraform manages, even when Terraform does not declare them.

Verification

A migration is verified when all four gates pass. The commands below are the canonical checks:

# Gate 1: empty plan.
terraform plan -no-color -detailed-exitcode
echo "exit=$?"

No changes. Your infrastructure matches the configuration.
exit=0

# Gate 2: API cross-check, batch 7.
aws ec2 describe-instances \
    --instance-ids i-0123456789abcdef0 \
    --query 'Reservations[0].Instances[0].State.Name' \
    --output text
aws s3api head-bucket --bucket acme-logs-2024
echo "exit=$?"

running
exit=0

# Gate 3: observability.
aws cloudtrail describe-trails \
    --query 'trailList[?Name==`acme-audit`].IsMultiRegionTrail' \
    --output text
curl -fsS http://prometheus.acme.internal/api/v1/targets | \
    jq '[.data.activeTargets[] | select(.health=="up")] | length'

True
42

# Gate 4: sign-off artefact present.
git tag --list 'migration-batch-007-*'
git log --oneline -1

migration-batch-007-empty-plan
a1b2c3d Merge migration batch 7: empty plan verified

If any gate fails, the migration is not verified. The batch does not close, the next batch does not start, and the rollback target stays in the change log.