Emergency Manual Changes and Reconciliation

There are changes that have to happen in the console. The change-management workflow is down. The change window does not open for another six hours. The service is on fire and the runbook for this incident class says “do this with the console, now”. The question is not whether to make the change. The question is how to make it without orphaning the resource and without leaving a future responder confused.

This lesson is the manual reconciliation loop. It runs in parallel with the lessons on codification and ignore_changes. They are the three responses to a drift finding; this one is the closest to the console and the fastest to execute.

When the manual console change is right

Not every emergency warrants a console change. The bar:

• The change is time-critical. A production service is down or degrading in a way the runbook already describes.
• The Terraform workflow cannot run. The CI pipeline is broken, the state backend is unreachable, or the change window is too late to wait for.
• The change is reversible. Reverting it is a known procedure (not “delete the resource and recreate it later”). Console changes that are hard to revert are poor candidates.
• The change has a documented incident ID. A runbook that says “do this if the symptom matches” with an attached incident is a valid driver for a console change. Ad-hoc decisions are not.

Outside those conditions, the right answer is the change-management workflow. The console is a tool that bypasses approval gates. Used outside an emergency, it produces the drift this whole chapter exists to detect.

The reconcile loop

   1. Detect the emergency
        |
        v
   2. Open the incident; capture the ID
        |
        v
   3. Make the console change; note the IAM and timestamp
        |
        v
   4. Verify the live behaviour
        |
        v
   5. Run `terraform apply -refresh-only` to absorb
        |
        v
   6. Open a follow-up ticket for the HCL codification
        |
        v
   7. Document the incident, including the change

Six steps. The state is brought back in sync with the world by step 5. The configuration catches up in step 6. The audit trail is step 7.

Step 1: detect the emergency

A monitoring alert, an on-call page, a customer report, a Service Desk call. The trigger is rarely Terraform-shaped; it is the symptom. The detection produces the incident ID that the rest of the loop references.

Step 2: open the incident, capture the ID

Every console change that does not have an incident ID is a suspect console change. The incident ID anchors the audit trail to the reason.

# READ-ONLY: opens a PagerDuty incident (or whatever tool you use).
pd-cli incident create \
  --service "prod-network" \
  --title "ALB failing health checks" \
  --urgency high
# Output: INCIDENT_ID=IN-4392

The incident ID goes into every subsequent step: the console change ticket, the audit log query, the follow-up code PR, the post-incident review.

Step 3: make the console change

The change itself. Three rules:

• Use the dedicated incident IAM. Most mature shops have a break-glass role that assumes the on-call’s identity for the duration of the incident. The audit log records the role, not a personal account.
• Annotate the change with the incident ID. Tag the resource, set a description field, add a console-side comment - whichever the provider supports. A future investigator reading the audit log will see the change attributed to a role, and the description will point at the incident.
• Note the timestamp and the change target. “Changed ALB security group rule at 22:14 GMT to allow traffic from 10.0.0.0/8 for incident IN-4392.”

# Break-glass; assumes incident role.
aws sts assume-role \
  --role-arn "arn:aws:iam::111122223333:role/incident-break-glass" \
  --role-session-name "INC-4392-jdoe"

export AWS_ACCESS_KEY_ID=ASIA...
export AWS_SECRET_ACCESS_KEY=...
export AWS_SESSION_TOKEN=...

# The actual change.
aws ec2 authorize-security-group-ingress \
  --group-id sg-0abc123 \
  --protocol tcp --port 443 --cidr 10.0.0.0/8

The role is critical. Personal credentials in a personal shell with no incident context produce an audit trail that the auditor cannot trust. The break-glass role and the session name give the auditor something to anchor to.

Step 4: verify the live behaviour

Before touching Terraform, verify the change did what the incident required. A health check that was failing should be passing. A connectivity test should succeed. The incident responder has eyeballs on the symptom now; this is the moment to confirm the symptom cleared.

# READ-ONLY: confirm the security group now has the rule.
aws ec2 describe-security-groups \
  --group-ids sg-0abc123 \
  --query 'SecurityGroups[0].IpPermissions'

If the symptom is not cleared, the change was not the right change. Roll back via console, go back to step 1, try something else. Do not touch Terraform yet.

Step 5: absorb the change with apply -refresh-only

Now Terraform finds out. A refresh-only apply reads the live API, writes the changed attribute back into the state, and proposes zero changes against the configuration.

# READ-ONLY against the API; writes the refreshed state.
terraform apply -refresh-only -auto-approve

Output:

aws_security_group.alb_sg: Refreshing state... [id=sg-0abc123]

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

The state file now records the new CIDR. The configuration still says the old CIDR. The next terraform plan (without -refresh-only) will propose a destructive change to put the CIDR back. That is correct: the configuration has not caught up yet.

Step 6: open a follow-up ticket for HCL codification

apply -refresh-only does not change the configuration. It only updates the state. The drift still exists in HCL.

A follow-up ticket captures the codification that needs to happen in normal working hours:

Ticket:    HCL-1294 - Codify IN-4392 ingress change
Owner:     networking team
Linked:    INC-4392, DT-2026-08-13-002
Action:    Update aws_security_group.alb_sg to allow
           10.0.0.0/8 ingress per incident response.
Out of scope: extending the CIDR list beyond what the
           incident required.

The PR that closes this ticket brings the configuration in line with the world. After the PR merges and the next regular apply runs, the drift is gone.

Step 7: document the incident

The post-incident review (PIR) records:

• The change that was made and the symptom it addressed.
• The IAM role used and the session name.
• The audit log entry hash.
• The follow-up ticket (HCL-1294 in the example).
• The estimate of time saved by skipping the change window.

The PIR is the artefact the auditor reads first when asking “what happened during IN-4392?”.

What this loop must not do

Two fences, again:

• Do not edit the state file directly. The state file is a JSON document, but it is not a human-editable document. A wrong field name and the state becomes unusable. apply -refresh-only writes the state from the API, which is the correct path.
• Do not skip the codification PR. State and configuration drift apart again within a week. The follow-up ticket is the line in the sand that says “this is temporary, and we will close it”.

Verification

# 1. Confirm the incident was opened and ID captured.
pd-cli incident show INC-4392 | grep -E 'status|opened'

# 2. Confirm the console change has an audit-log entry.
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=ResourceName,AttributeValue=sg-0abc123 \
  --max-results 1 --output json | jq '.Events[0].EventName'
# Expected: "AuthorizeSecurityGroupIngress"

# 3. Confirm apply -refresh-only absorbed the change.
terraform apply -refresh-only -auto-approve
terraform state show aws_security_group.alb_sg | grep -E 'cidr|10\.0\.0'
# Expected: the new CIDR appears in the state.

# 4. Confirm the codification ticket exists.
grep -E 'HCL-1294|codify' backlog/hcl-followup.md

To confirm the lesson:

• You can run the seven-step reconciliation loop on a realistic emergency.
• You can produce the right IAM evidence (incident role, session name) for the audit log.
• You can refuse the two fences (direct state edits and untracked follow-ups).