Responding to a Production Incident

Objective

Stay calm

What this lesson covers

• Follow the procedure
• Document the response

Why this matters in production

Production Terraform operations have blast radius. Responding to a Production Incident is one of the operational controls that determines whether a change is safe to apply. Skip this lesson at the cost of not understanding a production control.

How it works

Provide an overview of the topic. This lesson covers the responding to a production incident concept, the underlying mechanism, and the production implications.

Key concepts.

• The terminology used in the topic.
• The mechanism that determines the operational behaviour.
• The production control that mitigates the operational risk.

Operational implications.

The responding to a production incident concept affects the production change-management workflow. A change in this area has blast radius across the entire estate.

How to configure it

Apply the configuration pattern:

1. Set up the configuration block.
2. Validate the configuration.
3. Run the plan.
4. Verify the operations.
5. Document the change.

How to inspect it

Inspection pattern:

• Use the relevant CLI command to inspect the state.
• Verify the configuration matches the expectation.
• Confirm the plan is empty.

Validation

The validation pattern:

• Confirm the configuration is valid.
• Verify the plan is empty.
• Confirm the operations are correct.

Production failure modes

The following failure modes are the most common in production:

1. Misconfiguration of the relevant parameter.
2. Drift between the configuration and the real world.
3. Provider failure during the apply.
4. State corruption or loss.

How to recover

The recovery procedure:

1. Investigate the failure.
2. Identify the cause.
3. Apply the remediation.
4. Verify the state.
5. Document the incident.

Related runbooks

The following runbooks in this course apply to this lesson:

• terraform-runbook-investigate-state-lock
• terraform-runbook-recover-partial-apply
• terraform-runbook-investigate-provider-failure

What comes next

The next lesson in XXIX-Incident-Response continues the exploration of Responding to a Production Incident. Follow the order to maintain the production-first narrative.