State Boundaries as the Unit of Failure

Objective

Design state boundaries

What this lesson covers

• Identify the failure domain
• Plan the boundaries

Why this matters in production

Production Terraform operations have blast radius. State Boundaries as the Unit of Failure is one of the operational controls that determines whether a change is safe to apply. Skip this lesson at the cost of not understanding a production control.

How it works

Provide an overview of the topic. This lesson covers the state boundaries as the unit of failure concept, the underlying mechanism, and the production implications.

Key concepts.

• The terminology used in the topic.
• The mechanism that determines the operational behaviour.
• The production control that mitigates the operational risk.

Operational implications.

The state boundaries as the unit of failure concept affects the production change-management workflow. A change in this area has blast radius across the entire estate.

How to configure it

Apply the configuration pattern:

1. Set up the configuration block.
2. Validate the configuration.
3. Run the plan.
4. Verify the operations.
5. Document the change.

How to inspect it

Inspection pattern:

• Use the relevant CLI command to inspect the state.
• Verify the configuration matches the expectation.
• Confirm the plan is empty.

Validation

The validation pattern:

• Confirm the configuration is valid.
• Verify the plan is empty.
• Confirm the operations are correct.

Production failure modes

The following failure modes are the most common in production:

1. Misconfiguration of the relevant parameter.
2. Drift between the configuration and the real world.
3. Provider failure during the apply.
4. State corruption or loss.

How to recover

The recovery procedure:

1. Investigate the failure.
2. Identify the cause.
3. Apply the remediation.
4. Verify the state.
5. Document the incident.

Related runbooks

The following runbooks in this course apply to this lesson:

• terraform-runbook-investigate-state-lock
• terraform-runbook-recover-partial-apply
• terraform-runbook-investigate-provider-failure

What comes next

The next lesson in XV-Environment-Architecture continues the exploration of State Boundaries as the Unit of Failure. Follow the order to maintain the production-first narrative.