CI/CD Observability and Failures

Objective

Monitor CI/CD

What this lesson covers

• Alert on failures
• Plan the response

Why this matters in production

Production Terraform operations have blast radius. CI/CD Observability and Failures is one of the operational controls that determines whether a change is safe to apply. Skip this lesson at the cost of not understanding a production control.

How it works

Provide an overview of the topic. This lesson covers the ci/cd observability and failures concept, the underlying mechanism, and the production implications.

Key concepts.

• The terminology used in the topic.
• The mechanism that determines the operational behaviour.
• The production control that mitigates the operational risk.

Operational implications.

The ci/cd observability and failures concept affects the production change-management workflow. A change in this area has blast radius across the entire estate.

How to configure it

Apply the configuration pattern:

1. Set up the configuration block.
2. Validate the configuration.
3. Run the plan.
4. Verify the operations.
5. Document the change.

How to inspect it

Inspection pattern:

• Use the relevant CLI command to inspect the state.
• Verify the configuration matches the expectation.
• Confirm the plan is empty.

Validation

The validation pattern:

• Confirm the configuration is valid.
• Verify the plan is empty.
• Confirm the operations are correct.

Production failure modes

The following failure modes are the most common in production:

1. Misconfiguration of the relevant parameter.
2. Drift between the configuration and the real world.
3. Provider failure during the apply.
4. State corruption or loss.

How to recover

The recovery procedure:

1. Investigate the failure.
2. Identify the cause.
3. Apply the remediation.
4. Verify the state.
5. Document the incident.

Related runbooks

The following runbooks in this course apply to this lesson:

• terraform-runbook-investigate-state-lock
• terraform-runbook-recover-partial-apply
• terraform-runbook-investigate-provider-failure

What comes next

The next lesson in XXII-CI-CD continues the exploration of CI/CD Observability and Failures. Follow the order to maintain the production-first narrative.