Recovering secrets — the secret store; the rotation

Secrets are the credentials the workflows need to deploy to production, sign artifacts, and authenticate to external systems. A CI/CD disaster that destroys the secret store removes the workflows’ ability to act. The recovery is the third step in the CI/CD DR, after the control plane and the runners are rebuilt.

flowchart LR
    A["secret store recovery"] --> B["external store path"]
    A --> C["control-plane store path"]
    B --> D["Vault unsealed"]
    B --> E["AWS Secrets Manager restored"]
    C --> F["gh secret set invoked"]
    C --> G["GitLab CI variables restored"]
    D --> H["jobs can authenticate"]
    E --> H
    F --> H
    G --> H
    H --> I["rotate every recovered secret"]

The recovery path depends on whether the secret store lives in the control plane (lost with the orchestrator) or in an external service (recoverable independently).

External secret store

An external secret store — HashiCorp Vault, AWS Secrets Manager, Google Secret Manager — runs in the team’s own infrastructure or in a cloud service separate from the control plane. The store survives the loss of the control plane because the store is not part of the control plane.

The recovery:

1. Vault unseal. A Vault cluster is sealed after a restart; the unseal operation requires the unseal keys. Three of five keys (in a typical configuration) are required.
2. AWS Secrets Manager restore. A cross-region replica is promoted to primary. The replication lag defines the RPO.
3. Audit the access log. Every secret read during the recovery window is logged and reviewed.

Control-plane secret store

A control-plane secret store — GitHub Actions Secrets, GitLab CI Variables — lives inside the control plane and is lost with it. The recovery re-enters the secrets from the password manager once the control plane is rebuilt.

The re-entry command for GitHub Actions Secrets:

gh secret set SECRET_NAME

The command prompts for the value; the value is read from the password manager. For automation, the value is read from a file:

gh secret set SECRET_NAME < $SECRET_VALUE_FILE

The $SECRET_VALUE_FILE is a file the team holds only during the recovery; the file is deleted after. A file that lives beyond the recovery is a file the next incident will find.

The rotation discipline

Every secret that is recovered from the password manager is rotated after the recovery completes. The recovered secret is the secret the team believes is current; the rotation produces a secret the team knows is current.

The rotation sequence:

1. Generate the new credential at the source (cloud provider, registry, deploy identity).
2. Distribute the new credential to the password manager and to the rebuilt secret store.
3. Invalidate the old credential at the source.
4. Verify the old credential is rejected.

The runbook includes the rotation as a final step.

Production discipline

1. Secrets live in three places, not two. The password manager (source of truth), the secret store (consumed by workflows), and the cloud (validated). All three must agree before a workflow runs.
2. Re-entry uses the password manager, not memory. A secret typed from memory is a secret that has drifted.
3. Every recovered secret is rotated. The recovery produces the credential the team believes is current; the rotation produces one the team knows is current.

Cross-course references

• Git, CI/CD & GitOps — Part XCIV (Secret-Leak Incident Response) covers the rotation discipline.
• Git, CI/CD & GitOps — Part XCVII-02 (Recovering the Control Plane) covers the control-plane recovery.
• Linux for Production Sysadmins — Part XXXIV (Config Management) covers the password manager.

Quiz