Cleanup on failure — the cleanup path when a job fails midway

A job that runs halfway and fails leaves residue. A temporary credential is still valid. An artifact is partially uploaded. A test database has rows from a test that did not finish cleaning up after itself. A lock acquired by the failing job is held until the lock TTL expires. None of these is the deploy, none of these is the change, and none of these can be reversed by a rollback of the change. They are local residue from the job’s execution, and the pipeline that does not clean them up accumulates a stateful mess that the next run inherits.

Cleanup is not rollback

The first confusion to clear up is that cleanup and rollback are different operations:

• Rollback. Undo the change that was applied to the production system. Rollback recovers from a bad deploy. The change is reverted; the production system returns to its previous good state.
• Cleanup. Remove the residue the running job left behind in the build, test, and publish infrastructure. Cleanup recovers from a failed execution. The production system is unaffected; the build runner, the test environment, the artifact registry, and the lock service are brought back to a known state.

A job that failed before the deploy ran still needs cleanup. The temporary credential issued for that deploy is still valid; it will expire eventually, but until it does it is a credential the next job should not see. A test that failed mid-fixture still has rows in the test database; the next test that runs against the same database will see those rows.

The four kinds of residue

The residue a failed job leaves behind falls into four categories:

flowchart LR
    A["Failed job"] --> B["Credentials"]
    A --> C["Partial artifacts"]
    A --> D["Test data"]
    A --> E["Locks and leases"]
    B --> F["Cleanup stage"]
    C --> F
    D --> F
    E --> F
    F --> G["Known clean state"]

• Credentials. Short-lived credentials issued by a vault or STS to the running job. They have a TTL but the TTL is rarely tight enough to revoke the credential before the next job needs the namespace to be clean.
• Partial artifacts. An artifact upload that succeeded for two of three parts; the third part is missing; the artifact is now an unreadable stub that the next pipeline run will see in its list of available artifacts.
• Test data. Rows inserted by a test fixture that did not run its teardown; files written to a shared test volume; messages queued on a broker that the failing test did not drain.
• Locks and leases. A mutex held by the failing job because the job did not reach the lock-release line; a TTL on the lock saves the cluster eventually but every retry wastes a wait on the TTL.

Each kind of residue has its own cleanup function, and a job that produces more than one kind of residue needs a cleanup function for each.

The if: always() pattern

The platform-independent way to guarantee cleanup runs whether the job succeeded or failed is the if: always() executor:

jobs:
  deploy:
    steps:
      - run: ./deploy.sh
      - if: always()
        run: ./cleanup.sh

The cleanup step runs whether ./deploy.sh succeeded, failed, or was skipped. The condition is “this step’s prerequisites have been evaluated”, which is the platform’s way of saying “regardless of the previous step’s exit code”.

GitLab CI calls this after_script:

deploy:
  script: ./deploy.sh
  after_script: ./cleanup.sh

The after_script block runs whether the script block succeeded or failed. There is also before_script, which runs on every invocation; the discipline is to put cleanup in after_script, not before_script, because the cleanup must run on failure, not just on the next invocation.

Jenkins Pipeline’s declarative syntax uses post:

pipeline {
  stages {
    stage('deploy') { steps { sh './deploy.sh' } }
  }
  post {
    always { sh './cleanup.sh' }
  }
}

The always block is the Jenkins equivalent of if: always(). The post block also supports success, failure, unstable, and aborted for finer-grained cleanup that depends on the outcome.

What cleanup must not do

Cleanup runs whether the deploy succeeded or failed. A cleanup step that depends on the deploy having succeeded is a cleanup step that does not run when it is most needed. The discipline is to write cleanup that assumes nothing about the deploy’s outcome:

#!/usr/bin/env bash
set -eu
REVOKE_CRED="${REVOKE_CRED:-true}"
DELETE_PARTIAL="${DELETE_PARTIAL:-true}"
PURGE_TEST_DATA="${PURGE_TEST_DATA:-true}"
RELEASE_LOCKS="${RELEASE_LOCKS:-true}"

if [ "$REVOKE_CRED" = "true" ]; then
    vault token revoke -self || true
fi
if [ "$DELETE_PARTIAL" = "true" ]; then
    aws s3 rm "s3://artifacts/$JOB_ID-partial" || true
fi
if [ "$PURGE_TEST_DATA" = "true" ]; then
    psql "$TEST_DATABASE_URL" -c "TRUNCATE test_temp;" || true
fi
if [ "$RELEASE_LOCKS" = "true" ]; then
    consul lock release -name "$JOB_LOCK_NAME" || true
fi

Each cleanup step uses || true so a failure in cleanup does not mask the original failure. The variable flags allow the cleanup to be tuned per pipeline without editing the script.

Cleanup under failure versus cleanup under success

A common antipattern is cleanup that does the wrong thing on success:

• Cleanup under failure. Remove partial artifacts, revoke credentials, release locks. This is the recovery path.
• Cleanup under success. Promote the artifact to the long-term registry. This is not cleanup - this is the publish path - and putting it in the cleanup step means a retried job attempts to publish the artifact twice.

The discipline is to separate the two: the cleanup step removes residue, the publish step promotes artifacts, and neither does the other’s job.

Production discipline

1. Cleanup is if: always(). The step runs whether the job succeeded or failed.
2. Cleanup is a separate script, tested separately. A cleanup script that runs only on the happy path is not a cleanup script.
3. Cleanup does not promote. The cleanup step removes residue; the publish step promotes artifacts.
4. Cleanup logs to a known location. A cleanup that ran but produced no log is a cleanup that did not run from the incident-response perspective.
5. Cleanup failures do not mask the original failure. || true on each cleanup operation, with the failure logged separately.

Cross-course references

• This course, Part LXI-02 (Retries) covers why cleanup must run between retries as well as at the end of the job.
• This course, Part LXI-04 (PartialDeployment) covers the partial-deploy residue that cleanup must distinguish from the deploy itself.
• Linux for Production Sysadmins - Part XXXII (IncidentResponse) covers the discipline of treating cleanup as a first-class operational stage.

Quiz