Artifact retention and storage — retention policies, the cost, the production rules

Retention is the cost/audit tradeoff. The artifact store is a per-run store with a bounded retention window; the window is the cost multiplier, and the window is the audit window. A team that sets retention too short loses audit; a team that sets retention too long inflates cost. The production discipline is to set retention explicitly on every upload, design the retention to match the audit window, and archive to long-term storage for compliance windows longer than the vendor maximum.

The cost model

The artifact store charges per GB-month. The cost of a 500 MB binary retained for 90 days is roughly 1.5 GB-months; the cost of the same binary retained for 365 days is roughly 6 GB-months. The cost of 50 runs per day of the same binary is 50x the single-run cost. The production pipeline that uploads many large binaries routinely can produce a large storage bill through retention alone.

flowchart LR
    A["Single 500 MB binary"] --> B["90 days: 1.5 GB-month"]
    A --> C["365 days: 6 GB-month"]
    D["50 runs per day"] --> E["3 GB-month/day at 90 days"]
    D --> F["12 GB-month/day at 365 days"]
    E --> G["90 GB-month/month"]
    F --> H["360 GB-month/month"]

The numbers are illustrative; the principle is that retention multiplies linearly with both the artifact size and the run frequency. Two ways to reduce cost: shorten the retention, or archive to cheaper storage.

The audit window

The audit window is the time the team must be able to retrieve the artifact for audit purposes. Three typical windows:

• Application builds. 90 days. The audit window is the compliance window for routine application changes.
• Infrastructure changes. 365 days. The audit window is the compliance window for production infrastructure changes.
• Compliance-bound builds. 7 years (or whatever the regulator requires). The audit window is the regulator’s retention rule.

The vendor’s maximum retention is bounded by vendor policy, often 90 days. A compliance window longer than the vendor maximum requires external archival.

The four production rules

The four production rules for artifact retention and storage:

1. Set retention-days explicitly. The default is per-vendor policy, often 90 days. The production rule is to set the retention based on the audit window, not to rely on the default.

- name: Upload plan
  uses: actions/upload-artifact@v4
  with:
    name: terraform-plan
    path: tfplan
    retention-days: 365

2. Archive to long-term storage for compliance windows longer than the vendor maximum. The artifact store is not a compliance archive. A pipeline that needs seven-year retention must archive to S3 with versioning, an internal artifact registry, or a WORM (write-once-read-many) store.

- name: Archive to long-term storage
  if: always()
  run: |
    aws s3 cp tfplan s3://acme-terraform-plans/${ github.sha }/tfplan \
      --storage-class STANDARD_IA

3. Audit storage usage quarterly. A pipeline that uploads large binaries routinely can produce a storage bill that grows faster than the team expects. The audit step is to list the artifact stores, identify the largest artifacts, and reduce retention where the audit window allows.

4. Never store secrets in artifacts. The artifact store is accessible to anyone with workflow read permission. A secret in an artifact is a secret in a public-by-default store. Use the secrets manager for sensitive values; use artifacts for non-sensitive payloads only.

flowchart TB
    subgraph RULES["Production rules"]
        R1["Set retention-days explicitly"]
        R2["Archive to long-term storage for compliance"]
        R3["Audit storage usage quarterly"]
        R4["Never store secrets in artifacts"]
    end
    subgraph WINDOWS["Audit windows"]
        W1["Application builds: 90 days"]
        W2["Infrastructure changes: 365 days"]
        W3["Compliance-bound: 7 years (external archive)"]
    end
    R1 --> W1
    R1 --> W2
    R2 --> W3

Production discipline

1. Set retention-days explicitly on every upload. The default is per-vendor policy; the production rule is the audit window the team needs.
2. Archive to long-term storage for compliance windows longer than the vendor maximum. The artifact store is not a compliance archive.
3. Audit storage usage quarterly. A pipeline that uploads large binaries routinely can produce a storage bill that grows faster than the team expects.
4. Never store secrets in artifacts. The artifact store is accessible to anyone with workflow read permission.
5. Document the retention window in the team’s runbook. A future maintainer should not have to re-derive the audit window from the vendor policy.

Cross-course references

• Git, CI/CD & GitOps — Part XLIV-01 (What an artifact is) covers the artifact model.
• Git, CI/CD & GitOps — Part XLIV-02 (Build outputs and binary artefacts) covers the upload limits and the reproducibility invariant.
• Linux for Production Sysadmins — Part XXXIV (ConfigMgmt) applies the same retention model to package archives.

Quiz