Audit logs from the pipeline — who did what

The deploy annotation marks the wall-clock moment. The deploy event carries the structured record. The CI run log records the operational output - the steps the pipeline executed, the commands the runner ran, the artifacts the runner produced. None of the three is the compliance record. The compliance record is the audit log: the tamper-resistant, retention-guaranteed, forwarded-to-central-storage record of every pipeline action.

The seven fields the audit log must capture

A pipeline audit log is not a run log. A run log records what the pipeline did; an audit log records who triggered what action against what target. The distinction is the difference between an operational record (debug) and a compliance record (reconstruct).

The seven fields the audit log must carry:

1. Timestamp - the wall-clock moment the action was triggered, in UTC.
2. Actor - the identity that triggered the action (user, service account, deploy key).
3. Action - the verb (deploy, rollback, promote, approve, reject).
4. Target - the object of the action (service name, environment, region).
5. Commit SHA - the immutable code reference.
6. Build number - the CI run identifier.
7. Environment - the cluster, account, or region the action affected.

flowchart LR
    A["Pipeline action"] --> B["Audit log entry\n7 fields"]
    B --> C["Central store\ntamper-resistant"]
    C --> D["Query surface\nfor postmortems"]
    C --> E["Compliance surface\nfor auditors"]

The seven fields together answer “who deployed what, when, with whose approval, and from which build”. A postmortem six months later can reconstruct the full action without relying on the memory of the engineer who triggered it.

NAME=api
SHA=$(git rev-parse --short HEAD)
BUILD=4527
ACTOR=octocat
ACTION=deploy
ENV=production
MESSAGE="deploy $SHA release v3.4.1"
kubectl annotate deploy $NAME \
  kubernetes.io/change-cause="$MESSAGE" --overwrite
# emit audit log entry alongside the deploy event
curl -X POST "$AUDIT_ENDPOINT" \
  -H "Authorization: Bearer $AUDIT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "$(cat <<'JSON'
{
  "timestamp": "$(date -u +%FT%TZ)",
  "actor": "$ACTOR",
  "action": "$ACTION",
  "target": "$NAME",
  "sha": "$SHA",
  "build": "$BUILD",
  "environment": "$ENV"
}
JSON
)"

Forwarding to the central backend

The audit log is not a file in the CI system. The CI system is the producer; the central observability backend is the store. The log is forwarded at the moment of the action, with the same atomicity as the deploy event and the annotation. A log that lives only in the CI system is a log that is lost when the CI system is migrated, when the vendor changes, or when the system is breached.

The forwarding discipline is the same as the event discipline from the previous lesson. The audit log travels with the action; the action is not complete until the log is stored. A pipeline that emits the log best-effort and continues regardless has a log that is unreliable.

Retention that survives personnel changes

The audit log is consulted for two reasons: postmortems (within weeks of the action) and compliance reviews (months to years after the action). A retention policy that covers the postmortem window but not the compliance window loses the second use case.

The right retention is at least one year. A year covers the seasonal incident patterns, the compliance audit cycles, and the personnel changes that make the engineer’s memory unreliable. A team that retains the log for 30 days has a log that answers “what happened this month” but not “what happened last quarter”.

The retention must be enforced at the storage layer, not at the application layer. A retention policy that the application enforces can be bypassed by the application; a retention policy that the storage layer enforces cannot. Object storage with a lifecycle policy, a write-only bucket, or a SIEM with retention rules is the right primitive.

What the audit log is not

Three things are commonly mistaken for the audit log and are not:

• The CI run log. The run log records the steps the pipeline executed; the audit log records the action the user triggered. The run log is operational; the audit log is compliance.
• The Git commit history. The commit history records what code was committed; the audit log records what code was deployed. The two diverge when cherry-picks, reverts, or out-of-band deploys occur.
• The Kubernetes audit log. The Kubernetes audit log records API server actions; the pipeline audit log records pipeline actions. The two overlap when the pipeline calls the Kubernetes API; they diverge when the pipeline acts through other channels (cloud APIs, Terraform Cloud, ArgoCD).

Production discipline

1. Capture the seven fields for every pipeline action. The action is not complete until the log is written.
2. Forward the log to a central store with write-only access from the deployer. The separation is structural, not policy.
3. Enforce retention at the storage layer for at least one year. Application-layer retention is bypassable; storage-layer retention is not.
4. Distinguish the CI run log from the audit log in documentation and in tooling. The two are not interchangeable.
5. Audit the audit log access. Read access to the audit log is itself an event that must be logged.

Cross-course references

• Observability course - Part CX (MajorIncidents) covers the annotation sources that complement the audit log with operational context.
• This course, Part LXIV (AuditChain) covers the deployment claim, the deployment receipt, and the six-month reconstruction that the audit log makes possible.
• This course, Part LXIIX (RepositoryAudit) covers the repository-level audit that complements the pipeline-level audit log.
• Linux for Production Sysadmins - Parts covering auditd and the syslog pipeline cover the host-level analogue of the pipeline audit log.

Quiz