The deploy event log — the platform-side audit

The change-cause annotation marks the wall-clock moment of the deploy. The version label carries the immutable runtime identity. Neither surface records who triggered the deploy, which CI run produced the artifact, which approval was captured, or why the deploy succeeded or failed. The platform-side audit record is the deploy event log: the typed structured stream of deploy events emitted by the GitOps controller, the Kubernetes API, the CI system, and the central audit backend.

What the event log preserves

The annotation and the label are attached to the workload object and live as long as the object lives. The event log is a stream of typed structured records that travels with the deploy through the pipeline and lands in a queryable backend. The event carries the fields the annotation and the label cannot: the actor, the build URL, the approval chain, the deploy-failure reason, and the structured payload that the on-call engineer clicks through to recover.

flowchart LR
    A["Pipeline\ndeploy job"] -->|"GitOps sync"| B["ArgoCD / Flux\napplication event"]
    A -->|"kubectl apply"| C["Kubernetes\nEvents API"]
    A -->|"audit emit"| D["CI audit log"]
    A -->|"audit emit"| E["Central audit backend\ntamper-resistant"]
    B --> F["Event log\ntyped structured stream"]
    C --> F
    D --> F
    E --> F
    F --> G["Investigator\nclicks for full context"]

The four sources of the event log are complementary. The GitOps controller emits application events when the sync succeeds or fails; the event carries the commit SHA and the sync timestamp. The Kubernetes API emits Events for Deployment rollouts, Pod scheduling, and image pulls; the Events are queryable through kubectl get events. The CI audit log carries the structured record of the pipeline action. The central audit backend is the tamper-resistant store where the four streams converge.

The four sources

The GitOps controller event stream is the surface ArgoCD and Flux expose for application-level events. ArgoCD writes application events to its own notifications stream; the stream is queryable through the ArgoCD API and forwarded to Slack, PagerDuty, or webhook endpoints through the notifications controller. Flux writes events through its notifications controller to the same destinations. The event payload carries the revision, the sync status, and the health status.

The Kubernetes Events API is the surface the cluster exposes for resource-level events. Every Deployment rollout, every Pod scheduling decision, every image pull is recorded as an Event with a timestamp, a reason, and a message. The Events are queryable through kubectl get events --field-selector involvedObject.kind=Deployment and through the cluster’s event exporter. The Events record the runtime activity; they do not record the actor or the build URL.

The CI audit log is the structured record the pipeline emits at the moment of the action. The log carries the seven fields covered in lesson CXIII-05: timestamp, actor, action, target, SHA, build number, and environment. The log is forwarded to a central backend at the same step as the annotation and the label write.

The central audit backend is the tamper-resistant store where the streams converge. The backend is typically object storage with object lock (WORM - write once read many) accessed by a service identity with write-only permissions. The deployer role has no read, write, or delete access to the backend; the audit-writer role has write-only access; the auditor role has read-only access. The separation is structural.

What the event log answers that the marker does not

The annotation answers “why is this running”. The label answers “what version is running”. The event log answers four questions neither can:

• Who triggered the deploy? The event carries the actor identity - the user, the service account, the deploy key. The annotation carries only a human-readable message; the actor is not in the message.
• Which CI run produced the artifact? The event carries the build URL; clicking the URL opens the pipeline run with the full record of steps, inputs, secrets, and approvals.
• What was the approval chain? The event carries the approver identities and the timestamps of the approval actions; the postmortem six months later can reconstruct who approved what.
• Why did a deploy fail? The event carries the failure reason - the kubectl error, the pre-flight-validation failure, the resource-quota exceeded. The annotation and the label are written only when the deploy succeeds.

Retention and tamper-resistance

The event log is consulted for two reasons: postmortems (within weeks of the action) and compliance reviews (months to years after the action). A retention policy that covers the postmortem window but not the compliance window loses the second use case.

The right retention is at least one year. A year covers the seasonal incident patterns, the compliance audit cycles, and the personnel changes that make the engineer’s memory unreliable. A team that retains the log for 30 days has a log that answers “what happened this month” but not “what happened last quarter”.

The retention must be enforced at the storage layer, not at the application layer. A retention policy the application enforces can be bypassed by the application; a retention policy the storage layer enforces cannot. Object storage with object lock (WORM), accessed by a service identity with write-only permissions, is the structural primitive. The lifecycle policy moves aged objects to cold storage but never deletes them within the retention window.

Production discipline

1. Emit the deploy event from the same pipeline step as the annotation and the label. The three writes are atomic; an event without a marker is half a record.
2. Forward the event to a central backend with write-only access from the deployer. The identity separation is structural, not policy.
3. Retain the event log for at least one year at the storage layer. Application-layer retention is bypassable; storage-layer retention is not.
4. Carry the actor, the build URL, the approval chain, and the deploy-failure reason in the event payload. A future investigator needs all four.
5. Audit the event log access. Read access to the event log is itself an event that must be logged.

Cross-course references

• This course, Part CXIII-05 (AuditLogs) covers the seven fields the CI audit log must carry and the identity-separation defence.
• This course, Part CXI (KubernetesPipeline) covers the GitOps sync and the reconciliation events that feed the platform-side event stream.
• Linux for Production Sysadmins - Parts covering auditd and the syslog pipeline cover the host-level analogue of the platform-side event log.

Quiz