The observability and audit fleet — the operational visibility

The observability and audit fleet is the lens through which the other four fleets become queryable. Observations tell the team what the system is doing; attestations tell the team why it is running that way. A team that has metrics, logs, and traces but no audit trail sees the symptom but not the cause. A team that has the audit trail but no metrics has the history of decisions but no view of the runtime. The reference architecture requires both; the unification is the trail.

The four primitives

A production observability and audit fleet depends on four primitives. Forgetting any one of them turns the operations plane into a place where data exists but answers do not.

flowchart LR
    CLUS["Cluster"] --> MET["Metrics"]
    CLUS --> LOG["Logs"]
    CLUS --> TRC["Traces"]
    ACD["Argo CD"] --> RCPT["Deployment receipt"]
    WF["Actions run"] --> BLD["Build receipt"]
    MAN["Manifest repository"] --> GIT["Git history"]
    RCPT --> STORE["Receipt store"]
    BLD --> STORE
    MET --> PROM["Prometheus"]
    LOG --> LOKI["Loki"]
    TRC --> TEMPO["Tempo"]
    STORE --> AUD["Audit trail — derived"]
    GIT --> AUD
    PROM --> AUD
    LOKI --> AUD
    TEMPO --> AUD

• Metrics. Quantitative signals with a label dimension. Prometheus is the default; the metric is queryable by labels and time. The fleet ships metrics from every component: the runner’s queue depth, the Argo CD controller’s reconcile duration, the cluster’s node-state gauge.
• Logs. Discrete events with a timestamp and a structured payload. Loki is the default; logs join to metrics by label and to traces by trace ID. The fleet ships logs from every component: the workflow step, the controller reconcile, the kubelet’s event.
• Traces. Distributed requests across components. Tempo (or an OTel-compatible backend) is the default. The trace connects a request across the workflow, the registry call, the controller pull, and the cluster apply.
• Audit trail. The derivation that joins receipts to git history to runtime signals. Not a separate primitive in the runtime sense - it is a query over the other three plus the receipt store - but the primitive the operations team depends on when an incident asks “why is this running here?”.

The audit trail is derived, not pushed

The audit trail is not a fourth data store with its own ingestion path. It is a query that joins three data stores: the receipt store, git history, and the runtime signals. Three reasons this matters:

• Push-based audit trails lose data at the boundary. A separate audit-pipeline that copies receipts into an audit database must be kept in sync; gaps in that pipeline are gaps in the trail.
• Derived trails survive schema evolution. A new field on the receipt is a schema change in the receipt store and a backwards-compatible change in the query. A push-based trail would require a migration in the audit pipeline.
• The query defines the trail. Different incidents ask different questions, and a derived trail lets each question have its own query without a separate ingestion path per question.

The downside is that the query must be fast. A trail that takes ten minutes to answer “which commit reached which cluster” is a trail nobody uses during incidents. That is why the receipt store is indexed on the fields the question needs (cluster, application, applied_at) and the git history is queried with a stable diff hash.

Cross-store query

The query an incident responder actually runs looks like this:

"Which commit reached the production cluster in the last
24 hours, and which deploy stamp did it produce on each
node?"

The answer joins:

• The receipt store by (cluster=prod, application in [...]) on applied_at in the window.
• The git history by (commit in [...]) joined to the manifest at that commit.
• The runtime signals by (node, label) joined to applied_at.

A single SQL or LogQL query against a federated backend produces the answer; the same query against four independent stores does not. A federated query layer - Tempo, Loki, Prometheus, with a Grafana panel or an MCP server - is what makes the trail queryable.

Operating the fleet

A production observability and audit fleet is operated as a product with three recurring concerns:

• Retention. Observations (metrics, logs, traces) follow the standard observability retention curves: high resolution for a week, lower resolution for a quarter, archived for a year. Receipts follow the audit retention window: keep receipts for the longer of the compliance window or the engineering review window.
• Cost. Logs and traces are the big spenders. A fleet that ships every kubelet log line to a year of retention is a fleet whose bill will eventually reshape itself out of existence. Sampling strategies belong at the application level: log only the events the audit trail needs, not every event the runtime emits.
• Cross-store query latency. The query that joins receipts to git to runtime must answer in seconds, not minutes. A federated query layer that fans out to four stores with synchronous joins is too slow for incident use; a layer with cached indices on the hot path is the production discipline.

echo "Inspect a recent Argo CD receipt on the production cluster:"
argocd app history "${APP}" --server "${PROD_SERVER}" \
  | jq '.[] | select(.deployedAt > (now - 86400 | todate))'

The command above is the engineer’s hands-on path during incidents: list the deploys in the last 24 hours, get the digests and the source commits, and walk them through the git history.

Production discipline

1. The four primitives are mandatory. A team without traces during an incident is a team that walks the trail by hand.
2. The audit trail is derived, not pushed. Keep ingestion simple; invest in the query layer.
3. Receipts are indexed for the hot query. Cluster, application, applied_at - the fields the question needs are the fields the index has.
4. Retention matches the audit window. A receipt that is dropped before the audit window is a receipt the audit team does not have.

Cross-course references

• This course, Part LXIV (Auditability) - the six links the trail reconstructs.
• This course, Part LXXI (Observability stack) - the runtime signals the trail joins to.
• Kubernetes for Production Sysadmins - Parts XXXVIII-XLI cover the cluster-side observability this lesson assumes.

Quiz