The observability flow — what we see and what we do not

Observability and auditability are often conflated, but they answer different questions. Observability answers “what is happening?” - what the system is doing now, what it did ten seconds ago, what it did last Tuesday at 14:23. Auditability answers “why is this running?”

• which commit, which pipeline, which approver, which rationale. A team with rich observability but no auditability sees the symptom but not the cause. A team with auditability but no observability has the history of decisions but no view of the runtime. The reference architecture requires both; the unification is the audit trail.

The four signals

flowchart LR
    A["Cluster runtime"] -->|"emit"| B["Metrics"]
    A -->|"emit"| C["Logs"]
    A -->|"emit"| D["Traces"]
    A -->|"annotations"| E["Audit trail\n(receipts)"]
    B --> F["What is happening"]
    C --> F
    D --> F
    E --> G["Why this is happening"]

The four signals:

• Metrics. Numerical samples aggregated over time: latency, error rate, throughput, saturation.
• Logs. Discrete events with a timestamp and payload.
• Traces. Distributed spans correlating a request across services.
• Audit trail. A reconstruction of decisions from receipts, PRs, and change tickets.

The first three are runtime signals; the fourth is a derived signal whose source is the receipts stamped at apply time.

What observability sees

Observability sees the runtime. A Prometheus query returns a time series of latency samples. A Loki query returns log lines. A Tempo query returns a trace. None tells the auditor why the latency spiked at 14:23.

NS=payments
APP=checkout
PROMETHEUS=https://prom.example.com
curl -G --data-urlencode "query=histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket{app=\"$APP\",ns=\"$NS\"}[5m])))" \
  "$PROMETHEUS/api/v1/query" | jq .

The query returns a percentile; the operator’s next question is “why?”. The query engine does not answer; observability was never designed to.

What auditability sees

Auditability sees the decisions. The receipts on the running resources record what was applied and when. The PRs record what was reviewed and by whom. The change tickets record what was approved and why.

NS=payments
APP=checkout
kubectl get deployment "$APP" -n "$NS" \
  -o jsonpath='{.items[*].metadata.annotations.deploy\.time/revision}'

The output is the commit SHA. From there, the auditor walks to the commit, the PR, the approver, and the ticket.

The unification

The unification is a queryable audit trail whose entries are joined on a common key. The key is the receipt’s revision annotation: the commit SHA. Given a runtime anomaly (spike in error rate), the operator queries the audit trail for receipts with that revision and retrieves the PR, the approver, and the rationale. Given an audit question (which commit is running?), the auditor queries the runtime for the receipt.

APP=checkout
NS=payments
REV=$(kubectl get deployment "$APP" -n "$NS" \
  -o jsonpath='{.items[*].metadata.annotations.deploy\.time/revision}')
echo "running commit: $REV"
gh pr list --state merged --search "$REV" --json url,title,number,mergedBy,body

The first reads the receipt; the second queries the forge. The join key is the commit SHA.

A team that has observability and auditability but does not join them has two parallel systems that cannot answer questions that span both.

Production discipline

1. Observability and auditability are separate systems with a join key. Do not collapse them; the join key is what closes the gap.
2. Receipts are stamped at apply time, not retroactively. A receipt stamped after an incident is a receipt whose contents the operator chose.
3. The audit trail is queryable, not human-readable. A wiki page is not an audit trail.

Cross-course references

• This course, Part LXIV (Auditability) - the six links the audit trail walks.
• This course, Part CVI (ChangeMgmt) - the change record whose queries feed the audit trail.
• Kubernetes for Production Sysadmins - Parts XXXVIII-XL cover the runtime signals and the audit annotations.

Quiz