Audit log debugging — the troubleshooting patterns

The audit log debugging is the discipline of resolving the audit log issues. The common issues, the policy validation, the volume issues, the performance issues are the components. This lesson walks the debugging, the failure modes, and the production patterns.

The audit log issues

The audit log issues:

flowchart LR
    A[Audit log issue] --> B[Policy validation]
    A --> C[Volume issues]
    A --> D[Performance issues]
    A --> E[Webhook failures]

The issues are the failure modes.

The policy validation

The policy validation:

# Verify the audit policy is valid
kube-apiserver --audit-policy-file=/etc/kubernetes/audit-policy.yaml --dry-run

# Test the audit policy
kubectl create -f audit-policy-test.yaml --dry-run=server

# Verify the audit policy is mounted
ls -la /etc/kubernetes/audit-policy.yaml

The policy validation is direct.

The common policy issues

The common policy issues:

- Invalid YAML syntax
- Unknown kind "Policy" (should be audit.k8s.io/v1)
- Missing `omitStages` field
- Missing namespace selectors
- Missing resource selectors

The common issues are the taxonomy.

The volume issues

The volume issues:

# Check the disk usage
df -h /var/log/kubernetes

# Check the audit log files
ls -la /var/log/kubernetes/audit.log*

# Check the rotation
audit-log-maxsize: 100M
audit-log-maxbackup: 10
audit-log-maxage: 30

The volume issues are the disk constraints.

The performance issues

The performance issues:

# The audit log is overwhelming the kube-apiserver
- Reduce the audit level (Metadata instead of RequestResponse)
- Exempt the system service accounts
- Use the webhook backend with batching
- Use the sampling

The performance issues are the audit log volume.

The webhook failures

The webhook failures:

# Check the webhook backend
curl -I https://audit-collector.example.com/audit

# Check the kube-apiserver logs
journalctl -u kube-apiserver | grep -i audit

# Check the kube-apiserver flags
ps aux | grep kube-apiserver | grep audit

The webhook failures are the connectivity.

The audit log not being written

The audit log not being written:

# Check the audit policy
kubectl get --raw=/api/v1/namespaces

# Check the audit log
tail -f /var/log/kubernetes/audit.log

# Check the kube-apiserver logs
journalctl -u kube-apiserver

# Verify the audit log is enabled
ps aux | grep kube-apiserver | grep audit

The audit log not being written is the common issue.

The audit log too verbose

The audit log too verbose:

# Fix: reduce the audit level
rules:
- level: Metadata  # was RequestResponse
  resources:
  - group: ""
    resources: ["configmaps"]

The audit log too verbose is the performance issue.

The audit log missing events

The audit log missing events:

# Check the audit policy
cat /etc/kubernetes/audit-policy.yaml

# Check the audit level
kubectl get --raw=/api/v1/namespaces

# Check the audit log
tail -f /var/log/kubernetes/audit.log

# Check the webhook endpoint
curl -I https://audit-collector.example.com/audit

The audit log missing events is the data loss.

The production patterns

The production patterns:

flowchart LR
    A[Audit log] --> B[Validate the policy]
    B --> C[Test the backends]
    C --> D[Monitor the volume]
    D --> E[Monitor the performance]
    E --> F[Alert on issues]

The pattern is the production discipline.

The cross-course references

• The API server course covers the audit configuration.
• The Security course covers the audit policies.
• The Observability course covers the monitoring.

Quiz

Production discipline

• Validate the audit policy. The dry-run.
• Test the backends. The webhook.
• Monitor the volume. The disk usage.
• Monitor the performance. The audit log volume.
• Alert on issues. The audit log alerts.
• Document the debugging. The issues, the fixes.

The audit log debugging is the operational discipline. Operating it well is the policy validation, the volume monitoring, the performance monitoring, and the production patterns.