Operational anti-patterns — the cluster's runbook

The operational anti-patterns are the cluster’s runbook. The diagnostic is the anti-patterns, the impact, and the systematic approach to fixing them. The discipline is the same scale-free: every anti-pattern gets a fix.

The operational anti-patterns

The operational anti-patterns are:

• No backup. The workload has no backup; the data is not recoverable.
• No etcd test. The etcd snapshot is not tested; the restoration is not rehearsed.
• Manual edits. The cluster is managed by manual edits; the cluster’s state is not reproducible.
• No monitoring. The cluster has no monitoring; the failures are not detected.

flowchart TD
    A[Operational anti-patterns] --> B[No backup]
    A --> C[No etcd test]
    A --> D[Manual edits]
    A --> E[No monitoring]

The operational anti-patterns are the cluster’s runbook.

The diagnostic

The canonical diagnostic:

# Substitute your own values before running:
NS=velero
BACKUP_JOB=etcd-snapshot

# Use Polaris to detect anti-patterns
polaris audit --format yaml

# Check the backup
kubectl get cronjob -n "$NS" --field-selector metadata.name="$BACKUP_JOB"

# Check the etcd snapshot
ls -la /var/backups/etcd/

# Check the manual edits
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.annotations}{"\n"}{end}'

# Check the monitoring
kubectl get pods -n monitoring

The diagnostic is the Polaris audit, the backup, the etcd snapshot, the manual edits, and the monitoring.

The remediation

The remediation depends on the anti-pattern:

# Option 1: Set up the backup
kubectl apply -f backup-cronjob.yaml

# Option 2: Test the etcd snapshot
etcdutl snapshot restore /var/backups/etcd/snapshot.db --data-dir /var/lib/etcd-lab

# Option 3: Use GitOps
# (e.g., ArgoCD, Flux)

# Option 4: Set up the monitoring
kubectl apply -f monitoring-stack.yaml

The remediation is the anti-pattern fix.

Production discipline

The operational anti-patterns are the cluster’s hypothesis. The discipline is to walk the 11-step methodology applied to the anti-patterns, identify the cause, apply the remediation. The cluster’s discipline is the same scale-free: every anti-pattern gets a fix.

• Detect the anti-patterns with polaris audit. The audit names the ones the cluster actually has, and the backup CronJob, the snapshot directory, and the monitoring namespace answer for the rest.
• Run the detection in CI. A detection that runs only after an incident is a report, not a prevention.

Quiz