The sysadmin troubleshooting posture — habits, tools, and ergonomics

The sysadmin posture is the muscle memory of the 11-step methodology, the tooling stack that turns the methodology into a single command, and the on-call ergonomics that make the methodology sustainable. The posture is the difference between an operator who runs the methodology under pressure and an operator who applies the wrong fix.

The tooling stack

The production-grade Kubernetes sysadmin carries a small tooling stack:

• kubectl — the canonical Kubernetes CLI.
• kubectx — fast context switching across clusters.
• kubens — fast namespace switching within a cluster.
• stern — multi-pod log tailing.
• kube-ps1 — the current context in the prompt.
• fzf — fuzzy history search for kubectl commands.

# .bashrc / .zshrc
source <(kubectl completion bash)
alias k=kubectl
alias kg='kubectl get'
alias kgp='kubectl get pods'
alias kd='kubectl describe'
alias kl='kubectl logs'
alias kex='kubectl exec -it'
alias kaf='kubectl apply -f'
alias kdr='kubectl delete -f'

The aliases are the muscle memory. The operator who types kgp -n prod is the operator who runs the methodology; the operator who types kubectl get pods --namespace production is the operator who is fighting the CLI.

The on-call ergonomics

The on-call shift is a posture. The posture is:

• Two windows open. One terminal running kubectl get pods --watch -n prod; one terminal running the methodology.
• Prompt shows context. kube-ps1 shows the current cluster and namespace in the prompt.
• History is searchable. fzf or kubectl history lets the operator re-run a previous command.
• Runbook is one click away. The runbook is open in the browser; the operator can read the failure mode without leaving the terminal.

flowchart TD
    A[Two terminals] --> B[Watch pods]
    A --> C[Run methodology]
    D[Prompt] --> E[Cluster, namespace]
    F[History] --> G[Re-run previous]
    H[Runbook] --> I[Failure mode]

The posture is the difference between an operator who recovers in 5 minutes and an operator who recovers in 30.

The muscle memory of the 11-step methodology

The 11-step methodology is a sequence. The sequence becomes muscle memory when the operator runs the same commands in the same order for every incident. The commands are:

# Step 1: symptom (mental)
echo "Symptom: Pod billing-7d8f-abcde has restarted 12 times"

# Step 2: impact (mental)
echo "Impact: 5% of billing requests are failing"

# Step 3: inspect
kubectl get pod billing-7d8f-abcde -n prod -o yaml
kubectl describe pod billing-7d8f-abcde -n prod

# Step 4: events
kubectl get events -n prod --sort-by=.lastTimestamp \
  --field-selector involvedObject.name=billing-7d8f-abcde

# Step 5: logs
kubectl logs -n prod billing-7d8f-abcde -c billing --previous
kubectl logs -n prod billing-7d8f-abcde -c billing --tail=200

# Step 6: dependencies
kubectl get svc,endpointslices,pvc,configmap,secret -n prod -l app=billing

# Step 7: identify (mental)
echo "Component: container (image, startup, probe)"

# Step 8: hypothesis (mental)
echo "Hypothesis: if the new image is missing a library, then --previous logs will show ImportError"

# Step 9: test
kubectl logs -n prod billing-7d8f-abcde -c billing --previous | grep -i import

# Step 10: restore
kubectl rollout undo deployment/billing -n prod

# Step 11: validate
kubectl get pods -n prod -l app=billing
kubectl get pdb -n prod

The commands are the same for every incident. The operator who runs the commands is the operator who runs the methodology.

The lab

The lab is the dry-run environment. The lab is a cluster that mirrors production: the same CNI, the same CSI, the same ingress, the same workload shape. The lab is where the operator rehearses the methodology.

kind create cluster --config lab.yaml
kubectl apply -f workload.yaml
kubectl run chaos --image=chaos --rm -it --restart=Never -- \
  python -c "import time; time.sleep(1)"

The lab is the muscle memory’s gym. The operator who rehearses in the lab is the operator who runs the methodology in production.

Production discipline

The sysadmin posture is the cluster’s trunk. The aliases are the muscle memory; the on-call ergonomics are the system; the lab is the muscle memory’s gym. The discipline is the same scale-free: every operator runs the methodology; every incident is the methodology.

• Install the tooling stack before the shift. The shift is not the place to install the tooling.
• Use the two-terminal setup. One for kubectl get pods --watch, one for the methodology.
• Open the runbook in the browser. The runbook is the operational reference.
• Treat the posture as a system, not a personality. The operator who arrives with the system recovers in 5 minutes; the operator who relies on personality recovers in 30.

Quiz