Quota operations and anti-patterns — the operational discipline

Quota operations and anti-patterns are the operational discipline of resource contracts. This lesson walks the operations, the anti-patterns, the response, and the discipline.

Setting quotas deliberately

flowchart LR
    A[Setting quotas] --> B["Observe usage: kubectl top, metrics"]
    B --> C[Add 30-50% headroom]
    C --> D[Set requests quota]
    D --> E[Set limits at 2x requests]
    E --> F[Apply LimitRange for defaults]
    F --> G[Monitor and adjust]

The operations:

1. Observe usage. Use kubectl top, Prometheus, or the metrics server to determine current CPU and memory consumption.
2. Add 30-50% headroom. Allow for growth and bursts.
3. Set requests quota. requests.cpu and requests.memory.
4. Set limits at 2x requests. Allow bursting.
5. Apply LimitRange for defaults. Every Pod has requests.
6. Monitor and adjust. Quarterly review.

Anti-pattern 1: No quota on kube-system

# Anti-pattern
kubectl create namespace kube-system
# No ResourceQuota applied

A kube-system without a quota can have any number of Pods. A bug in kube-system (a runaway Pod that schedules many replicas) can exhaust node resources.

The fix: kube-system should have generous (or no) quotas, but never be without limits. Critical components must always have room.

Anti-pattern 2: Oversized workloads

# Anti-pattern
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  template:
    spec:
      containers:
        - name: app
          image: myapp
          resources:
            requests:
              cpu: 64  # 64 cores per Pod
              memory: 128Gi

A workload that requests 64 cores per Pod exceeds most namespace quotas. The team increases the quota to allow the oversized workload. The next team sees the precedent and asks for more.

The fix: enforce requests close to actual usage. A Pod that needs 64 cores is probably multiple Pods; split the workload.

Anti-pattern 3: Frequent quota adjustments

# Day 1: quota is 32 cores
# Day 5: increased to 64 cores
# Day 10: increased to 128 cores
# Day 15: increased to 256 cores
# Day 20: cluster exhausted

A team that frequently adjusts quotas is not setting them deliberately. Each adjustment trains the team to ask for more.

The fix: set quotas based on observed usage and headroom. Adjustments should be infrequent (quarterly) and justified.

Anti-pattern 4: No LimitRange

# Anti-pattern: ResourceQuota without LimitRange
apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute-quota
spec:
  hard:
    requests.cpu: "32"

A quota without a LimitRange has Pods without requests. The quota cannot enforce what is not declared.

The fix: always apply LimitRange alongside ResourceQuota.

The operational response

flowchart TD
    A[Quota anti-pattern detected] --> B{Which anti-pattern?}
    B -->|No kube-system quota| C[Add generous kube-system quota]
    B -->|Oversized workload| D[Reduce workload's requests]
    B -->|Frequent adjustments| E[Establish adjustment cadence]
    B -->|No LimitRange| F[Apply LimitRange]
    C --> G[Document change in runbook]
    D --> G
    E --> G
    F --> G

The response:

1. Identify the anti-pattern.
2. Apply the fix (specific to the anti-pattern).
3. Document the change in the runbook.
4. Establish the cadence to prevent regression.

Quiz

The operational discipline

Quota operations in production rest on five non-negotiable elements:

• Set based on observation. Data, not intuition.
• Apply LimitRange alongside. Defaults for every Pod.
• Don’t quota kube-system strictly. Allow cluster components.
• Don’t bypass for oversized workloads. Train the team to size appropriately.
• Quarterly review. Adjustments are infrequent and justified.

Quotas are production contracts. The discipline is to set them deliberately, monitor them, and never bypass them for convenience.