Quota enforcement lifecycle — admission, observation, and the failure modes

The quota enforcement lifecycle has multiple failure modes. This lesson walks the lifecycle, the failure modes, the operational response, and the discipline.

The lifecycle

flowchart LR
    A[Pod creation request] --> B[Admission controller]
    B --> C{Check quota}
    C -->|Within quota| D[Accepted]
    C -->|Exceeds quota| E[Rejected with 403]
    E --> F[Pod creation fails]
    F --> G[Operator investigates]
    G --> H{Adjust quota or workload?}
    H -->|Quota| I[Edit ResourceQuota]
    H -->|Workload| J["Reduce requests/limits"]
    I --> K[Re-submit Pod]
    J --> K
    K --> D

The lifecycle:

1. The user submits a Pod creation request.
2. The admission controller checks the quota.
3. If within quota, the Pod is accepted.
4. If exceeds quota, the Pod is rejected with 403.
5. The operator investigates, adjusts the quota or the workload, and re-submits.

The observation

kubectl describe resourcequota -n tenant-a-prod

Name:            compute-quota
Namespace:       tenant-a-prod
Resource         Used    Hard
--------         ----    ----
requests.cpu     32      32
requests.memory  64Gi    64Gi
limits.cpu       48      64
limits.memory    96Gi    128Gi

Name:            object-count-quota
Namespace:       tenant-a-prod
Resource         Used    Hard
--------         ----    ----
pods             200     200
services         95      100
secrets          80      100
configmaps       90      100

The Used column shows current usage; the Hard column shows the quota. A Used equal to Hard means the next request will be rejected.

The failure modes

flowchart LR
    A[Failure modes] --> B[Rollout failure]
    A --> C[HPA scaling blocked]
    A --> D[Debug Pod blocked]
    A --> E[Cluster upgrade blocked]

The failure modes:

• Rollout failure. A new ReplicaSet needs Pods; the quota is at 100%; the rollout fails.
• HPA scaling blocked. HPA needs to create new Pods; the quota is at 100%; scaling fails.
• Debug Pod blocked. An operator tries to create a debug Pod; the quota is at 100%; the Pod is rejected.
• Cluster upgrade blocked. New control-plane Pods need resources; the quota blocks them.

The response

flowchart TD
    A[Quota rejection] --> B{Which quota?}
    B -->|Compute| C["Increase requests.cpu/memory quota"]
    B -->|Object count| D["Increase pods/services/secrets quota"]
    B -->|Storage| E[Increase requests.storage quota]
    C --> F[Document change in runbook]
    D --> F
    E --> F
    F --> G[Re-submit Pod]
    G --> H{Accepted?}
    H -->|Yes| I[Done]
    H -->|No| J[Reduce workload's requests]

The response:

1. Identify the quota that was exceeded.
2. Adjust the quota (increase the limit).
3. Document the change in the runbook.
4. Re-submit the Pod.
5. If still rejected, reduce the workload’s requests.

The discipline is to identify the cause before adjusting. Adjusting blindly can mask a real issue.

Cluster upgrade and quotas

flowchart LR
    A[Cluster upgrade] --> B[New control-plane Pods]
    B --> C{Namespace has quota?}
    C -->|Yes| D[Quota may block upgrade]
    C -->|No| E[Upgrade proceeds]
    D --> F[Increase quota or exclude kube-system]

A cluster upgrade may create new Pods in kube-system or other system namespaces. If those namespaces have quotas, the upgrade may be blocked. The fix:

• Do not apply ResourceQuota to kube-system.
• Ensure system namespaces have generous quotas (or no quotas).

Quiz

The operational discipline

Quota enforcement in production rests on five non-negotiable elements:

• Monitor utilisation. Alert on namespaces approaching quota.
• Document quota changes. Every change is in the runbook.
• Don’t bypass quotas. Adjusting quotas to allow oversized workloads trains the team to ignore them.
• Don’t quota system namespaces. kube-system and similar are not subject to quotas.
• Test quota failures. Quarterly: create a Pod that exceeds the quota; verify it’s rejected.

Quotas are production contracts. The discipline is to monitor the contract, respond to failures deliberately, and never bypass it.