QoS-based production patterns — designing for the right class

The QoS class is a design choice with real production implications. This lesson covers the matching of QoS to workload type, the audit process, and the enforcement mechanisms.

The QoS-to-workload mapping

flowchart TD
    DB[(Database)] --> G[Guaranteed]
    Queue[Message queue] --> G
    HTTP[HTTP service] --> B[Burstable]
    Worker[Worker] --> B
    Batch[Batch job] --> BE[BestEffort]
    G --> Eviction[Evicted last]
    B --> Eviction
    BE --> Eviction

The mapping is a starting point. Each workload has its own characteristics; tune based on actual usage.

Designing a Guaranteed workload

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: postgres
        image: postgres:16
        resources:
          requests:
            cpu: 2
            memory: 4Gi
          limits:
            cpu: 2
            memory: 4Gi
        # ... other configuration
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: [ReadWriteOnce]
      resources:
        requests:
          storage: 100Gi

Design choices:

• CPU request = limit: matches peak usage (2 cores). The scheduler reserves 2 cores; the cgroup enforces the limit.
• Memory request = limit: matches peak usage (4Gi). Predictable; no bursting.
• Storage via PVC: durable; survives Pod eviction.

This Pod is Guaranteed. Under node pressure, it is evicted last.

Designing a Burstable workload

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 5
  template:
    spec:
      containers:
      - name: nginx
        image: nginx:1.27.2
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 500m
            memory: 512Mi

Design choices:

• CPU request = 100m: typical idle usage.
• CPU limit = 500m: peak traffic (5x burst).
• Memory request = 128Mi: typical usage.
• Memory limit = 512Mi: peak with cache growth (4x burst).

This Pod is Burstable. The scheduler reserves 100m and 128Mi; the workload can burst up to 500m and 512Mi.

Designing a BestEffort workload

apiVersion: batch/v1
kind: Job
metadata:
  name: batch-report
spec:
  template:
    spec:
      restartPolicy: OnFailure
      containers:
      - name: reporter
        image: reporter:1.0
        # no resources — BestEffort

Design choices:

• No resources: the Job is best-effort. If evicted, the Job retries (restartPolicy: OnFailure).
• Short-lived: the Job runs once and exits.

This Pod is BestEffort. Under node pressure, it is evicted first; the Job controller creates a new Pod.

When Burstable assumptions break

A common production pitfall: a Burstable Pod that always bursts above its requests:

# Pod with requests too low for actual usage
resources:
  requests: {cpu: 100m, memory: 128Mi}
  limits:   {cpu: 500m, memory: 512Mi}

If the workload consistently uses 400m CPU and 400Mi memory, the usage-vs-request ratio is 4 (CPU) and 3 (memory). Under node pressure, this Pod is evicted first among Burstable Pods.

The fix: increase requests to match typical usage.

# Right-sized requests
resources:
  requests: {cpu: 400m, memory: 400Mi}
  limits:   {cpu: 500m, memory: 512Mi}

Now the usage-vs-request ratio is 1.25; the Pod is evicted last among Burstable Pods.

Auditing QoS across the cluster

kubectl get pods -A -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,QOS:.status.qosClass \
  | sort | uniq -c | sort -rn

Output (counts by QoS):

  142 Burstable
   12 Guaranteed
    3 BestEffort

A healthy production cluster has:

• Most workloads Burstable (typical HTTP services, workers).
• Some Guaranteed (databases, message queues, critical workloads).
• Few or no BestEffort (batch jobs only; debugging containers are short-lived).

If BestEffort counts are high in production namespaces, investigate.

# Identify BestEffort Pods in production namespaces
kubectl get pods -n team-a-prod -o custom-columns=NAME:.metadata.name,QOS:.status.qosClass \
  | grep BestEffort

Enforcing QoS with LimitRange

apiVersion: v1
kind: LimitRange
metadata:
  name: prod-policy
  namespace: team-a-prod
spec:
  limits:
  - type: Container
    default:
      cpu: 500m
      memory: 512Mi
    defaultRequest:
      cpu: 100m
      memory: 128Mi
    min:
      cpu: 10m
      memory: 16Mi
    max:
      cpu: 4
      memory: 8Gi
    maxLimitRequestRatio:
      cpu: "4"
      memory: "2"

This LimitRange:

• Applies defaults so every Pod has resources (no BestEffort).
• Allows small resources (10m, 16Mi min).
• Caps large resources (4, 8Gi max).
• Prevents unbounded bursting (4:1, 2:1 ratio).

Production discipline: every production namespace has a LimitRange. New Pods without resources get defaults and are Burstable; production workloads declare resources explicitly.

Enforcing Guaranteed for critical workloads

For workloads that must be Guaranteed (databases, critical services), use a stricter LimitRange or admission controller:

apiVersion: v1
kind: LimitRange
metadata:
  name: db-policy
  namespace: databases
spec:
  limits:
  - type: Container
    default: {cpu: 2, memory: 4Gi}
    defaultRequest: {cpu: 2, memory: 4Gi}
    max: {cpu: 16, memory: 32Gi}

A Pod in this namespace that does not declare requests == limits is rejected (the kubelet validates limits, but the defaulting rule won’t make a Pod Guaranteed unless requests == limits). Actually, LimitRange does not enforce Guaranteed; it only enforces defaults, min, max, ratio.

For Guaranteed enforcement, use a custom admission controller or a Kyverno/OPA policy:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-guaranteed
spec:
  validationFailureAction: enforce
  rules:
  - name: check-qos
    match:
      resources:
        kinds: ["Pod"]
    validate:
      message: "Pods in the databases namespace must be Guaranteed."
      pattern:
        spec:
          containers:
          - resources:
              requests:
                cpu: "?*"
                memory: "?*"
              limits: "?*"

This Kyverno policy enforces that every Pod in the databases namespace has matching requests and limits.

Production patterns

Database namespace with Guaranteed enforcement:

# LimitRange with strict defaults
apiVersion: v1
kind: LimitRange
metadata:
  name: db-defaults
  namespace: databases
spec:
  limits:
  - type: Container
    default: {cpu: 2, memory: 4Gi}
    defaultRequest: {cpu: 2, memory: 4Gi}
    max: {cpu: 16, memory: 32Gi}

# Kyverno policy to enforce matching requests/limits
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: db-qos
spec:
  validationFailureAction: enforce
  rules:
  - name: check-qos
    match:
      resources:
        kinds: ["Pod"]
        namespaces: ["databases"]
    validate:
      message: "Database Pods must have matching requests and limits."
      pattern:
        spec:
          containers:
          - resources:
              requests: "?*"
              limits: "?*"

HTTP service namespace with Burstable defaults:

apiVersion: v1
kind: LimitRange
metadata:
  name: web-defaults
  namespace: team-a-prod
spec:
  limits:
  - type: Container
    default: {cpu: 500m, memory: 512Mi}
    defaultRequest: {cpu: 100m, memory: 128Mi}
    min: {cpu: 10m, memory: 16Mi}
    max: {cpu: 4, memory: 8Gi}
    maxLimitRequestRatio:
      cpu: "4"
      memory: "2"

CI checks for QoS

Add CI checks to manifest linters:

# .konflint.yaml or similar
rules:
- name: no-besteffort-in-prod
  message: "Production namespaces must not have BestEffort Pods."
  rule:
    any:
    - has:
        field: spec.containers
        every:
          has:
            field: resources.requests

Or use kube-linter, polaris, or kubectl ad-hoc:

# Fail if any production Pod is BestEffort
kubectl get pods -n team-a-prod -o json | \
  jq '.items[] | select(.status.qosClass=="BestEffort") | .metadata.name' | \
  head -5

Cross-course references

• The Linux course part XXXVII-Linux-Resources covers cgroup resource management; QoS is the cluster-level equivalent.
• The Ansible course part XXXV-Ansible-Scripting covers service priority; QoS is the cluster-level equivalent.
• The Observability course part LXXXIV-Kubernetes-CapacityPlanning covers capacity planning; QoS is part of the planning inputs.

Quiz

Production discipline

• Match QoS to workload type. Databases Guaranteed; HTTP Burstable; batch BestEffort.
• Right-size requests for typical usage. A Burstable Pod that always bursts has poor eviction priority.
• Use LimitRange defaults to prevent BestEffort. Every production namespace has sane defaults.
• Audit QoS weekly. kubectl get pods -A -o custom-columns shows the cluster’s QoS distribution.
• Use admission controllers (Kyverno, OPA) for strict QoS enforcement. LimitRange does not enforce Guaranteed; admission does.