Lab 8: Requests, limits, and QoS classes

Objective

By the end of this lab you will have written down what QoS class you expect for six resource shapes, been wrong about at least one of them, and then opened the cgroup files inside the running containers to see the limit as the kernel stores it: a quota in microseconds and a byte count.

The measurement that matters is Task 5. Two containers running the same busy loop, one limited to 100m and one to 500m, produce very different nr_throttled counters over the same sixty seconds — and the third, with no CPU limit at all, produces none. That table is what turns “set a CPU limit” from a policy into a decision with a number attached.

Architecture

Everything lives inside one namespace, plus one label on one worker node which Cleanup removes.

kubeadm cluster (1.34.x)
  cp-1        control plane
  worker-1    schedulable, at least 2 CPU   <- you will label this one
  worker-2    schedulable

namespace qoslab
  qos-guaranteed     requests == limits, both resources
  qos-burstable      requests < limits
  qos-requests-only  requests set, no limits          -> the first trap
  qos-limits-only    limits set, no requests          -> the second trap
  qos-mixed          one sized container, one bare    -> the third trap
  qos-besteffort     no resources at all
  burn-100m          busy loop, cpu limit 100m
  burn-500m          busy loop, cpu limit 500m
  burn-nolimit       busy loop, no cpu limit
  hog / hog-2        request-only Pods used to fill the node's allocatable

workstation
  ~/k8s-qoslab/manifests/   the six shapes and the burn Pods
  ~/k8s-qoslab/evidence/    predictions, API answers, cgroup reads, throttling

Requirements

A kubeadm cluster on Kubernetes 1.34.x with at least one schedulable worker carrying at least 2 CPU. B-nested is sufficient.
kubectl 1.34.x, with a context allowed to create and delete a namespace and to add and remove one node label.
cgroups v2 on the nodes. Kubernetes 1.25 and later expect it and modern distributions ship it. Task 1 verifies this before anything depends on it.
Outbound access from the nodes to docker.io for busybox:1.36. Nothing else is pulled.
Roughly 400 MiB of memory and 1.5 CPU of headroom on the labelled worker. Task 5 deliberately saturates one core for sixty seconds.
No metrics-server, no Prometheus, no kubectl top. Every number in this lab comes from the API or from a cgroup file, so the lab runs on a bare cluster.
No out-of-band access requirement. Nothing here reconfigures networking, SSH or the firewall.

Scenario

Somebody has opened a pull request that adds resources to forty Deployments. The numbers came from a wiki page. In review, three questions have no agreed answer: whether a container with requests and no limits is BestEffort, whether setting a CPU limit is a safety measure or a latency tax, and whether the node dashboard showing 85% allocated means the node is 85% busy.

All three have exact answers, and none of them are matters of opinion. The first is computed by the API server at admission, the second is a CFS quota in the kernel, and the third is arithmetic over numbers that nobody measured. This lab produces all three from the cluster rather than from the wiki.

Tasks

Task 1: Baseline, and the gap between capacity and allocatable

# Substitute your own value if qoslab is taken on this cluster:
NS=qoslab
WORKDIR="$HOME/k8s-qoslab"

mkdir -p "$WORKDIR/manifests" "$WORKDIR/evidence"
cd "$WORKDIR"

kubectl version -o yaml > evidence/00-versions.yaml
kubectl get nodes -o wide > evidence/00-nodes.txt
kubectl get namespace "$NS" > evidence/00-namespace-before.txt 2>&1

kubectl create namespace "$NS"

If evidence/00-namespace-before.txt says Error from server (NotFound), Cleanup may delete the namespace. If it says anything else, the namespace predates the lab and deleting it destroys work that is not yours.

Pick the worker you will label, and record what it reports:

# Substitute a schedulable worker with at least 2 CPU, from evidence/00-nodes.txt:
NODE=worker-1

kubectl get node "$NODE" \
  -o jsonpath='capacity:    {.status.capacity}{"\n"}allocatable: {.status.allocatable}{"\n"}' \
  | tee evidence/01-node-capacity.txt

kubectl label node "$NODE" lab.runbook.academy/burn=yes
kubectl describe node "$NODE" > evidence/01-node-before.txt

Subtract the two lines. On a node the kubelet has not been tuned on, the CPU gap is often small and the memory gap is hundreds of mebibytes; on a tuned node it can be a core and several gibibytes. Either way, allocatable is the number the scheduler uses, and a capacity plan built on capacity over-provisions by exactly this difference.

Confirm cgroups v2, because Tasks 4 and 5 read v2 file names:

kubectl -n "$NS" run cgcheck --image=busybox:1.36 --restart=Never -it --rm -- \
  sh -c 'ls /sys/fs/cgroup/cgroup.controllers && echo "cgroups v2" || echo "not v2"'

/sys/fs/cgroup/cgroup.controllers exists only under the cgroups v2 unified hierarchy, so listing it is a definitive test. If it is missing, the node is on cgroups v1; the QoS tasks still work, but the file names in Tasks 4 and 5 do not exist.

Task 2: Predict first, then apply

Write your predictions before you apply anything. This is the whole point of the task: the QoS class is computed by a rule you either know or do not, and finding out which by looking at the answer teaches nothing.

Create evidence/predictions.txt and fill in the right-hand column from memory:

pod                 resources                                  predicted QoS
qos-guaranteed      requests == limits, cpu and memory         ____________
qos-burstable       requests 100m/128Mi, limits 500m/512Mi     ____________
qos-requests-only   requests 100m/128Mi, no limits             ____________
qos-limits-only     limits 200m/256Mi, no requests             ____________
qos-mixed           app sized+equal, sidecar with no resources ____________
qos-besteffort      no resources at all                        ____________

manifests/01-qos-shapes.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: qos-guaranteed
  namespace: qoslab
spec:
  containers:
    - name: app
      image: busybox:1.36
      command: ["sh", "-c", "sleep 3600"]
      resources:
        requests: {cpu: 200m, memory: 256Mi}
        limits:   {cpu: 200m, memory: 256Mi}
---
apiVersion: v1
kind: Pod
metadata:
  name: qos-burstable
  namespace: qoslab
spec:
  containers:
    - name: app
      image: busybox:1.36
      command: ["sh", "-c", "sleep 3600"]
      resources:
        requests: {cpu: 100m, memory: 128Mi}
        limits:   {cpu: 500m, memory: 512Mi}
---
apiVersion: v1
kind: Pod
metadata:
  name: qos-requests-only
  namespace: qoslab
spec:
  containers:
    - name: app
      image: busybox:1.36
      command: ["sh", "-c", "sleep 3600"]
      resources:
        requests: {cpu: 100m, memory: 128Mi}
---
apiVersion: v1
kind: Pod
metadata:
  name: qos-limits-only
  namespace: qoslab
spec:
  containers:
    - name: app
      image: busybox:1.36
      command: ["sh", "-c", "sleep 3600"]
      resources:
        limits: {cpu: 200m, memory: 256Mi}
---
apiVersion: v1
kind: Pod
metadata:
  name: qos-mixed
  namespace: qoslab
spec:
  containers:
    - name: app
      image: busybox:1.36
      command: ["sh", "-c", "sleep 3600"]
      resources:
        requests: {cpu: 100m, memory: 128Mi}
        limits:   {cpu: 100m, memory: 128Mi}
    - name: sidecar
      image: busybox:1.36
      command: ["sh", "-c", "sleep 3600"]
---
apiVersion: v1
kind: Pod
metadata:
  name: qos-besteffort
  namespace: qoslab
spec:
  containers:
    - name: app
      image: busybox:1.36
      command: ["sh", "-c", "sleep 3600"]

Configuration changeworkstation— Creates six Pods that differ only in their resources block

$ kubectl apply -f manifests/01-qos-shapes.yaml

kubectl -n "$NS" wait --for=condition=Ready pod --all --timeout=120s

kubectl -n "$NS" get pods \
  -o custom-columns='NAME:.metadata.name,QOS:.status.qosClass,NODE:.spec.nodeName' \
  | tee evidence/02-qos-actual.txt

Read-only / Safeworkstation— The class the API server computed for each shape

$ kubectl -n qoslab get pods -o custom-columns='NAME:.metadata.name,QOS:.status.qosClass'

NAME                QOS
qos-besteffort      BestEffort
qos-burstable       Burstable
qos-guaranteed      Guaranteed
qos-limits-only     Guaranteed
qos-mixed           Burstable
qos-requests-only   Burstable

Illustrative output

Compare against your predictions. Three of these catch people.

qos-requests-only is Burstable, not BestEffort. BestEffort means the Pod declared nothing. Declaring a request and omitting the limit is a deliberate, common and entirely valid shape — it means “reserve this much, and let me use whatever else is idle”. The two classes sit at opposite ends of the eviction order, so reading this one wrong inverts your understanding of which Pods die first.

qos-limits-only is Guaranteed. When a container sets a limit and no request, the request defaults to the limit. Requests then equal limits for every resource on every container, which is the definition. So the least explicit way to write a resources block produces the strongest QoS class, and it also silently reserves 200m of scheduler budget the author never typed.

qos-mixed is Burstable, because of the sidecar. The class is computed for the Pod, and one bare container is enough to disqualify it. A log shipper or a service-mesh proxy injected by a webhook, with no resources of its own, will demote a carefully sized application Pod from Guaranteed to Burstable, and nothing in the application’s own manifest will show it.

Task 3: Where the class actually bites

The class is not decoration. Record what each one means, from the lessons, and keep it next to the table:

Guaranteed is evicted last under node pressure, and its limits are still enforced. Guaranteed does not mean unlimited; a Guaranteed Pod that exceeds its memory limit is OOMKilled like any other.
Burstable sits in the middle, and is the correct class for most services.
BestEffort is evicted first, and does not count against a ResourceQuota at all, because it has no requests to count.

That last point is worth one command. Nothing you did to qos-besteffort reserved anything on the node:

kubectl describe node "$NODE" | sed -n '/Allocated resources/,/^Events/p' \
  | tee evidence/03-allocated-after-shapes.txt

Add up the CPU requests of the Pods that landed on this node and check the figure. qos-besteffort contributes zero, and qos-limits-only contributes 200m that its author never wrote.

Task 4: Read the limit as the kernel stores it

The limit is not a Kubernetes concept at runtime. It is a number in a cgroup file, written by the kubelet and enforced by the kernel. Read it from inside the containers.

for p in qos-guaranteed qos-burstable qos-requests-only qos-limits-only; do
  echo "=== $p ==="
  kubectl -n "$NS" exec "$p" -c app -- sh -c \
    'echo -n "cpu.max:    "; cat /sys/fs/cgroup/cpu.max;
     echo -n "memory.max: "; cat /sys/fs/cgroup/memory.max'
done | tee evidence/04-cgroups.txt

Read-only / Safeworkstation— The CPU quota and memory ceiling as the kernel holds them

$ kubectl -n qoslab exec qos-burstable -c app -- cat /sys/fs/cgroup/cpu.max

50000 100000

Illustrative output

cpu.max is quota period, both in microseconds. The period is 100000 µs — 100 ms, the CFS default. The quota is what your limit became:

limits.cpu: 500m becomes 50000 100000. Fifty milliseconds of CPU time per hundred-millisecond period.
limits.cpu: 200m becomes 20000 100000.
No CPU limit becomes max 100000. There is a period, and no quota.

memory.max is a plain byte count: 256Mi is 268435456, and a container with no memory limit reads max.

Do the arithmetic yourself for one of them and confirm it matches. This is the whole mechanism: a CPU limit of n millicores is a promise that the container gets n/1000 of a CPU-second per second, enforced by suspending it when the quota for the current 100 ms window runs out.

Under the hood

CPU and memory limits are enforced by different kernel machinery, and the difference is why they fail differently.

cpu.max is bandwidth control. When a cgroup exhausts its quota inside a period, the kernel stops scheduling its tasks until the next period begins. Nothing is killed, no error is returned, and the application observes only that some of its work took longer. This is why CPU is called compressible: the resource is time, and the kernel can always give you less of it.

memory.max is a hard ceiling on charged pages. There is no equivalent of “wait until next period” for a page that must exist now, so when a cgroup cannot reclaim its way under the limit the kernel invokes the OOM killer inside that cgroup. Nothing is throttled and the process is gone. This lab stops at reading memory.max; Lab 9 is the kill.

The practical asymmetry: a CPU limit set too low degrades a service. A memory limit set too low destroys it. They are not the same kind of setting and should not be reviewed with the same care.

Task 5: Measure the throttling

Three Pods, one busy loop, three CPU limits.

manifests/02-burn.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: burn-100m
  namespace: qoslab
spec:
  nodeSelector:
    lab.runbook.academy/burn: "yes"
  containers:
    - name: app
      image: busybox:1.36
      command: ["sh", "-c", "while true; do :; done"]
      resources:
        requests: {cpu: 10m, memory: 32Mi}
        limits:   {cpu: 100m, memory: 64Mi}
---
apiVersion: v1
kind: Pod
metadata:
  name: burn-500m
  namespace: qoslab
spec:
  nodeSelector:
    lab.runbook.academy/burn: "yes"
  containers:
    - name: app
      image: busybox:1.36
      command: ["sh", "-c", "while true; do :; done"]
      resources:
        requests: {cpu: 10m, memory: 32Mi}
        limits:   {cpu: 500m, memory: 64Mi}
---
apiVersion: v1
kind: Pod
metadata:
  name: burn-nolimit
  namespace: qoslab
spec:
  nodeSelector:
    lab.runbook.academy/burn: "yes"
  containers:
    - name: app
      image: busybox:1.36
      command: ["sh", "-c", "while true; do :; done"]
      resources:
        requests: {cpu: 10m, memory: 32Mi}

Every one of the three requests the same 10m. Only the ceiling differs.

Service impact possibleworkstation— Starts three identical busy loops with three different CPU limits on the labelled node

$ kubectl apply -f manifests/02-burn.yaml

kubectl -n "$NS" wait --for=condition=Ready pod \
  burn-100m burn-500m burn-nolimit --timeout=120s

read_stat() {
  kubectl -n "$NS" exec "$1" -c app -- \
    sh -c 'grep -E "usage_usec|nr_periods|nr_throttled|throttled_usec" /sys/fs/cgroup/cpu.stat' \
    2>/dev/null || echo "no throttling counters (no quota configured)"
}

for p in burn-100m burn-500m burn-nolimit; do
  echo "=== $p t=0 ==="; read_stat "$p"
done > evidence/05-throttle-t0.txt

sleep 60

for p in burn-100m burn-500m burn-nolimit; do
  echo "=== $p t=60 ==="; read_stat "$p"
done > evidence/05-throttle-t60.txt

kubectl describe node "$NODE" | sed -n '/Allocated resources/,/^Events/p' \
  > evidence/05-allocated-during-burn.txt

paste evidence/05-throttle-t0.txt evidence/05-throttle-t60.txt

The allocated-resources capture is taken while the loops are running. Task 6 compares it against the one you took in Task 3, so take it before you delete anything.

Now subtract, per Pod, and record the result in evidence/05-throttling.txt:

pod            limit   nr_periods  nr_throttled  throttled_usec  throttled %
burn-100m      100m    ______      ______        ______          ______
burn-500m      500m    ______      ______        ______          ______
burn-nolimit   none    ______      ______        ______          ______

The throttled percentage is nr_throttled / nr_periods. What you should see, and why:

burn-100m is throttled in nearly every period. The loop wants a whole core; the quota gives it 10 ms out of every 100. It spends roughly 90% of every period suspended, and throttled_usec climbs by roughly 90 ms per wall-clock second.
burn-500m is throttled in nearly every period as well, but throttled_usec climbs at about half that rate. The counter for how often and the counter for how long say different things, which is why alerting on nr_throttled alone produces panic about workloads that are fine.
burn-nolimit has no quota, so it is never throttled and its usage_usec climbs by about one CPU-second per wall-clock second. Depending on kernel version the throttling lines may be present and zero, or absent entirely.

A CPU limit is a latency decision, not a safety measure

Nothing here protected the node. All three Pods requested 10m and the scheduler believed all three; the only thing the limits changed was how much work each one got done. A CPU limit constrains the container that carries it and nobody else — under contention, the thing that actually protects other workloads is the CPU request, which sets the proportional share the kernel falls back on.

That is the substance of the argument about CPU limits. Removing them removes a latency tax you may be paying for nothing; keeping them makes a service’s performance independent of what else lands on the node, which is worth something on a noisy cluster. Either position is defensible. “We set limits so the node stays healthy” is not, and this table is why.

Delete the burn Pods now:

Destructiveworkstation— Stops the three busy loops and frees the core

$ kubectl -n qoslab delete pod burn-100m burn-500m burn-nolimit

Task 6: The scheduler counts requests and nothing else

Before deleting them you saw three Pods using between them well over a core. Look at what the node reported while that was true:

diff evidence/03-allocated-after-shapes.txt evidence/05-allocated-during-burn.txt \
  | tee evidence/06-allocated-diff.txt

The CPU line moved by 30m — three Pods at their 10m request each — and not by the core and a half they were actually consuming at the moment of the capture. Allocated resources is the sum of requests. It is a bookkeeping figure about declarations, and it has no relationship to load.

Now push it the other way. Find the node’s allocatable CPU, then ask for almost all of what is left:

kubectl get node "$NODE" -o jsonpath='{.status.allocatable.cpu}{"\n"}'

# Substitute a request that leaves a little headroom below your own
# node's remaining allocatable CPU, read from the command above:
HOG_CPU=1200m

sed "s/HOG_CPU_PLACEHOLDER/$HOG_CPU/" manifests/04-hog.yaml | kubectl apply -f -

kubectl -n "$NS" get pods hog hog-2
kubectl -n "$NS" describe pod hog-2 | tail -15 | tee evidence/06-unschedulable.txt

manifests/04-hog.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: hog
  namespace: qoslab
spec:
  nodeSelector:
    lab.runbook.academy/burn: "yes"
  containers:
    - name: app
      image: busybox:1.36
      command: ["sh", "-c", "sleep 3600"]
      resources:
        requests: {cpu: HOG_CPU_PLACEHOLDER, memory: 32Mi}
---
apiVersion: v1
kind: Pod
metadata:
  name: hog-2
  namespace: qoslab
spec:
  nodeSelector:
    lab.runbook.academy/burn: "yes"
  containers:
    - name: app
      image: busybox:1.36
      command: ["sh", "-c", "sleep 3600"]
      resources:
        requests: {cpu: HOG_CPU_PLACEHOLDER, memory: 32Mi}

hog is Running and doing nothing at all. hog-2 is Pending with a FailedScheduling event naming Insufficient cpu. Two Pods that will never execute an instruction between them, and the node is full — because full is a statement about the sum of requests, and both of them declared 1200m.

An 85% allocated node can be idle, and a 40% allocated node can be on fire

Every capacity dashboard built on Allocated resources is reporting how much people have claimed, not how much is being used. The two diverge in both directions and the failure modes are opposite.

Requests set far above real usage produce nodes that refuse to schedule while sitting idle: you buy hardware to satisfy arithmetic. Requests set far below real usage produce nodes that schedule happily and then collapse, because the scheduler’s model said there was room. Both look identical on a dashboard that shows only one of the two numbers.

Right-sizing is the practice of closing that gap, and it needs a usage measurement the scheduler never takes. That is what metrics-server and Prometheus are for, and why a cluster without either cannot size anything.

kubectl -n "$NS" delete pod hog hog-2

Task 7: Change a Pod’s class without touching the Pod

manifests/03-limitrange.yaml:

apiVersion: v1
kind: LimitRange
metadata:
  name: defaults
  namespace: qoslab
spec:
  limits:
    - type: Container
      default:
        cpu: 500m
        memory: 512Mi
      defaultRequest:
        cpu: 100m
        memory: 128Mi

Configuration changeworkstation— Adds namespace-wide default requests and limits

$ kubectl apply -f manifests/03-limitrange.yaml

kubectl -n "$NS" get pod qos-besteffort -o jsonpath='{.status.qosClass}{"\n"}'

kubectl -n "$NS" delete pod qos-besteffort
kubectl apply -f manifests/01-qos-shapes.yaml

kubectl -n "$NS" wait --for=condition=Ready pod qos-besteffort --timeout=120s
kubectl -n "$NS" get pod qos-besteffort \
  -o jsonpath='qos={.status.qosClass}{"\n"}resources={.spec.containers[0].resources}{"\n"}' \
  | tee evidence/07-limitrange.txt

The existing Pod’s class did not change — it could not, because QoS is fixed at admission. The recreated Pod, from a manifest with no resources field at all, now carries requests of 100m and 128Mi, limits of 500m and 512Mi, and a class of Burstable.

Nothing in the repository changed. Somebody applied a LimitRange, and forty Deployments started reserving 100m each at their next restart — 4 CPU of scheduler budget that no manifest mentions and no author chose.

Task 8: Write the sizing note

This is the deliverable. Create evidence/sizing.md and answer, from your own evidence:

1. Which of your six predictions were wrong, and what rule did you have
   in your head that produced the wrong answer?

2. For a service whose steady-state usage you do not know, which of these
   would you set first: CPU request, CPU limit, memory request, memory
   limit? Justify the order from the throttling table and from the
   compressible/incompressible split, not from convention.

3. Your table shows burn-100m throttled in almost every period. Name a
   workload for which that is the correct configuration, and one for
   which it would be an incident.

4. What measurement would you need before setting a request, and where
   would it come from on this cluster? (Note that this cluster cannot
   currently produce it.)

Validation

cd "$HOME/k8s-qoslab"

diff <(awk '{print $1}' evidence/predictions.txt) /dev/null || true
cat evidence/02-qos-actual.txt
grep -E "cpu.max|memory.max" evidence/04-cgroups.txt
cat evidence/05-throttling.txt
grep -i "insufficient cpu" evidence/06-unschedulable.txt
grep -i "burstable" evidence/07-limitrange.txt
grep -c . evidence/sizing.md

The lab succeeded when all of the following hold:

evidence/02-qos-actual.txt shows Guaranteed for both qos-guaranteed and qos-limits-only, and Burstable for both qos-requests-only and qos-mixed.
evidence/04-cgroups.txt shows 50000 100000 for the 500m limit, 20000 100000 for the 200m limit, and max 100000 for the Pod with requests and no limits.
evidence/05-throttling.txt has a non-zero nr_throttled for burn-100m and burn-500m, with throttled_usec roughly twice as large for burn-100m over the same window, and no throttling for burn-nolimit.
evidence/06-allocated-diff.txt shows the node’s allocated CPU moving by the requests you set, not by the CPU the burn Pods consumed.
evidence/06-unschedulable.txt contains Insufficient cpu for a Pod that was executing nothing.
evidence/07-limitrange.txt shows Burstable and a populated resources field for a Pod whose manifest has no resources field.

Expected Outcome

~/k8s-qoslab/
├── manifests/
│   ├── 01-qos-shapes.yaml
│   ├── 02-burn.yaml
│   ├── 03-limitrange.yaml
│   └── 04-hog.yaml
└── evidence/
    ├── 00-versions.yaml, 00-nodes.txt, 00-namespace-before.txt
    ├── 01-node-capacity.txt, 01-node-before.txt
    ├── predictions.txt, 02-qos-actual.txt
    ├── 03-allocated-after-shapes.txt
    ├── 04-cgroups.txt
    ├── 05-throttle-t0.txt, 05-throttle-t60.txt, 05-throttling.txt
    ├── 05-allocated-during-burn.txt
    ├── 06-allocated-diff.txt, 06-unschedulable.txt
    ├── 07-limitrange.txt
    └── sizing.md

A prediction table with your errors preserved next to the API’s answers, a set of cgroup reads in which a Kubernetes field and a kernel number are shown to be the same thing, and a throttling measurement with real counters in it.

Production notes

Reviewing a resources block. Read it for the class it produces, not for the numbers. A missing limit is a deliberate shape, not an omission; a limit with no request reserves scheduler budget nobody typed; a bare sidecar demotes the whole Pod. Ask for .status.qosClass from a running Pod rather than reasoning from the manifest, because injected containers do not appear in the repository.

Adding a LimitRange. Treat it as a capacity change to every workload in the namespace, arriving gradually. Compute default request times expected container count before applying, and schedule it like any other capacity change.

Setting CPU limits. Decide the policy explicitly and write down which of the two things you are buying: predictable per-container performance, or the absence of a throttling tax. Then measure nr_throttled and throttled_usec on the workloads you applied it to, because the cost is measurable and usually unmeasured.

Setting memory limits. These are not optional and they are not the same kind of decision. The failure mode is a killed process, which is Lab 9.

Capacity reviews. Never present allocated-versus-allocatable without actual usage beside it. A change window planned against requests alone will either refuse work on an idle cluster or admit work onto a full one, and the dashboard looks the same in both cases.

Troubleshooting

stat -fc %T /sys/fs/cgroup/ returns tmpfs. The node is on cgroups v1. Tasks 2, 3, 6 and 7 work unchanged; Tasks 4 and 5 need the v1 paths (/sys/fs/cgroup/cpu/cpu.cfs_quota_us and cpu.stat with different field names) and are not covered here.

cat: can't open '/sys/fs/cgroup/cpu.max'. The container is not in its own cgroup namespace, so it sees the host’s cgroup root instead of its own. Confirm the runtime is containerd on cgroups v2. As a fallback, find the file with kubectl -n qoslab exec POD -c app -- find /sys/fs/cgroup -maxdepth 3 -name cpu.max.

nr_throttled is absent from cpu.stat for burn-nolimit. Expected on some kernels: with no quota there is no bandwidth accounting to report. Record “absent” rather than zero; they mean the same thing here.

burn-100m shows nr_periods of 0. You read the counters before the Pod was Ready, or the container had not started its loop. Re-run read_stat after confirming kubectl -n qoslab get pod burn-100m shows Running.

The node went NotReady during Task 5. burn-nolimit starved the kubelet on a node with too little CPU. Delete the burn Pods, wait for the node to recover, and re-run on a worker with at least 2 CPU — or drop burn-nolimit and compare only the two limited Pods.

hog is Pending as well as hog-2. Your HOG_CPU exceeded the remaining allocatable on its own. Read kubectl describe node for the current allocated total, subtract from allocatable, and pick a value below the difference.

sed on 04-hog.yaml produces an invalid quantity. HOG_CPU must be a Kubernetes quantity such as 1200m or 1.2, not a bare number of cores with a unit the API does not know. Check with kubectl apply --dry-run=server -f - before applying for real.

A Pod stays Pending with Insufficient memory in Task 2. The six shapes request about 700Mi in total. Reduce the memory requests proportionally across the manifest; the QoS classes are unaffected by the magnitudes.

Cleanup

The lab created one namespace and one node label. Both go.

Step 1. Confirm what you are about to remove:

NS=qoslab
NODE=worker-1

kubectl -n "$NS" get all
cat "$HOME/k8s-qoslab/evidence/00-namespace-before.txt"

Step 2. Remove the namespace and the node label. The label is the part people forget, and it will silently attract any future Pod that happens to use the same selector:

Destructiveworkstation— Removes the lab namespace and every object in it

$ kubectl delete namespace qoslab --wait=true

NODE=worker-1

kubectl label node "$NODE" lab.runbook.academy/burn-
kubectl get node "$NODE" --show-labels | grep -c "lab.runbook.academy" || \
  echo "label gone, as expected"

Step 3. Verify the node is as you found it:

NODE=worker-1

kubectl get nodes -o wide
kubectl describe node "$NODE" | sed -n '/Allocated resources/,/^Events/p'
diff <(kubectl get nodes -o wide) "$HOME/k8s-qoslab/evidence/00-nodes.txt" \
  && echo "node inventory unchanged"

The allocated figure should be back to what evidence/01-node-before.txt recorded. If it is not, a Pod from the lab survived; check for Pods in other namespaces that matched the burn selector.

Step 4. Keep the deliverables, then remove the working directory.

ls -la "$HOME/k8s-qoslab"

mkdir -p "$HOME/k8s-lab-deliverables/requests-limits"
cp -a "$HOME/k8s-qoslab/evidence" "$HOME/k8s-qoslab/manifests" \
      "$HOME/k8s-lab-deliverables/requests-limits/"

rm -rf "$HOME/k8s-qoslab"

What You Learned

QoS is computed from the resources block, not declared. Six shapes, three classes, and at least one of your predictions was wrong.
Requests with no limits is Burstable; limits with no requests is Guaranteed. The second happens because the request defaults to the limit, which also reserves scheduler budget nobody typed.
One bare container demotes the whole Pod. An injected sidecar with no resources turns a Guaranteed application into a Burstable one, invisibly.
The limit is a cgroup file. 500m is 50000 100000 in cpu.max, and 256Mi is 268435456 in memory.max. You read both from inside the container.
A CPU limit costs latency and protects nobody else. Three identical loops, three limits, three very different nr_throttled counters — and the node was equally exposed in all three cases.
The scheduler counts requests only. A container burning a core against a 10m request is invisible to it, and two idle Pods requesting 1200m each can fill a node.
A LimitRange changes future Pods only. The class of a running Pod is immutable; the effect of the policy arrives with the next restart.

Why this matters

Resource settings are the part of a manifest that reviewers skim. They are numbers, they look arbitrary, and there is rarely a reviewer who can say what the right ones are — so blocks get copied between repositories and the question of whether they are correct is never asked, only inherited.

The cost shows up in two directions at once, and neither is visible from a dashboard. On one side, requests set well above real usage make a cluster refuse work while its CPUs idle; the response is usually to buy more nodes, which fixes the arithmetic and not the waste. On the other, requests set well below real usage make the scheduler cheerfully overcommit a node until something on it dies, and the postmortem blames the workload that happened to be last.

Between the two sits the throttling table you measured. A team that has never looked at nr_throttled cannot tell a service that is slow because it is badly written from one that is slow because it is suspended for ninety milliseconds out of every hundred, and those two problems have nothing in common except the graph. Reading the number takes one kubectl exec, and it is the difference between tuning a CPU limit and rewriting an application that was never the problem.

Objectives

Prerequisites