Lab: Apply cgroup resource limits to a service

This lab applies cgroup resource limits to a systemd service, runs a stress workload, and observes the limits taking effect.

Objective

By the end of this lab, you can:

• Apply CPU, memory and I/O limits to a systemd unit.
• Read the live cgroup counters that prove each limit is enforced, while the unit is still running.
• Tell a MemoryHigh throttle apart from a MemoryMax OOM kill from the evidence, not from guesswork.
• Explain why a limit that looks configured can be doing nothing.

Architecture

One host, one throwaway systemd unit, one stress workload. The unit runs under system.slice, so its controls appear at /sys/fs/cgroup/system.slice/stress-test.service/. systemd writes the limits into that directory’s control files (cpu.max, memory.max, memory.high, io.weight) and the kernel enforces them; nothing in this lab enforces anything in userspace.

The critical structural fact: the cgroup exists only while the unit’s processes do. When the unit exits, systemd removes the directory and every counter in it. All observation therefore happens during the 120-second window in Task 3, and anything you need afterwards has to come from systemctl show or the journal.

Requirements

Check these before you start; two of them change what you will see.

# 1. cgroup v2 unified hierarchy - must print 'cgroup2fs'
stat -fc %T /sys/fs/cgroup

# 2. At least 4 CPUs. CPUQuota=200% means 'two cores worth', so
#    on a 2-core host it is not a limit at all and nr_throttled
#    stays at zero - the lab silently demonstrates nothing.
nproc

# 3. Note whether the host has swap. The unit sets
#    MemorySwapMax=0 so the result is the same either way, but
#    you should know which host you are on.
swapon --show
free -h | grep -i swap

# 4. Root or sudo, and a package manager you can install into.

A host with fewer than 4 cores still works if you lower --cpu 4 and CPUQuota=200% together, keeping the workload above the quota.

Scenario

A batch job on a shared application host went into a loop last month, took every core, and pushed the host into swap until the OOM killer picked the database. The postmortem action was “put limits on it”. You have been asked to work out what limits actually do before applying them to anything that matters - in particular whether MemoryMax caps a service gently or kills it, because the answer determines whether it can go anywhere near the database.

Tasks

Task 1: Install stress-ng

sudo apt install stress-ng

Task 2: Create a service with limits

sudo tee /etc/systemd/system/stress-test.service <<EOF
[Unit]
Description=Stress test with resource limits

[Service]
Type=simple
ExecStart=/usr/bin/stress-ng --cpu 4 --vm 2 --vm-bytes 1G --timeout 120s
CPUQuota=200%
MemoryMax=1G
MemoryHigh=750M
MemorySwapMax=0
IOWeight=500
EOF

Three choices in that unit deserve an explanation, because the obvious versions of them break the lab.

Type=simple with a 120-second workload, not Type=oneshot with 30 seconds. systemd deletes a unit’s cgroup as soon as the unit’s processes exit, taking memory.events, cpu.stat and every other counter with it. Verify a short-lived oneshot by reading its cgroup directory and you get No such file or directory — not because the limits failed, but because there is nothing left to read. A long-lived unit gives you a window to observe in.

IOWeight=500, not 100. The documented default of io.weight is 100, so IOWeight=100 writes the value that was already there. It is a directive that looks like a limit, changes nothing, and makes an I/O-contention problem look configured when it is not. Set a value that differs from the default, or do not claim the control.

MemorySwapMax=0, so the result does not depend on whether the host has swap. MemoryMax caps anonymous memory plus swap separately: on a swapless host a 2 GiB working set against MemoryMax=1G reaches the OOM killer, but on a host with swap the cgroup spills the excess and grinds along, thrashing, with memory.current parked near the limit and oom_kill still at zero. Two hosts, two different lessons, from the same unit file. Pinning swap to zero makes the enforcement deterministic and makes it visible that MemoryMax alone was never the whole ceiling.

Check which kind of host you are on before you start, so you know what you are seeing:

free -h | grep -i swap
nproc

Task 3: Run the service

sudo systemctl daemon-reload
sudo systemctl start stress-test.service
sudo systemctl status stress-test.service --no-pager

With Type=simple the start command returns immediately and the workload runs in the background. Move to Task 4 straight away — the observation window is the 120 seconds the unit is alive.

Task 4: Observe during run (in another terminal)

# Live cgroup view
systemd-cgtop

# Detailed metrics
cat /sys/fs/cgroup/system.slice/stress-test.service/memory.current
cat /sys/fs/cgroup/system.slice/stress-test.service/memory.events

# CPU usage
cat /sys/fs/cgroup/system.slice/stress-test.service/cpu.stat

Task 5: Verify limits

Run this while the unit is still active — the cgroup exists only for the lifetime of the unit’s processes:

CG=/sys/fs/cgroup/system.slice/stress-test.service

watch -n1 "cat $CG/memory.current $CG/memory.events; \
           grep -E 'nr_throttled|throttled_usec' $CG/cpu.stat"

Expect memory.current to sit near 1G and never exceed it, memory.events to show high climbing and then oom_kill incrementing, and nr_throttled to rise continuously because four CPU workers are being held to a 200% quota.

After the unit has exited the cgroup is gone, so ask systemd and the journal instead — those records survive:

systemctl show stress-test -p MemoryPeak -p Result -p NRestarts -p ExecMainStatus
journalctl -u stress-test | grep -iE 'oom|killed|memory'

Task 6: Test without limits (comparison)

sudo systemctl stop stress-test.service

# Remove limits
sudo sed -i '/^CPUQuota/d;/^MemoryMax/d;/^MemoryHigh/d;/^MemorySwapMax/d;/^IOWeight/d' /etc/systemd/system/stress-test.service
grep -E 'CPUQuota|Memory|IOWeight' /etc/systemd/system/stress-test.service || echo "all limits removed"
sudo systemctl daemon-reload
sudo systemctl start stress-test.service

# Observe - now the service uses more resources

Task 7: Document

CGROUP LIMITS LAB
=================
Service: stress-test.service
Limits applied:
- CPUQuota: 200%
- MemoryMax: 1G
- MemoryHigh: 750M
- MemorySwapMax: 0 (so the result does not depend on host swap)
- IOWeight: 500 (100 would be the default: a no-op)

Stress workload:
- 4 CPU workers
- 2 VM workers, 1G each => ~2G of touched anonymous memory
- 120 second duration

Observations (expected):
- cpu.stat nr_throttled > 0 and rising: the 200% quota is
  enforced by throttling, not by refusing to run
- memory.events: high > 0, then oom_kill >= 1
- systemctl show stress-test -p Result => Result=oom-kill
- The unit did NOT complete successfully, and that is the
  correct outcome: the workload asked for 2G against a 1G
  MemoryMax, so the cgroup OOM killer stopped it

Without limits:
- CPU used all 4 cores
- Memory used > 2G
- No cgroup enforcement, no OOM, workload runs to completion

Validation

The lab is complete when you can point at evidence, not recollection:

• memory.events captured during the run shows oom_kill incrementing.
• cpu.stat captured during the run shows nr_throttled rising.
• systemctl show stress-test -p Result reports oom-kill after the run.
• The unlimited comparison run shows memory above 2G and no throttling.
• Your write-up states which directive caused which observation.

Run the assertions:

CG=/sys/fs/cgroup/system.slice/stress-test.service

# During the run
systemctl show stress-test.service -p CPUQuotaPerSecUSec -p MemoryMax -p MemoryHigh -p IOWeight
grep -E 'high|oom_kill' "$CG/memory.events"
grep -E 'nr_throttled|throttled_usec' "$CG/cpu.stat"

# After the run - these survive the cgroup
systemctl show stress-test -p Result -p MemoryPeak -p ExecMainStatus
systemctl is-failed stress-test.service    # expect 'failed'

Expected outcome

The unit failing is the correct result. If it completed successfully with the limits applied, the limits did not take effect - go to Troubleshooting.

Troubleshooting

Cleanup

sudo systemctl stop stress-test.service
sudo rm -f /etc/systemd/system/stress-test.service
sudo systemctl daemon-reload

# Confirm the unit and its cgroup are gone
systemctl status stress-test.service --no-pager 2>&1 | head -3
ls /sys/fs/cgroup/system.slice/ | grep stress-test || echo "cgroup removed"

# Optional: remove the tool if the host did not have it before
sudo apt remove -y stress-ng

The host should be exactly as you found it: no unit file, no cgroup directory, no running stress workload.

What you learned

• cgroup limits are enforced by the kernel through control files in /sys/fs/cgroup, and systemd directives are just a way of writing them.
• MemoryHigh throttles and applies reclaim pressure; MemoryMax is a kill threshold. Putting MemoryMax on a service without also deciding its OOMPolicy and alerting on memory.events means it dies silently.
• A cgroup’s evidence disappears the moment the unit exits, so observation is something you plan for, not something you do afterwards.
• A limit set to its own default (IOWeight=100) reads as configured and does nothing. Check the effective value, not the unit file.