Noisy neighbours and how to prevent them

A noisy neighbour is one workload consuming more than its share of a shared resource and degrading the others. The characteristic that makes it hard is not the contention — it is the shape of the evidence: one tenant is badly degraded, and every host-level metric looks normal.

Load average is fine. CPU utilisation is fine. Free memory is fine. And one service’s p99 has tripled. This lesson is mostly about finding the offender when the dashboards say there is no problem.

What containers actually share

The last four rows are where the difficult incidents come from, because there is no flag to set and therefore no line item anyone forgot.

Two are worth naming explicitly. Docker has no network bandwidth limit — no --network-bps, no equivalent. Rate limiting a container’s traffic means tc queueing disciplines on its veth interface, or an external network policy layer. And page cache is shared and unallocatable; the next callout is entirely about it.

Finding the offender: per-cgroup pressure

Pressure Stall Information is the tool for this, and the part most people miss is that PSI exists per cgroup, not only at /proc/pressure. That is what turns “the host is contended” into “this container is contended, and that one is causing it”.

$ for r in cpu memory io; do printf '%-7s ' "$r"; head -1 "/proc/pressure/$r"; done

cpu     some avg10=31.44 avg60=28.10 avg300=19.77 total=884213551
memory  some avg10=0.11 avg60=0.09 avg300=0.05 total=9912004
io      some avg10=62.08 avg60=58.31 avg300=44.02 total=2201884391

That capture says the host is stalled on I/O 62% of the time and on CPU 31%, while memory is fine. Load average would show “high” and tell you nothing about which of the three to chase.

for c in $(docker ps -q); do
  name=$(docker inspect --format '{{.Name}}' "$c" | tr -d /)
  pid=$(docker inspect --format '{{.State.Pid}}' "$c")
  cg=$(awk -F: '/^0::/{print $3}' "/proc/$pid/cgroup")
  cpu=$(awk '/^some/{print $2}' /sys/fs/cgroup"$cg"/cpu.pressure 2>/dev/null)
  io=$(awk '/^some/{print $2}' /sys/fs/cgroup"$cg"/io.pressure 2>/dev/null)
  printf '%-22s cpu:%-14s io:%s\n' "$name" "$cpu" "$io"
done

Read the result as two lists:

• High pressure, low usage — the victim. It is waiting.
• Low pressure, high usage — the offender. It is getting everything it asks for.

That inversion is the whole diagnostic trick, and it is the opposite of what intuition suggests. The container with the alarming graph is usually not the one to limit.

Then confirm with the per-resource counters covered in the other lessons in this part: cpu.stat (nr_throttled), io.stat (rbytes/wbytes per device), memory.events.

The mitigations, and what each one actually does

1. Resource limits on every container. The foundation. A container without --memory and --cpus is unbounded by definition, and no amount of monitoring compensates.
2. CPU pinning (--cpuset-cpus). Puts latency-sensitive workloads on their own cores. Also cuts cache-line contention and scheduler migration, which a quota does not. Needs a registry — two containers pinned to the same pair of cores contend hard while the rest of the machine idles.
3. Weights within a tier (--cpu-shares, --blkio-weight). Decides who wins when a contended moment arrives. Remember these only act under contention, and that --blkio-weight is inert on hosts using the none scheduler without iocost.
4. --memory-reservation. Sets memory.low, telling the kernel to reclaim from other cgroups first. The only lever that helps a victim rather than restraining an offender.
5. Separate hosts. Underrated. If a workload’s contention profile is genuinely incompatible with its neighbours, moving it is cheaper than a year of tuning.

Capacity: the arithmetic

docker ps -q | xargs -r docker inspect \
  --format '{{.Name}} {{.HostConfig.NanoCpus}}' \
  | awk '{sum += $2} END {printf "allocated: %.2f cores\n", sum/1e9}'
echo "host cores: $(nproc)"

docker ps -q | xargs -r docker inspect \
  --format '{{.Name}} {{.HostConfig.Memory}}' \
  | awk '{if ($2 == 0) unl++; else sum += $2}
         END {printf "allocated: %.1f GiB, unlimited containers: %d\n", sum/1073741824, unl}'
awk '/MemTotal/ {printf "host: %.1f GiB\n", $2/1048576}' /proc/meminfo

Two rules that are not the obvious ones:

• Memory must not be oversubscribed. The sum of --memory plus a host reservation of 15–20% must fit in RAM. Memory a container holds is memory nobody else can have, so overcommitting it means the global OOM killer decides your availability.
• CPU may be oversubscribed, deliberately. Summing --cpus to more than nproc is normal and usually correct, because workloads do not peak together. What matters is not the sum but the throttling ratio under real load. A host at 250% CPU allocation with nr_throttled/nr_periods near zero everywhere is properly packed; a host at 90% allocation with 20% throttling is not.

The single number that matters more than either sum is the count of unlimited containers. One of those makes the arithmetic irrelevant.

Operational discipline

• Every new service ships with --memory, --cpus and --pids-limit set in the Compose file. Gate it in CI; a service without limits fails the pipeline.
• Track the count of unlimited containers per host as a metric. It should be zero, and a non-zero value is a specific ticket rather than a trend.
• Alert on per-container some avg60 pressure, not on host CPU utilisation. Utilisation tells you the machine is busy; pressure tells you somebody is waiting.
• Quarterly, compare memory.peak and pids.peak against the configured limits and adjust in both directions. Limits that only ever go up are not limits.
• A noisy-neighbour incident gets a postmortem with the owning team, because the fix is nearly always a missing limit in somebody’s manifest rather than an on-call action.

Knowledge check