Skip to main content
RunBook Academy

LinuxLXVI · Capacity Planning for ClustersThe plan

Writing the capacity plan - finding the binding constraint

Advanced⏱ ~15 minbashdfsssysctl

What you'll learn

  • Enumerate the resources that exhaust independently of CPU
  • Identify the binding constraint from a set of per-resource run-out dates
  • Assemble a capacity plan whose output is a table of dated, owned actions
  • Recognise the soft limits that exhaust before any hardware does

Prerequisites

Verified against Ubuntu 24.04 LTS · Debian 12 (Bookworm) · RHEL 9.x · Rocky Linux 9.x · AlmaLinux 9.x · Linux kernel 6.1 LTS / 6.6 LTS · systemd 255+ · OpenSSH 8.7p1 (RHEL 9) / 9.6p1 (Ubuntu 24.04) · nftables 1.0.x · chrony 4.x · Pacemaker 2.1.x · Corosync 3.1.x · 2026-08-11

Not yet marked complete on this device.

The previous four lessons produce a number and a date for one resource at a time. A capacity plan is the document that puts them side by side and answers the only question the organisation actually asked: what runs out first, and when?

That question has a name. The resource with the nearest run-out date is the binding constraint, and it is the only one whose date matters, because relieving any other constraint changes nothing about when the service degrades.

The resources that run out

CPU is the resource everyone measures and rarely the one that binds. Forecast all of these, each against its own ceiling.

ResourceMeasure withTypical ceilingFailure when exceeded
CPUsar -u, PSI cpuSurvivable ceiling for NLatency climbs, then queues grow without bound
Memorysar -r on kbavail, PSI memory80% committed, swap near zeroReclaim stalls, then the OOM killer picks a victim
Filesystem spacedf on a timer85%Writes fail; on a full root, so does logging and often sshd
Inodesdf -i80%ENOSPC with space visibly free - the confusing one
Disk throughput and IOPSiostat -xz await, sar -dMeasured knee, not %utilLatency, then application timeouts
Networksar -n DEV, sar -n EDEV60-70% of link rateBuffer bloat, retransmits, drops

Each needs the same three inputs from the forecasting lesson: current p95, ceiling, growth rate. Each produces its own date.

Read-only / Safetwo ceilings on one filesystem
$ df -h /srv; df -i /srv
Filesystem      Size  Used Avail Use% Mounted on
/dev/mapper/vg0-srv  2.0T  1.3T  700G  65% /srv
Filesystem       Inodes  IUsed   IFree IUse% Mounted on
/dev/mapper/vg0-srv 131072000 124518402 6553598   95% /srv

Illustrative output

Soft limits exhaust before hardware does

A second class of ceiling has nothing to do with hardware. These are configured limits, usually set to a default that was generous in 2005, and they bind long before any disk or CPU does.

Read-only / Safethe limits nobody forecasts
$ ss -s | head -3; sysctl fs.file-nr net.ipv4.ip_local_port_range net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max 2>/dev/null
Total: 412
TCP:   118 (estab 41, closed 52, orphaned 0, timewait 51)
fs.file-nr = 9536	0	9223372036854775807
net.ipv4.ip_local_port_range = 32768	60999
net.netfilter.nf_conntrack_count = 21884
net.netfilter.nf_conntrack_max = 262144

Illustrative output

The ones worth putting in the plan:

  • File descriptors: per-service LimitNOFILE, and fs.file-max system-wide
  • Conntrack table size on any host doing NAT or stateful filtering
  • Ephemeral port range, which caps outbound connections per destination tuple
  • Process and thread limits: TasksMax in systemd, and pids.max in the cgroup
  • Database connections and pool sizes, which usually bind before the database host does
  • Cloud and hypervisor quotas: instances, addresses, volumes, API rate limits

Every one of these has a failure mode that presents as an application error rather than a capacity error, which is why they are found late. They are also the cheapest to fix - most are one configuration line - so the value of forecasting them is almost entirely in knowing about them in advance.

Finding the binding constraint

Lay the dates next to each other. The plan writes itself from there.

Cluster: srv-file (3 nodes)          Measured: 2026-08-11

Resource      Current p95   Ceiling   Growth      Run-out    Lead   Order by
------------  -----------   -------   ---------   --------   ----   ----------
CPU                  41%      66.7%   +12%/yr     2030-01     8w    2029-11
Memory               58%        80%   +18%/yr     2028-02     8w    2027-12
FS space (srv)       65%        85%   18 GB/wk    2027-02    15w    2026-11
FS inodes (srv)      95%        80%   +2.1%/mo    BREACHED     -    immediate
Disk await          4 ms      12 ms   +9%/yr      2029-06    15w    2029-03
Network              22%        65%   +25%/yr     2030-08     6w    2030-06
Conntrack             8%        80%   +40%/yr     2033-01      0    n/a

BINDING CONSTRAINT: inodes on /srv, already past ceiling.
NEXT AFTER THAT:    filesystem space on /srv, order by 2026-11.

The table makes three things obvious that a prose report hides.

One row is already red. Inodes are past the ceiling now. Everything else is a 2027 problem or later, so the plan has exactly one urgent item, and arguing about the CPU forecast is a waste of the meeting.

The second constraint is on the same volume. Both /srv rows point at the same storage, which means one project - a larger, XFS-backed volume - relieves the binding constraint and the next one together. That is worth knowing before you scope the work.

Lead time reorders the list. Network has the latest run-out date and a six-week lead time; filesystem space runs out three years earlier but needs fifteen weeks. Sorting by run-out date gives a different order from sorting by order-by date, and the order-by column is the one that drives the calendar.

What the plan must contain

Keep it short enough that it gets read and updated. Five sections is enough.

  1. Scope and date: which systems, measured over which window, at which sampling interval
  2. The baseline table: current p95 and ceiling per resource, per role
  3. The forecast table: growth rate, run-out date, lead time, order-by date
  4. The binding constraint, named explicitly, with the next two behind it
  5. Actions: one row each, with an owner, an order-by date, and the cost

Then two rules about keeping it alive.

Re-forecast quarterly. A capacity plan older than one quarter is a historical document. Put the re-forecast on a calendar with an owner, because it is exactly the kind of work that never becomes urgent until it is too late to matter.

Re-forecast after any step change. A large customer, a new region, a retention policy change, a migration, or an architectural change all invalidate the fit. A plan carried across a step change is worse than no plan, because it carries the authority of having been measured.

Knowledge check

Knowledge check · 6 questions

  1. Q1. A filesystem shows 65% space used and 95% inodes used. Which is the binding constraint, and what does it mean for the plan?

  2. Q2. Why is relieving a non-binding constraint usually wasted money?

  3. Q3. A service doubles its request rate while transferring the same total bytes. Which capacity limit is most likely to bind first?

  4. Q4. Which of these are soft limits that can exhaust long before the underlying hardware does? Select all that apply.

  5. Q5. Sorting the forecast by run-out date gives the same priority order as sorting it by order-by date.

  6. Q6. A capacity plan should be re-forecast after onboarding a large new customer, even if the quarterly re-forecast is not yet due.

Passing score: 75%. Answers are checked in this browser.