LinuxLXVI · Capacity Planning for ClustersThe plan
Writing the capacity plan - finding the binding constraint
What you'll learn
- Enumerate the resources that exhaust independently of CPU
- Identify the binding constraint from a set of per-resource run-out dates
- Assemble a capacity plan whose output is a table of dated, owned actions
- Recognise the soft limits that exhaust before any hardware does
Prerequisites
Verified against Ubuntu 24.04 LTS · Debian 12 (Bookworm) · RHEL 9.x · Rocky Linux 9.x · AlmaLinux 9.x · Linux kernel 6.1 LTS / 6.6 LTS · systemd 255+ · OpenSSH 8.7p1 (RHEL 9) / 9.6p1 (Ubuntu 24.04) · nftables 1.0.x · chrony 4.x · Pacemaker 2.1.x · Corosync 3.1.x · 2026-08-11
The previous four lessons produce a number and a date for one resource at a time. A capacity plan is the document that puts them side by side and answers the only question the organisation actually asked: what runs out first, and when?
That question has a name. The resource with the nearest run-out date is the binding constraint, and it is the only one whose date matters, because relieving any other constraint changes nothing about when the service degrades.
The resources that run out
CPU is the resource everyone measures and rarely the one that binds. Forecast all of these, each against its own ceiling.
| Resource | Measure with | Typical ceiling | Failure when exceeded |
|---|---|---|---|
| CPU | sar -u, PSI cpu | Survivable ceiling for N | Latency climbs, then queues grow without bound |
| Memory | sar -r on kbavail, PSI memory | 80% committed, swap near zero | Reclaim stalls, then the OOM killer picks a victim |
| Filesystem space | df on a timer | 85% | Writes fail; on a full root, so does logging and often sshd |
| Inodes | df -i | 80% | ENOSPC with space visibly free - the confusing one |
| Disk throughput and IOPS | iostat -xz await, sar -d | Measured knee, not %util | Latency, then application timeouts |
| Network | sar -n DEV, sar -n EDEV | 60-70% of link rate | Buffer bloat, retransmits, drops |
Each needs the same three inputs from the forecasting lesson: current p95, ceiling, growth rate. Each produces its own date.
$ df -h /srv; df -i /srvFilesystem Size Used Avail Use% Mounted on
/dev/mapper/vg0-srv 2.0T 1.3T 700G 65% /srv
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/vg0-srv 131072000 124518402 6553598 95% /srvIllustrative output
Soft limits exhaust before hardware does
A second class of ceiling has nothing to do with hardware. These are configured limits, usually set to a default that was generous in 2005, and they bind long before any disk or CPU does.
$ ss -s | head -3; sysctl fs.file-nr net.ipv4.ip_local_port_range net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max 2>/dev/nullTotal: 412
TCP: 118 (estab 41, closed 52, orphaned 0, timewait 51)
fs.file-nr = 9536 0 9223372036854775807
net.ipv4.ip_local_port_range = 32768 60999
net.netfilter.nf_conntrack_count = 21884
net.netfilter.nf_conntrack_max = 262144Illustrative output
The ones worth putting in the plan:
- File descriptors: per-service LimitNOFILE, and fs.file-max system-wide
- Conntrack table size on any host doing NAT or stateful filtering
- Ephemeral port range, which caps outbound connections per destination tuple
- Process and thread limits: TasksMax in systemd, and pids.max in the cgroup
- Database connections and pool sizes, which usually bind before the database host does
- Cloud and hypervisor quotas: instances, addresses, volumes, API rate limits
Every one of these has a failure mode that presents as an application error rather than a capacity error, which is why they are found late. They are also the cheapest to fix - most are one configuration line - so the value of forecasting them is almost entirely in knowing about them in advance.
Finding the binding constraint
Lay the dates next to each other. The plan writes itself from there.
Cluster: srv-file (3 nodes) Measured: 2026-08-11
Resource Current p95 Ceiling Growth Run-out Lead Order by
------------ ----------- ------- --------- -------- ---- ----------
CPU 41% 66.7% +12%/yr 2030-01 8w 2029-11
Memory 58% 80% +18%/yr 2028-02 8w 2027-12
FS space (srv) 65% 85% 18 GB/wk 2027-02 15w 2026-11
FS inodes (srv) 95% 80% +2.1%/mo BREACHED - immediate
Disk await 4 ms 12 ms +9%/yr 2029-06 15w 2029-03
Network 22% 65% +25%/yr 2030-08 6w 2030-06
Conntrack 8% 80% +40%/yr 2033-01 0 n/a
BINDING CONSTRAINT: inodes on /srv, already past ceiling.
NEXT AFTER THAT: filesystem space on /srv, order by 2026-11.
The table makes three things obvious that a prose report hides.
One row is already red. Inodes are past the ceiling now. Everything else is a 2027 problem or later, so the plan has exactly one urgent item, and arguing about the CPU forecast is a waste of the meeting.
The second constraint is on the same volume. Both /srv
rows point at the same storage, which means one project - a
larger, XFS-backed volume - relieves the binding constraint and
the next one together. That is worth knowing before you scope
the work.
Lead time reorders the list. Network has the latest run-out date and a six-week lead time; filesystem space runs out three years earlier but needs fifteen weeks. Sorting by run-out date gives a different order from sorting by order-by date, and the order-by column is the one that drives the calendar.
What the plan must contain
Keep it short enough that it gets read and updated. Five sections is enough.
- Scope and date: which systems, measured over which window, at which sampling interval
- The baseline table: current p95 and ceiling per resource, per role
- The forecast table: growth rate, run-out date, lead time, order-by date
- The binding constraint, named explicitly, with the next two behind it
- Actions: one row each, with an owner, an order-by date, and the cost
Then two rules about keeping it alive.
Re-forecast quarterly. A capacity plan older than one quarter is a historical document. Put the re-forecast on a calendar with an owner, because it is exactly the kind of work that never becomes urgent until it is too late to matter.
Re-forecast after any step change. A large customer, a new region, a retention policy change, a migration, or an architectural change all invalidate the fit. A plan carried across a step change is worse than no plan, because it carries the authority of having been measured.
Knowledge check
Knowledge check · 6 questions
Q1. A filesystem shows 65% space used and 95% inodes used. Which is the binding constraint, and what does it mean for the plan?
Q2. Why is relieving a non-binding constraint usually wasted money?
Q3. A service doubles its request rate while transferring the same total bytes. Which capacity limit is most likely to bind first?
Q4. Which of these are soft limits that can exhaust long before the underlying hardware does? Select all that apply.
Q5. Sorting the forecast by run-out date gives the same priority order as sorting it by order-by date.
Q6. A capacity plan should be re-forecast after onboarding a large new customer, even if the quarterly re-forecast is not yet due.
Passing score: 75%. Answers are checked in this browser.