Scaling the management plane - what breaks between 50 and 5000 nodes

A fleet architecture does not degrade smoothly. It works, and then a specific component hits a specific wall, and the symptoms look like a general slowness that nobody can localise.

This lesson is the map of those walls. The useful thing about them is that they are predictable: you can name which component will fail next from the node count alone, and fix it before it becomes an incident.

The three regimes

The row that surprises people is the last one at 50 nodes. At that size nothing technical breaks; the fleet is limited by undocumented knowledge in one person’s head. Solving the wrong problem there - buying a management platform when the actual gap is written procedure - is common and expensive.

Push versus pull

At 50 nodes the choice is aesthetic. By 500 it is structural.

Push (ansible, salt in some modes, ssh in a loop) is synchronous. You run it, you watch it, you get an exit code. That immediacy is genuinely valuable and it is why push survives at scale for orchestration.

Its costs at scale:

• Fan-out is O(n) on one operator’s machine or one control node. Forks, file descriptors and SSH handshakes all bound it.
• Failure is partial, and partial failure is the hardest state to reason about. Forty hosts converged, three timed out, one is half-way through a package transaction.
• It only reaches hosts that are up at that moment. A host that was rebooting during the run is now silently different from its peers, and nothing records that.

Pull (puppet, chef, salt in masterless mode, GitOps-style agents) is asynchronous. Each host fetches its desired state on a schedule and applies it.

Its costs:

• “Did it apply?” is answered by a reporting pipeline, not by an exit code. If you do not build the reporting, you have automation with no feedback.
• Convergence is eventual. An urgent change takes up to one interval unless you also have a way to trigger a run.

Its advantage is that the server is stateless and horizontal: adding capacity is adding replicas, and a host that was down converges when it returns without anybody noticing.

The arrangement that works at 5000 is pull for desired state, push for orchestration. Configuration converges continuously; the operations that need ordering and immediacy - a rolling restart, a drain, an emergency change - are pushed at the specific hosts they concern.

The thundering herd

5000 agents on a 30-minute interval averages under three runs per second, which sounds trivial. The average is not the problem. The problem is that every one of those hosts was configured by the same automation, so their timers are aligned.

$ systemctl show puppet-run.timer -p RandomizedDelaySec -p OnUnitActiveSec

OnUnitActiveSec=30min
RandomizedDelaySec=0

Splay is set once and prevents a category of incident:

sudo systemctl edit puppet-run.timer
# [Timer]
# RandomizedDelaySec=25m

systemctl list-timers puppet-run.timer

The same reasoning applies to every periodic fleet-wide activity: patching, backup, log rotation, certificate renewal, inventory reporting. Anything that runs on all hosts on a schedule needs jitter proportional to the fleet size.

Credential issuance rate

Short-lived credentials are the right design and they turn identity into a throughput problem.

5000 hosts with a 24-hour certificate lifetime is one issuance every 17 seconds, sustained, forever, assuming perfect spread. Halve the lifetime and it doubles. Add per-service identities and it multiplies again.

That is a small number and it is not the design constraint. The design constraints are:

• The recovery spike, from the callout above.
• Renewal timing. Agents that renew at a fixed fraction of the lifetime - say two-thirds - will cluster if the certificates were all issued during the same fleet build. Renew at a randomised fraction of the remaining lifetime.
• Revocation. At 5000 hosts a CRL is a large object fetched frequently. Short lifetimes are what let you keep it small, because an expired certificate does not need revoking. The TLS part covers the trade-off in detail.
• CA availability. If issuance is down for longer than the shortest certificate lifetime, hosts start failing to authenticate. The lifetime is therefore also your maximum tolerable CA outage, which is an argument against making it too short.

Cardinality and volume

Two resources scale with fleet size and both have a cliff rather than a slope.

Metric cardinality is the number of distinct time series. It is multiplicative: exporters per host, metrics per exporter, label combinations per metric, hosts.

200 series per exporter
  x 3 exporters per host
  x 5000 hosts
  = 3,000,000 active series

Three million is manageable with a properly sized store. The failure comes from one careless label. Add a label whose values are unbounded - a request path, a container ID, a customer identifier - and the multiplication runs away.

The rule that holds: never combine a label that grows with fleet size with a label that grows with anything else. Per-host metrics get host labels and nothing high-cardinality. Per-request detail goes to logs or traces, which are built for it.

Log volume is simpler arithmetic and a bigger bill:

50 MB per host per day x 5000 hosts = 250 GB/day
                                    = ~91 TB/year before compression

At 50 hosts you keep everything for a year without thinking. At 5000 you route by severity - security and audit events to long retention, application debug to short - and you have an explicit conversation about what compliance actually requires rather than keeping everything because deleting feels risky.

The plane must not depend on the fleet

This is the failure that turns an incident into an unrecoverable one, and it is invisible until the day it matters.

The config server’s TLS certificate is issued by the internal CA. The internal CA runs on a host whose configuration is managed by the config server. Both are monitored by a system that authenticates through the same CA. Nothing is wrong on any ordinary day.

Then a bad catalogue is pushed, the CA host stops serving, the config server’s certificate expires while the CA is down, and the tool you would use to fix the CA cannot authenticate.

The requirements that break the cycle:

• The plane’s own hosts are bootstrappable without the plane: a documented, tested manual path, and console or out-of-band access that does not depend on fleet identity.
• Long-lived credentials for the plane’s core components, deliberately different from the fleet’s short-lived ones.
• Changes to the plane get a canary of their own. The plane is the highest-blast-radius system in the estate and it usually has the weakest change process, because it is “just infrastructure”.
• Draw the dependency graph and look for cycles. If you cannot draw it, that is the finding.

The human interface

At 5000 nodes, an operator who types a hostname is doing something the process should have prevented. The interface is groups: role, environment, wave, region.

Practical consequences:

• Every host has machine-readable facts, and every operation selects on facts rather than names.
• A one-off change to one host is an exception that needs a reason, because it creates a snowflake nobody will remember.
• Reporting is aggregate by default: “97% converged, 41 hosts failing, grouped by error” rather than 5000 lines.
• Anything that requires a human to be correct 5000 times will be wrong.

Knowledge check