Break/Fix: half the fleet is unreachable, and it is a different half every run

Reported symptoms

The nightly configuration run against 120 web hosts has failed three nights running. Each time roughly half the fleet is reported UNREACHABLE.

The team has spent two days on it:

• They picked three failing hosts and logged into all three by hand. Instant, no delay, no error.
• They re-ran the playbook with --limit against just the failed hosts. It succeeded completely.
• They compared failing and succeeding hosts by rack, by region, by operating system version, by uptime and by SSH daemon version. No correlation.
• They checked interface counters and switch logs. Clean.
• Someone suggested the target hosts were out of memory. They are not.

The one thing nobody noticed for two days is that the failing set is different every night.

Evidence provided

$ ansible web -i inventory -m ansible.builtin.ping -o | grep -c UNREACHABLE

57

$ ansible web -i inventory -m ansible.builtin.ping -o | grep UNREACHABLE | awk '{print $1}' | sort > /tmp/run2.hosts; comm -12 /tmp/run1.hosts /tmp/run2.hosts | wc -l

26

$ ansible-playbook -i inventory site.yml --limit web044 -vvv 2>&1 | grep -A2 'ESTABLISH SSH'

ESTABLISH SSH CONNECTION FOR USER: deploy
SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no ...
kex_exchange_identification: Connection closed by remote host
Connection closed by UNKNOWN port 65535

$ ansible-config dump --only-changed

CONFIG_FILE() = /srv/automation/ansible.cfg
DEFAULT_FORKS(/srv/automation/ansible.cfg) = 50
HOST_KEY_CHECKING(/srv/automation/ansible.cfg) = True

$ ssh bastion.example.com 'journalctl -u ssh --since -10min | grep -c MaxStartups'

412

$ ssh bastion.example.com 'grep -i maxstartups /etc/ssh/sshd_config'

#MaxStartups 10:30:100

Work the evidence before reading on

Three facts do not fit a per-host explanation, and one fits it exactly.

1. The failing set changes between runs. A broken host stays broken. Something that picks a different victim each time is not a property of the victim.
2. --limit against the failures succeeds. What is different about that run, other than which hosts it names?
3. The failure happens at kex_exchange_identification, which is before authentication - before keys, before sshd has any idea who is calling.

Before continuing: what does every one of those 50 connections have in common with the other 49, and what does a single manual ssh not share with anything?

Root cause

1. Every connection goes through one daemon

The fleet is reached through bastion.example.com. Whether that is a ProxyJump in the SSH client configuration or an explicit ansible_ssh_common_args, the shape is the same: 120 target hosts, one front door.

forks controls how many hosts Ansible works on simultaneously. Each fork opens its own SSH connection. Raising forks from 5 to 50 did not change anything about any target host - it changed the number of simultaneous connections arriving at one daemon from 5 to 50.

2. MaxStartups drops connections on purpose, at random

OpenSSH limits how many connections may sit in the pre-authentication state at once. The default is 10:30:100, read as three numbers:

This is a denial-of-service control, and it is doing its job. Above the first threshold the daemon drops connections it has not authenticated, chosen at random. Random selection is why the failing set changes every night, and it is the single most important clue in the incident.

Because the drop happens before the protocol banner exchange completes, the client cannot report anything more specific than the connection having been closed by the peer. That message describes the symptom accurately and says nothing about the cause.

3. Nothing retried

Ansible retries a connection only when SSH exits with return code 255, and reconnection_retries defaults to 0 in any case. A dropped pre-authentication connection is therefore a terminal UNREACHABLE for that host in that run, with no second attempt and no backoff.

That is also why --limit looked like a fix. Limiting the run to 57 hosts still uses forks: 50, but 57 hosts finish in fewer overlapping waves and the pre-authentication count stays lower for less of the run. The success was luck with better odds, not a different code path.

Resolution

1. Restore the previous concurrency immediately. Set forks = 10 in ansible.cfg or pass --forks 10, and confirm with ansible-config dump --only-changed that the value the run will use is the value you set.
2. Re-run the reachability check five times, not once. ansible web -i inventory -m ansible.builtin.ping must report zero unreachable on every attempt; a probabilistic failure needs repetition to be ruled out.
3. Watch the bastion during those runs with journalctl -fu ssh and confirm no throttling lines appear. This is the authoritative check, because it is the component that was refusing.
4. Confirm SSH multiplexing is active. The default ssh_args include ControlMaster=auto and ControlPersist=60s; verify control sockets appear under the control path directory during a run. Multiplexing means one connection setup per host rather than one per task, which is the difference between 50 concurrent setups and 50 concurrent sessions.
5. Decide the real concurrency budget with the team that owns the bastion. Raising MaxStartups there is a legitimate answer, but it costs memory and process slots on their machine and it is their capacity to spend.
6. If the fleet needs more than the bastion can give, split the run by region or add a second bastion. Two daemons at ten each is not the same problem as one daemon at twenty.
7. Record the chosen forks value and the reason in the repository, next to the setting. The next person to describe a forks increase as a performance improvement should find the note first.

Verification

1. Five consecutive reachability runs report zero unreachable hosts. One clean run is not evidence about a random failure, and this is the check the original investigation never performed.
2. The bastion logs no throttling during a full run. journalctl -u ssh --since -30min | grep -c MaxStartups returns 0.
3. The limit is known rather than assumed. In a maintenance window, run once with --forks deliberately above the agreed ceiling and confirm the throttling lines and unreachable hosts return. A ceiling you have never touched is a guess, and this is the check that can fail.
4. Multiplexing is in use. Control sockets exist under the control path directory during a run and disappear after ControlPersist expires.
5. The effective configuration is what you think it is. ansible-config dump --only-changed run from the same directory the pipeline uses shows the intended forks, and names the configuration file it came from.
6. The full playbook completes end to end with no unreachable hosts, twice, at the new setting. Reachability and a real run exercise different amounts of the connection budget.
7. Alerting exists on the bastion for throttling events, and it has been tested by triggering one.

Prevention

• Read intermittency as structural evidence. A failure set that changes between runs is a shared-resource fault; a failure set that stays the same is a per-host fault. Deciding which one you have takes two runs and a comm, and it eliminates half the possible causes.
• Treat forks as a claim about shared infrastructure. Name the chokepoint before raising it, and find out what its limit is from the team that owns it.
• Keep SSH multiplexing on. ControlMaster=auto with ControlPersist turns one connection setup per task into one per host, which is the largest single reduction in pressure on the shared path.
• Monitor and alert on MaxStartups throttling wherever a bastion exists. It is the only place in this incident where the truth was written down, and nobody was reading it.
• Change one concurrency setting at a time, and record the value. Concurrency changes have no effect until they have a large effect, which makes them hard to attribute after the fact.
• Never conclude from a successful manual login that the connection path is healthy. A sequential test cannot observe a concurrency limit.