Reaching a fleet through a bastion · Ansible

Most production fleets are not directly reachable from the machine you run Ansible on, and that is deliberate. A segment whose SSH port is open to the office network is a segment whose SSH port is open to whoever is on the office network.

So there is a bastion: one hardened, monitored, aggressively logged host that is reachable from outside, and through which everything else is reached. The security argument is sound. The operational consequence is that your entire automation run now depends on one machine, and that is a fact worth designing around rather than discovering.

The four argument settings, and which tools they reach

Ansible does not run one program. Depending on the transfer method it runs ssh, and it may run sftp or scp. Four settings let you add options, and they differ in which of those they reach:

Read-only / Safethe four, verbatim— From ansible-doc -t connection ansible.builtin.ssh on ansible-core 2.21.3. All four default to the empty string.

$ ansible-doc -t connection ansible.builtin.ssh

   ssh_common_args  Common extra args for all SSH CLI tools.
      vars:
        - name: ansible_ssh_common_args
      default: ''

 ssh_extra_args  Extra exclusive to the SSH CLI.
      vars:
        - name: ansible_ssh_extra_args
      default: ''

 sftp_extra_args  Extra exclusive to the 'sftp' CLI
      vars:
        - name: ansible_sftp_extra_args
      default: ''

 scp_extra_args  Extra exclusive to the 'scp' CLI
      vars:
        - name: ansible_scp_extra_args
      default: ''

The documentation for the connection details states the behaviour that makes them usable: “Ansible always appends this setting to the default sftp, scp, and ssh.”

Appends. Not replaces. This is the entire difference between these settings and ssh_args, and it is the reason ssh_common_args is the correct place to put a ProxyJump while ssh_args is not.

Routing belongs in ssh_common_args, because a file transfer needs to reach the host by the same path the command did. If you put ProxyJump in ssh_extra_args instead, your tasks will connect and your file copies will fail, in a way that looks like a module bug.

scp_extra_args has one common legitimate use, mentioned in the transport lesson: -O, for an estate pinned to scp on OpenSSH 9.0 or later.

Proving that it appends

Do not take this on trust — the failure mode when a setting replaces instead of appending is silent, and losing ControlPersist this way is one of the most confusing performance regressions in this course.

Read-only / Safethe group that lives behind the bastion

[web]
web1.example.com ansible_host=192.0.2.10

[web:vars]
ansible_ssh_common_args=-o ProxyJump=bastion@198.51.100.7

Read-only / Safethe command line Ansible built— Executed on ansible-core 2.21.3. The target is in the documentation range and unreachable by design; only the built command line matters here. Wrapped for the page.

$ ansible-playbook -i inv2.ini ping.yml -vvvv | grep -m1 'SSH: EXEC'

<192.0.2.10> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s
-o KbdInteractiveAuthentication=no
-o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey
-o PasswordAuthentication=no -o ConnectTimeout=10
-o ProxyJump=bastion@198.51.100.7
-o 'ControlPath="/home/ebrandi/.ansible/cp/fa537f329b"'
-o NumberOfPasswordPrompts=1 192.0.2.10 '/bin/sh -c '"'"'echo ~'"'"''

-C, ControlMaster=auto and ControlPersist=60s are all still there, with ProxyJump added among them. Compare that to what happens when somebody writes the same thing into ssh_args: the defaults vanish, multiplexing stops, and every task pays a full handshake — through the bastion, which is where it hurts most.

Put it on the group, not in ansible.cfg

ansible_ssh_common_args is a vars setting, so it can be scoped:

# inventory/production/group_vars/behind_dmz_bastion.yml
# All of production is reached through bastion-a. Direct access to
# 192.0.2.0/24 is intentionally not routed from the controller.
ansible_ssh_common_args: '-o ProxyJump=svc-ansible@bastion-a.example.com'

Three reasons this belongs on the group rather than in the config file.

Fleets are not uniform. The controller itself, appliances with their own access path, and the bastion when you manage it as a host all need to be reached directly. A global ProxyJump sends the bastion through itself.

It documents the topology. Somebody reading group_vars/ learns which segments exist and how each is reached. Somebody reading ansible.cfg learns that everything goes through one box, which stops being true the moment a second segment appears.

It survives a second controller. Config files are per-controller; inventory travels with the repository. A run from a different machine inherits the routing automatically.

`ProxyJump` versus `ProxyCommand`

Older material uses ProxyCommand with nc or ssh -W. ProxyJump (-J) is the modern form and is what you should write. Both establish a connection through an intermediate host; the differences that matter are about verification and legibility.

	`ProxyJump`	`ProxyCommand`
Syntax	`-o ProxyJump=user@bastion`	a shell command string
Target host key	verified by your client, end to end	verified by your client if the command is `ssh -W`; not if the command terminates the session
Bastion host key	verified by your client	verified by your client
Multiple hops	comma-separated list	nested commands
Failure messages	name the hop that failed	come from whatever you invoked

The verification row is the one to internalise.

With ProxyJump, the bastion forwards an encrypted stream. Your client completes the SSH handshake with the target, so it checks the target’s host key itself, and the bastion is a router that cannot read or alter the traffic. You get two independent verifications: the bastion, then the target.

That property is what makes a bastion an acceptable design rather than a mandatory trust anchor. A compromised bastion can see that you connected and to where, and it can deny service — but it cannot substitute itself for the target without failing the target’s host key check on your controller.

A hand-rolled ProxyCommand that opens its own session and runs something on the bastion loses this. Now the bastion is an endpoint, it holds plaintext, and you are trusting it fully. ssh -W %h:%p retains the property; almost every other shape does not. Use ProxyJump and the question does not arise.

Read-only / Safetwo hops, when the estate has an inner bastion

# inventory/production/group_vars/pci_segment.yml
# Controller -> edge bastion -> PCI bastion -> host.
# Every hop is host-key verified independently by the controller.
ansible_ssh_common_args: >-
-o ProxyJump=svc-ansible@bastion-edge.example.com,svc-ansible@bastion-pci.example.com

Under the hood

ProxyJump is implemented by the client as an internally generated ProxyCommand using -W, so the two are not different mechanisms — one is a safe spelling of the other.

That matters for a practical reason: the known_hosts used to verify the bastion is the controller’s, not some file on the bastion. All verification happens on your machine. So the managed known_hosts from the previous lesson needs an entry for every bastion as well as every target, and the bastion entries are the ones most likely to be missing, because nobody thinks of the bastion as a managed host.

It also explains the shape of the -vvv output when a jump fails. You will see two connections being set up, with the inner one reported through the outer, and the error naming which one refused. Reading that carefully is usually the whole diagnosis.

The bastion is a single point of failure for the whole run

This is the part that belongs in a design review.

Two hundred hosts behind one bastion means every task, on every host, for the entire run, traverses one machine. Its CPU, its sshd limits, its network interface and its uptime are now properties of your automation.

forks does not mean what you think it means once there is a bastion

forks controls how many hosts Ansible works on simultaneously. With a bastion, it also controls how many simultaneous SSH connections arrive at one sshd.

forks = 50 is a reasonable number for a fleet and an unreasonable number to point at a single unauthenticated-connection limit. From sshd_config(5):

MaxStartups — Specifies the maximum number of concurrent unauthenticated connections to the SSH daemon. Additional connections will be dropped until authentication succeeds or the LoginGraceTime expires for a connection. The default is 10:30:100.

… sshd(8) will refuse connection attempts with a probability of rate/100 (30%) if there are currently start (10) unauthenticated connections. The probability increases linearly and all connection attempts are refused if the number of unauthenticated connections reaches full.

Read the numbers. Beyond ten connections that have not yet authenticated, the default configuration starts refusing new ones at random, with the probability rising until everything is refused at 100.

So a run with forks = 50 through a default-configured bastion does not fail. It partially fails, differently every time, on a different subset of hosts, with no pattern anybody can find in the playbook or the inventory.

That signature is worth memorising, because it is one of the most commonly misdiagnosed failures in this whole course:

A varying number of hosts come back unreachable on each run.
It is a different set each time.
Re-running with --limit on the failed hosts succeeds, because there are now few enough of them.
Nothing is wrong with any individual host, and each one passes a manual ssh test.
The bastion is not obviously loaded, because the connections are refused before they consume anything.

The last two are what send people down the wrong path. Every host you test individually works, so it must be intermittent, so it must be the network.

Three fixes, and you should probably do the first and third:

Reduce forks for the groups behind that bastion, to something the bastion can accept. This costs wall-clock time and requires no change to a shared piece of infrastructure.
Raise MaxStartups on the bastion, deliberately, with the understanding that the limit exists as a denial-of-service defence and you are trading some of it away. Not something to do silently.
Enable connection reuse, which is the next lesson and the biggest single lever here. With multiplexing, a run against 200 hosts opens 200 connections once, not one per task — which removes most of the contention rather than accommodating it.

Sizing forks against a bastion

The arithmetic that matters is connections in flight, not hosts. As a starting point, keep forks for a bastioned group at or below the bastion’s MaxStartups start value, and confirm the value rather than assuming the default:

ssh svc-ansible@bastion-a.example.com 'sshd -T | grep -i maxstartups'

sshd -T prints the effective configuration and needs privilege on most systems, so this may be a question for whoever owns the bastion rather than a command you run. Ask it anyway. “What is MaxStartups on the jump host” is a five-minute question that has saved entire afternoons.

Also worth asking: LoginGraceTime, because that is how long a connection counts as unauthenticated. A slow authentication path — a directory lookup, a hardware token, an overloaded bastion — holds each connection in the unauthenticated pool for longer and makes the limit bite at a lower fork count than the raw numbers suggest.

What happens if the bastion goes down mid-run?

Every remaining host becomes unreachable, in a burst, and the run reports something that looks like a catastrophic fleet failure.

The recap is what tells the two apart, and this is exactly the judgement call the last lesson of this part deals with. A fleet event and a path event have different shapes:

Path event. Every host behind that bastion fails; hosts reached by other routes are unaffected; the failures start at the same moment and the error is a connection failure to the jump host, not to the target.
Fleet event. Failures are spread across routing groups, and each error names the target.

The immediate operational question is not “how do I get around the bastion”. It is what state the fleet is in. A run that stopped halfway has left some hosts changed, some unchanged, and — if a task was in flight when the path dropped — possibly some in neither state.

So: do not reach for a second route and re-run. Establish what completed first. The recap names every host that succeeded, and Ansible is idempotent by design precisely so that a re-run over the completed hosts is safe — but only if your tasks actually are idempotent, which is a property of what you wrote, not a guarantee of the tool.

There is one preparatory step worth taking now rather than then: know whether a second path exists before you need it. An out-of-band route, a second bastion, a console server. Discovering during an incident that the answer is no is a bad time to find out that your automation and your recovery mechanism share a single point of failure.

Knowledge check

Why this matters

A bastion concentrates risk on purpose. That is the design: one machine to harden, one machine to monitor, one set of logs to keep. The security argument is good and this lesson is not arguing with it.

What the design also concentrates is availability, and that half rarely gets the same attention. The bastion was sized for a handful of engineers typing, and then automation arrived and started opening fifty connections at once, several times an hour, at three in the morning. Nobody re-sized anything, because nothing failed — until the fleet grew, and then it started failing thirty hosts at a time, at random, in a way that looked like the network.

Knowing that signature is worth more than any of the settings in this lesson. A different subset failing every run, with every host passing an individual test, is not a flaky network. It is a queue you have not measured.

The four argument settings, and which tools they reach

Proving that it appends

ProxyJump versus ProxyCommand

The bastion is a single point of failure for the whole run

Knowledge check

`ProxyJump` versus `ProxyCommand`