Ansible · Self-assessment
Knowledge checks
Every knowledge check in this course, in curriculum order. Each link opens the page at its quiz. The questions are auto-graded in the browser and nothing is recorded — a wrong answer costs you only the explanation, which is the part worth reading.
- Knowledge checks
- 374
- Parts covered
- 52
- Of all lessons
- 100%
Part I
Why Configuration Management Exists
8 checks
- Welcome to Ansible for Production SysadminsWhat this course assumes, what it targets, and why it is organised around blast radius rather than syntax.→
- Drift, and how a snowflake is actually madeThe mechanism by which identical hosts stop being identical: the emergency fix at 02:00, the package installed to test something, the sysctl a vendor engineer set. Why drift is invisible until an incident, and why nobody can currently prove those ten servers match.→
- Manual administration, honestlyThe case for doing it by hand: one host, one time, under a diagnosis you do not yet understand. Where the cost curve actually turns, and how to state the threshold without mocking manual work.→
- Shell scripts: the first automation, and where it breaksScripts as a legitimate stage rather than a failure. The four specific things an ssh loop over a host list cannot give you, and exactly which line fails when host 40 of 200 is down.→
- The configuration management modelWhat configuration management is independently of any tool: declared desired state, a convergence run, and a report of what differed. What the model costs to maintain, and the failure mode where the code becomes the snowflake.→
- Immutable infrastructure, and why it does not delete this courseRebuild instead of converge: what it genuinely solves, what it does not, and why the image still has to be built by something. Stateful hosts, appliances and network gear are not rebuildable on demand, and the hybrid estate is the normal case.→
- What automation costsThe part of the pitch nobody puts in the pitch: the review burden, the test environment, the CI pipeline, the person who understands the repository, and the automation everyone is now too frightened to run.→
- Choosing an approach for a given estateA decision framework across manual administration, scripting, configuration management and replacement, using the dimensions that actually decide it. Three contrasting estates worked to three different correct answers, one of which is not Ansible.→
Part II
Ansible Architecture
7 checks
- What agentless actually meansAgentless is a deployment property, not a magic one. What you gain - nothing to install, patch, or run as root permanently - and what you lose: no continuous enforcement, no state between runs, and nothing happens unless something runs the controller.→
- Control node and managed node responsibilitiesPrecisely which machine needs what: ansible-core, Python 3.12-3.14, the repository and the keys on the controller; a reachable SSH service, a Python in the 3.9-3.14 range, a writable temp directory and a way to become root on the managed node.→
- Anatomy of a run, start to finishThe ordered sequence of a single ansible-playbook invocation: locate config, parse inventory, resolve variables, open connections, gather facts, and for each task generate the payload, ship it, execute, read back JSON, evaluate the result, and aggregate into the recap.→
- How a module gets to the target: AnsiballZ and pipeliningThe concrete transport mechanism: a base64-encoded zipfile of the module plus its module_utils, wrapped in a Python script, extracted into a remote temporary directory and imported as __main__. What ANSIBLE_KEEP_REMOTE_FILES leaves behind, and what pipelining removes.→
- Connection plugins: SSH is a default, not a lawThe transport is pluggable. What ansible-core actually ships, what arrives with collections, and why a Molecule container scenario and a production VM run use different connection plugins and therefore prove different things.→
- The push model and who can reach whomPush means the controller opens the connection, which makes network topology an automation design constraint: NAT, firewalls, DMZ segments, air-gapped estates and laptops that are not always on. Contrasted with pull, and the security trade in both directions.→
- What Ansible does not do for youThe negative space, stated once so no later lesson has to apologise: no daemon, no continuous enforcement, no state database, no dependency graph across hosts, no automatic rollback, and no guarantee about a host it could not reach.→
Part III
Installing and Designing the Controller
7 checks
- ansible-core versus the ansible community packageTwo different artefacts with two different release cadences, and why this course installs one of them and pins the rest.→
- The support matrix, and which Python is whichController Python and managed-node Python are separate requirements. Reading the matrix answers "can this controller manage that host" without trying it.→
- Installing: pipx, virtualenv, pip and the distro packageFour install paths compared on the axes that decide operational outcomes: which Python they bind to, whether they upgrade independently, whether versions coexist, and whether the result is reproducible.→
- A controller you can rebuild identicallyPinning as a discipline: requirements.txt for ansible-core, requirements.yml for collections, and the two-machine test that proves the pin worked.→
- Controller filesystem layout and ownershipWhere the repository, config, keys, vault password, logs and temporary files live on a controller, who owns each, and why a group-writable directory is an escalation path.→
- Whose machine runs production change?Laptop, jump host, CI runner or platform: the governance question the install method quietly decides, and why the machine allowed to change production should not be the one with a browser on it.→
- Upgrading Ansible without surprising the fleetTreating the controller version as a change with fleet-wide blast radius: read the porting guide, install alongside, prove against both, then switch.→
Part IV
Inventory Fundamentals
7 checks
- INI and YAML inventories, side by sideThe same fleet in both formats, proved equivalent with ansible-inventory, and why an inventory outside the repository is an inventory outside review.→
- Groups, children, and the two groups you never wroteall and ungrouped exist whether you declare them or not, membership inherits upward through children, and hosts: production can include machines never listed under production.→
- group_vars and host_vars on diskThe file layout, the lexicographic read order, and the detail that causes real incidents: these directories are searched next to the inventory and next to the playbook, and one silently wins.→
- Ranges, aliases and connection variablesHost range expressions with strides and letters, inventory aliases via ansible_host, and why the behavioural connection variables belong in inventory rather than in a playbook.→
- Proving what your inventory containsThe read-only inventory toolkit as a discipline: --graph, --graph --vars, --list, --host and --list-hosts. You do not know what hosts: all means until you have printed it.→
- More than one inventory sourcePassing -i twice, pointing -i at a directory, how sources merge and in what order, and the file somebody dropped in the inventory directory.→
- When two groups disagreeThe resolution rule stated exactly - child beats parent, host beats group, same-level groups merge alphabetically unless ansible_group_priority says otherwise - and why relying on it is a coincidence rather than a design.→
Part V
Inventory Design at Fleet Scale
7 checks
- The four dimensions of a fleetEnvironment, site, role and lifecycle as orthogonal group families, and why compound group names destroy your ability to target a fleet precisely.→
- Environment separation you can rely onSeparate inventory files per environment make a production host unnameable from a staging run, and that is a stronger control than any pattern discipline.→
- Role groups and who owns themGrouping by service function, attaching an owning team to every group, and naming conventions that are still correct five years later.→
- Lifecycle stages and hosts you must not touchThe forgotten fourth dimension, and exception groups as a reviewable, expiring record of every host that is deliberately different.→
- Contradictory membership is an outage waitingA host in two mutually exclusive groups resolves silently by alphabetical merge order, which means a rename can change production behaviour.→
- Reading an inventory as a blast-radius mapThe reviewer skill: given an unfamiliar Ansible repository, answer what hosts: all reaches, which group is largest, and which group is targeted most - in under five minutes.→
- Where the truth about the fleet livesChoosing whether Git, the CMDB or the infrastructure API is authoritative for your inventory, and reconciling the two failure states that follow either way.→
Part VI
Configuration and Precedence
7 checks
- Which ansible.cfg is actually in effectFour locations, first match wins, no merging - and the working directory decides. Proven by execution on ansible-core 2.21.3.→
- The config file Ansible refuses to readAnsible ignores ./ansible.cfg in a world-writable directory because that file can execute code on your controller. What the check does, what it misses, and both fixes.→
- Proving a setting's value and its sourceansible-config list, view, dump, validate and init - and why --only-changed is the one that turns 219 settings into the handful your estate actually altered.→
- Forks, timeouts and what they costforks defaults to 5 and timeout to 10. Both are wrong for a large fleet, and the number of hosts changed simultaneously is a safety parameter rather than a performance one.→
- Interpreter discovery, configuredWhat INTERPRETER_PYTHON auto really does on 2.21.3, the exact candidate list it searches, and why silencing the warning globally converts it into a future incident.→
- The settings you will be tempted to weakenhost_key_checking, the StrictHostKeyChecking back door that no audit sees, deprecation warnings and retry files - each with the incident it enables and a compensating control.→
- The project config is part of the automationCommit ansible.cfg beside the playbooks so every operator and the CI runner behave identically - and treat a personal ~/.ansible.cfg as a divergence waiting to happen.→
Part VII
Ad-Hoc Execution
6 checks
- Anatomy of an ad-hoc commandDecomposing ansible <pattern> -m <module> -a "<args>", including the default module nobody chooses on purpose, and the flags that change how many hosts move at once.→
- When ad-hoc is right, and when it is notFour questions that decide whether a one-line command is the correct instrument, and a decision rule short enough to apply at 03:00.→
- Ad-hoc as a fleet inspection instrumentAnswering questions about an estate in seconds with read-only modules, and capturing the answers as evidence you can hand to someone else.→
- Blast radius of a one-line commandWhy a one-line ad-hoc command is more dangerous than a playbook, and the three-step habit that keeps a pattern typo from reaching the whole estate.→
- What an ad-hoc change leaves behindNothing, by default, except shell history on one machine — and what that costs when someone asks who changed 300 servers on Tuesday.→
- Promoting an ad-hoc command to a playbookThe same intent expressed twice — once as three shell one-liners, once as a reviewable play — and an exact account of what the conversion bought.→
Part VIII
Modules and the Module Model
7 checks
- The module contractA module is a program that receives arguments as data, runs on the target, and returns JSON — which means everything Ansible reports about a task is a claim made by the module.→
- Reading module documentation like an operatoransible-doc beyond the options list: the plugin types, the JSON output, and the attributes table that tells you whether --check will report anything at all.→
- Fully qualified names and how a module is foundWhat ansible.builtin.copy means that copy does not, demonstrated by shadowing a core module with a local file and watching only one of the two names change behaviour.→
- Not everything runs on the targetSome things that look like modules do controller-side work first — which explains why template renders locally, why a lookup reads your workstation, and why copy needs a file you have.→
- Choosing a module, and what you lose without oneA search procedure for finding the right module, and the five capabilities a shell line gives up — stated as losses rather than as style advice.→
- Check-mode support is a per-module propertyA play with green check output can contain tasks that were never evaluated at all — because --check is only as informative as the modules in it.→
- How modules fail, and what the error meansFive recurring module failures, told apart from the first three lines of the error, and sorted into controller problems and managed-node problems.→
Part IX
command, shell and raw
7 checks
- command: no shell, and why that is the safe oneWhat ansible.builtin.command actually does with your string, why shell metacharacters are inert, and the two options that give it any check-mode behaviour at all.→
- shell: when you genuinely need a shell, and what you acceptThe narrow cases that justify ansible.builtin.shell, the quoting and portability burden it transfers to you, and the rule this course adopts.→
- A variable inside a shell line is an injection pointWhat happens when inventory data, a fact or an extra-var reaches ansible.builtin.shell unquoted, and the three mitigations in the order you should apply them.→
- raw: the escape hatch, and its exact priceWhat ansible.builtin.raw skips in the execution pipeline, why it needs no Python on the target, and the three things it gives up in exchange.→
- Making an unavoidable shell task idempotentThe three honest techniques for making a command or shell task report changed truthfully, in the order you should try them, and the one shortcut that is not a fix.→
- script: transferring a script and running itHow ansible.builtin.script differs from copy plus command, the guards it does support, and the auditability you trade away when logic moves into a file.→
- A worked replacement: one shell-heavy play, rewrittenA realistic install-configure-enable-restart-verify play written entirely in shell, rewritten task by task with purpose-built modules, with the concrete gain shown at each step.→
Part X
YAML for Reliable Automation
6 checks
- The YAML a playbook actually isMappings, sequences and scalars, and the exact shape a playbook file has - so that an indentation error reads as a structural statement rather than a typo.→
- Types you did not ask forThe parser decides the type of every unquoted value, and its decisions include booleans from yes and no, floats from version numbers, octal from leading zeros, and base-60 integers from anything with a colon.→
- Quoting, and the double-brace ruleWhy a value beginning with a template must be quoted, which quote style to choose, what the parser sees that Jinja does not, and the when: exception.→
- Multiline values that surviveLiteral and folded block scalars, the chomping indicators, and the same content shown four ways with what actually arrives on the target.→
- File modes: quote themWhat unquoted 644 and unquoted 0644 actually become, why one of them accidentally works, and the exact permission bits each form produces.→
- The traps that do not raise an errorDuplicate keys where the last one wins, colons inside unquoted strings, anchors and aliases, and why yamllint and --syntax-check catch different classes of defect.→
Part XI
Playbooks
7 checks
- Anatomy of a playThe play as the unit that matters: hosts, become, gather_facts, vars, tasks and handlers, why one file can hold several plays, and why every task needs a name.→
- Execution order inside a playpre_tasks, roles, tasks, post_tasks, where handlers actually flush, and why the linear strategy makes a partial failure interpretable.→
- The hosts: line is the blast radiusWhy the play target expression is the most consequential line in a playbook, how to read it in review, and how to prove what it resolves to before you run.→
- The pre-flight options, and what each one provesThe ansible-playbook pre-flight ladder — syntax-check, list-hosts, list-tasks, list-tags, check, diff, start-at-task and step — with the exact claim and exact limit of each.→
- Composing a site.ymlHow an estate assembles from several playbook files into one entry point with import_playbook, what site.yml should and should not contain, and running a subset without editing it.→
- A playbook someone else can operateThe properties that make a playbook survivable: one purpose per play, named tasks, no business logic in the file, variables from inventory, and the point at which a task list should become a role.→
- Reading the recap, and the exit codeEvery field of the PLAY RECAP, what each one tells you about a host, and the process exit code as its machine-readable form — including why unreachable takes precedence over failed.→
Part XII
Idempotency and Change Reporting
8 checks
- Describing state instead of stepsThe imperative/declarative distinction done properly: why a declarative task re-evaluates reality on every run, and why a shell line in a playbook is imperative code wearing declarative clothing.→
- What idempotent means, preciselyNot "safe to run twice" in the loose sense: convergence to a declared state, and a second run against an unchanged host reporting zero changed tasks.→
- How Ansible decides ok versus changedThe mechanism a module uses to set changed, and the full result vocabulary — ok, changed, failed, skipped, unreachable, rescued and ignored — with a precise definition of each.→
- changed_when: telling the truth about a commandSupplying the knowledge a command module cannot obtain — from rc, from stdout, from a registered probe — and why the goal is an accurate report rather than a quiet one.→
- failed_when: when non-zero is not a failureExpressing what failure actually means for a command — including the inverse and more dangerous case, a command that exits 0 while having failed — without discarding the distinction the way ignore_errors does.→
- Why a wrong changed breaks the machineryThe concrete damage: a task that always reports changed restarts a service nightly for no reason, and a task that never reports changed leaves a deployed config unloaded on a green run.→
- changed=0 as the cheapest signal an estate producesOnce change reporting is accurate, a scheduled no-op run becomes a drift detector that costs almost nothing — and every inaccurate changed raises the noise floor until nobody reads the recap.→
- The operations that cannot be idempotentReboots, one-shot migrations, write-only API calls and vendor installers — how to fence them so the play stays rerunnable, and how to record that the exception was a decision.→
Part XIII
Variables and Precedence
8 checks
- Every place a variable can come fromThe catalogue of Ansible variable sources organised by who owns them - inventory, role, play and runtime - so you can read a repository and say who is entitled to set what.→
- The precedence order, verified by experimentThe full twenty-two level Ansible variable precedence list, established by successive elimination against ansible-core 2.21.3 rather than transcribed, plus the three principles that make it memorable.→
- Finding out what a variable actually isThe diagnostic sequence for resolving a variable on a real host - grep, ansible-inventory, ad-hoc debug, hostvars and -vvv - and the working-directory trap that makes two of those tools disagree with the playbook.→
- A layering rule you can hold in your headThree layers with three owners - role defaults as contract, group_vars as policy, host_vars as exception - and the group merge rules that decide whether your layout is predictable by reading or only by running.→
- defaults/ and vars/ are a policy decisionThe two variable files a role ships sit at opposite ends of the precedence ladder. Choosing between them declares whether callers may override a value, and that is an interface decision rather than a filing one.→
- -e always wins, and that is the problemExtra vars sit above every other source, which makes them the ideal emergency override and a corrosive habit. What an estate looks like after two years of -e, and how to constrain the legitimate uses.→
- Names that do not collideVariables share one flat namespace per host, so an unprefixed name set by one role is visible to every other. Role prefixes, the reserved ansible_ space, magic variables, and why a collision presents as a wrong value rather than an error.→
- A precedence incident, worked end to endA correct playbook deploys the wrong database host to production. Worked from the symptom through the diagnostic sequence to the structural fix, in the order a competent operator would actually find it.→
Part XIV
Facts and Registered Variables
7 checks
- What fact gathering actually costsThe implicit setup task runs against every host before your first real task, and on a large inventory it is usually the largest single item in the run - measured here, not asserted.→
- Reading ansible_facts properlyThe ansible_facts dictionary is the real structure; the top-level ansible_ names are an injected convenience whose default is deprecated and scheduled for removal in ansible-core 2.24.→
- Which facts you can trustFacts are evidence with varying reliability - some are stable identity, some change under virtualisation or resize, and some are simply absent on a minimal host.→
- Custom facts with facts.dA host can declare things about itself that no fact module knows - and the best use of that is letting a host record a documented exception your automation must respect.→
- Fact caching and stale truthCaching facts removes gathering from most runs, and introduces the one hazard unique to caching: a decision made confidently on a fact that stopped being true two days ago.→
- register and the shape of a resultA registered result is whatever the module returned, and its shape changes completely when the task loops - read the structure rather than guessing at key names.→
- set_fact, register or inventory?Three places a runtime value can live, three precedence positions, and the design question underneath: is this genuinely computed, or is it a decision hiding from review?→
Part XV
Conditionals and Loops
7 checks
- when expressions that hold upwhen is raw Jinja without the braces, and on ansible-core 2.21 a conditional that does not evaluate to a boolean is a hard failure rather than a silently-true condition.→
- Conditioning on a previous taskThe task result tests - is failed, is succeeded, is skipped, is changed - and why reading result.rc blows up when the task that registered it was skipped.→
- loop, and when not to use oneModern loop syntax over lists and dictionaries - and the production judgement that comes first, because a loop calling a module once per item is slower and less atomic than one call with a list.→
- loop_control and readable runslabel, loop_var, index_var, pause and extended - and the one that is a security control rather than a cosmetic one, because a loop over credential-bearing dictionaries prints them by default.→
- Retry loops for genuinely transient faultsuntil, retries and delay are for a fault that resolves itself - and the diagnostic damage of using them on one that does not is worse than the failure they suppress.→
- Reading inherited with_* codeThe only lesson in this course where with_* appears: what each form does in code you inherit, what it maps to under loop, and the one conversion that silently changes behaviour.→
- Conditionals that should have been groupsA task needing four conditions to decide whether it applies is an inventory design problem wearing task logic - and moving it makes the blast radius of a run visible before you run it.→
Part XVI
Handlers
6 checks
- Handlers as a safety mechanismA handler restarts a service because something actually changed. That conditional is both a safety property and an availability property, and it fails in two directions.→
- Handler ordering and flush_handlersHandlers run at the end of each section, and they run in the order they were defined rather than the order they were notified. Both facts surprise people, and both change what a play does.→
- listen topics as a handler interfaceA listen topic decouples what a task asks for from which handlers provide it, which makes it a stabler contract than a handler name across renames and role boundaries.→
- The handler that never ranA host that fails between the notify and the flush keeps its old configuration in memory and its new configuration on disk. force_handlers, rescue-time flushing, and finding those hosts afterwards.→
- Naming handlers, and role collisionsHandler names are resolved before any host variable exists, and handlers from every role land in one flat namespace on the play. Both facts produce failures that look like something else.→
- Restart, reload, and doing it to everyone at onceChoosing reload where the service supports it, and confronting the fact that a correct handler on a play targeting 400 hosts is a 400-host simultaneous service event.→
Part XVII
Templates and Jinja2
8 checks
- Jinja2 for sysadminsExpressions, statements and whitespace control - and the single most load-bearing fact about templating: it happens on the controller, not on the managed node.→
- The template module in productionThe parameters that carry operational weight - mode, owner, backup, force, newline_sequence - and the module attributes that tell you what --check and --diff will actually prove.→
- The filters you will actually usedefault, mandatory, ternary, combine, to_nice_yaml, from_json and regex_replace - chosen for readability, with the explicit position that a clever one-line filter chain is a maintenance cost.→
- Undefined variables and silent wrong outputAn undefined value propagates through attribute access without complaint, so the failure surfaces where the value is used rather than where it went missing - and a default() can hide it completely.→
- Validate before it becomes liveThe validate parameter runs the service own checker against a temporary file, and moves the file into place only if it exits zero. What %s means, what happens on failure, and where the guarantee ends.→
- Where validate is not enoughA validator that needs the whole include tree cannot judge a fragment in a temporary file. Building render, check and activate by hand when the parameter cannot do it - and keeping that sequence idempotent.→
- Template a drop-in, not the whole fileWhen Ansible cannot own an entire configuration file, templating a fragment under conf.d gives a smaller diff, a smaller blast radius, and a clean answer to what did Ansible change here.→
- Templates that survive reviewA generated file should say it is generated and where it came from. A 300-line template with nested conditionals should be several templates, or a different design.→
Part XVIII
Files and Configuration Management
7 checks
- Which file module, and whyA decision framework across copy, template, file, lineinfile, blockinfile and assemble, built on the question of whether the automation declares what the file should contain.→
- file and the meaning of stateThe six values of state on the file module are six different operations, not variations on one. What each does, what recurse actually covers, and the trap of state: file.→
- lineinfile without the idempotency trapregexp versus search_string, backrefs, firstmatch and insertafter, and the two failure modes that actually occur: a file that grows every run, and a file with two contradictory settings that reports ok forever.→
- Owning a region of a file you do not ownblockinfile markers as a contract between your automation and everything else that writes the file, the marker rules that cause repeated insertion, and what happens when a human edits inside the region.→
- Permissions and context as desired stateowner, group, mode and SELinux context belong in the task that writes the file, not in a later remediation pass. Why an omitted mode is a security decision, and the config file that deploys perfectly and cannot be read.→
- backup: and atomic replacementWhere backup: true puts the timestamped copy, why that is a rollback aid and not a rollback strategy, what safe_file_operations guarantees, and what unsafe_writes gives away.→
- Knowing what is there before you change itstat with checksums, slurp and fetch: answering whether a host file is what you believe it is, and collecting evidence from a divergent host without editing it.→
Part XIX
Privilege Escalation
6 checks
- What become actually doesbecome wraps command execution on the target after login; it is not part of the connection. The keywords, the variables, the defaults, and the command Ansible actually runs.→
- Escalation methods and where they applyOnly three become plugins ship with ansible-core. What the rest are, which collection provides them, which matter on a Linux fleet, and how to set the method per group rather than globally.→
- Escalation passwords without leaking themWhere a become password can be supplied, which of those routes writes it somewhere durable, and why -e is the worst of them.→
- Least privilege for the automation accountDesigning the grant the fleet gives your controller. Why blanket NOPASSWD: ALL makes controller compromise identical to fleet root, and which levers remain when Ansible cannot scope by command.→
- Unprivileged-to-unprivileged escalationWhen neither the connection user nor the become user is privileged, Ansible has a temporary-file problem. The documented fallback chain ends in a world-readable directory, and the fix is pipelining or not making the transition.→
- Reading a become failureThe five error shapes escalation actually produces, what each one proves about the managed node, and how to diagnose them without pasting a password into a ticket.→
Part XX
SSH Architecture and Connectivity
7 checks
- What happens on the wire during a runThe ssh connection plugin is the actual execution path. Reading the command line Ansible builds, the defaults baked into it, and why most Ansible problems are SSH problems wearing an Ansible error message.→
- Keys and agents for unattended runsansible_ssh_private_key_file per group, separate keys per environment, and the passphrase problem an unattended run creates - including the ssh_agent, private_key and private_key_passphrase settings that give it a real answer on 2.21.→
- Host key verification is a security controlHOST_KEY_CHECKING = True adds nothing to the command line - it only declines to disable. Managing known_hosts deliberately with ansible.builtin.known_hosts so verification can stay on, and scoping the exception to the group that needs it.→
- Reaching a fleet through a bastionansible_ssh_common_args appends to the defaults rather than replacing them, which is what makes ProxyJump safe to add per group. Plus the bastion as a single point of failure for the entire run, and the MaxStartups drop that looks like a flaky network.→
- Connection reuse and why runs get slowControlMaster and ControlPersist are in the default ssh_args for a reason: without them every task on every host pays a full SSH handshake. The control path length limit, reproduced, and what a stale socket looks like.→
- Pipelining and its preconditionsThe requiretty conflict is really a conflict about the -tt flag, and the connection plugin decides that flag with one line you can read. How to establish the precondition across a fleet before you enable anything, and roll it out per group.→
- Triaging unreachable hostsReading the ssh error text as a taxonomy - refused, timed out, no route, permission denied, host key mismatch - and the operational judgement when 40 of 400 hosts come back unreachable in the middle of a run.→
Part XXI
Secrets Management
8 checks
- The ways secrets actually leakThe full enumeration of exit routes before any tooling: Git history that a deletion does not clear, a debug of a registered result, --diff, log_path, verbose output, shell history, backup files, temp files on the target and CI artifacts.→
- Vault file operationscreate, encrypt, view, edit, rekey and decrypt - what the header line tells you, what the file mode becomes, and why decrypt is the one subcommand you almost never want.→
- Encrypting one variable, not the whole fileencrypt_string and the inline !vault form keep a vars file reviewable with only the secret values opaque - at the documented cost of failing at the point of use rather than at load, which is measured here.→
- Vault IDs across environmentsLabels are documentation, not enforcement. Measured on 2.21.3: with vault_id_match off - the default - a password offered under the wrong label still opens the file, and the dev password can open production.→
- Where the vault password livesA password file beside the repository, readable by anyone who can read the repository, has bought you nothing. Executable password sources, the -client script contract measured on 2.21.3, and what actually improves the situation.→
- What Vault does not protectVault protects data at rest in your repository. Measured on 2.21.3: a vault-encrypted variable prints in full from a debug task, templates into any string, and travels to the managed node as an ordinary module argument.→
- no_log, and what it costs youMeasured on 2.21.3: no_log censors the task result and does not censor a failure message the task itself built, so a secret interpolated into fail_msg is printed in full. Plus the documented ANSIBLE_DEBUG exception, reproduced.→
- When the secret should not be in the repository at allLookups execute on the controller, which is what makes external secret stores work - demonstrated against an unreachable host. Short-lived dynamic credentials versus a long-lived encrypted file, and the dependency nobody mentions.→
Part XXII
Roles and Reuse
8 checks
- Role anatomy and where roles are foundThe directories a role is made of, what Ansible loads from each one automatically, and the search path that decides which role a bare name resolves to.→
- What belongs in one roleThe boundary test for a role is operational rather than aesthetic: can it run against a host that has nothing else of yours on it? A role that assumes a sibling ran first has a blast radius nobody can state.→
- defaults/ is the interface, vars/ is notDesigning a role as something other people consume: which directory each value belongs in, what the README owes a caller, and why a tunable filed in vars/ makes the role unusable by anyone but its author.→
- Roles that refuse bad inputmeta/argument_specs.yml turns a role interface into something the run enforces — entry points, types, required options and choices — plus the two behaviours that surprise everyone: the spec default does not define the variable, and validation is coercive on acceptance only.→
- Static and dynamic role reuseimport_role and include_role differ at parse time versus run time, and the consequences land on --list-tasks, --tags, when and loop. Verified by execution, because this is the behaviour people describe confidently and wrongly.→
- Dependencies and the roles you did not ask formeta/main.yml dependencies run before the role that lists them, deduplicate in ways --list-tasks does not show, and turn a two-line play into an execution order nobody can read off the file.→
- Handlers across role boundariesRole handlers are merged into the parent play, so two roles with a handler named restart nginx share one namespace and the wrong one fires. Verified by execution, along with the two fixes.→
- Refactoring a monolith into rolesThe migration nobody plans for: what to extract first, how to keep each step reviewable, how to prove behaviour is unchanged, and how not to silently break the --limit and --tags behaviour operators already depend on.→
Part XXIII
Tags, Blocks and Error Handling
8 checks
- Tags as an operational controlTags on tasks, blocks, plays and roles; --tags and --skip-tags with the rule that skip always wins; and --list-tags as the thing you run before relying on a selection that fails silently when it matches nothing.→
- always, never, tagged and untaggedThe four reserved tags and their exact behaviour, executed rather than recalled: what always survives, how never keeps a destructive task out of every ordinary run, and what tagged and untagged select.→
- The tag that selected nothingStatic imports apply tag inheritance to every task inside; dynamic includes tag only the include statement. The failure that produces: --tags patch runs, reports success, and touches nothing.→
- Blocks and shared directivesA block applies a directive to a group of tasks at once — when, become, tags, ignore_errors — and is the unit of transactional intent. It also cannot be looped, and the reason that limitation exists is worth understanding.→
- rescue and alwaysA rescue reverts the failed status and lets the play continue, while the recap still records what happened. The right thing to put in one is a compensating action that returns the host to a state you can name.→
- Overriding the module verdictfailed_when and changed_when make a task tell the operational truth instead of the module default. The gotcha that costs people an outage: a list of conditions is joined with an implicit and.→
- Why ignore_errors: true hides outagesignore_errors is more dangerous and less useful than people assume: it does not cover unreachable hosts, and ignore_unreachable produces a recap that reports zero unreachable for a host nothing ever contacted.→
- When a retry makes it worseuntil/retries/delay is right for a genuinely transient condition and wrong for everything else: a retry around a non-idempotent action multiplies the damage, and a retry around a deterministic failure just delays the alert.→
Part XXIV
Assertions and Guardrails
6 checks
- Check the world before you change itThe precondition pattern: automation that states its assumptions and verifies them before acting, and why a guard that fires costs less than a rollback that works.→
- Writing assertions operators can act onansible.builtin.assert in full - that, fail_msg, success_msg and quiet - and the craft of a failure message that tells the operator what to do next.→
- Automation that refusesansible.builtin.fail with when as a deliberate stop, the mandatory filter for missing inputs, and the difference between failing because something broke and failing because you declined.→
- Guarding against the wrong fleetThe I-ran-the-staging-playbook-against-production incident class, and the guards that make it an error message: environment agreement, group membership and a blast-radius ceiling.→
- Preconditions on capacity and healthAsserting on disk, service health, quorum and backup freshness - and the follow-up question that decides whether the assertion means anything: where did that fact come from, and how old is it?→
- Guardrails operators will not bypassThe failure mode of over-guarding: how a tagged guard is skipped, why a guard that fires on legitimate work protects nothing, and what a documented override looks like.→
Part XXV
Check Mode, Diff and Static Validation
7 checks
- The cheapest gates: syntax and listingWhat --syntax-check, --list-hosts, --list-tasks, --list-tags, ansible-inventory and ansible-config genuinely prove on ansible-core 2.21.3 - established by running them against playbooks that are wrong in specific ways.→
- ansible-lint on an existing repositoryThe profile ladder from min to production, what each profile is for, and a realistic adoption path for an inherited repository that fails thousands of rules on day one.→
- How check mode worksThe three controls that decide whether a task simulates - the --check flag, the check_mode keyword on play, block or task, and the ansible_check_mode variable - and the one place they disagree.→
- What a clean --check does not proveFour classes of defect that pass a green dry run, demonstrated - including a check run that printed a perfect diff for a change the real run could not make.→
- Designing a play whose dry run is informativeThe constructive half: creates guards, check_mode false on probes, restructuring sequential plays, and deciding honestly when a play needs a canary instead of a dry run.→
- --diff and what it printsThe best configuration review surface Ansible has, and a disclosure risk: a diff of a rendered template prints the credential in it, into every log that captured the run.→
- The sequence before a production runAssembling the rungs into a procedure - syntax, lint, host list, check and diff on one canary, then a real run under --limit - with what to read at each rung and the condition that stops you.→
Part XXVI
Testing Automation
8 checks
- The layered testing model: what each rung actually provesSyntax check, lint, static validation, disposable integration, staging, canary, production — what each layer costs, what defect class it catches, and the classes it structurally cannot reach.→
- The second run is a testRun twice, assert zero changes on the second. What a persistently changed task actually indicates, how to automate the assertion robustly, and how to record a legitimate exception instead of suppressing it.→
- Where container fidelity stopsContainers are adequate for package, file, config and role logic. They are not adequate where behaviour depends on systemd as PID 1, the kernel, reboots, network reconfiguration or storage — and that gap is where production incidents live.→
- Molecule: a first scenarioWhat molecule init scenario actually generates on Molecule 26.x, what the create / converge / verify / destroy lifecycle does, and one small scenario you can run end to end.→
- One role, three distributionsRunning the same role against Ubuntu 24.04, Debian 12 and Rocky 9 — what a platform matrix costs in run time and maintenance, and the judgement call about where to stop.→
- Verify the outcome, not the task resultAnsible reporting ok means the module was satisfied, not that the service works. Writing verification that probes real state — the port listens, the config parses on the target, the endpoint answers.→
- Staging that is worth havingA staging environment that does not resemble production manufactures false confidence. The minimum fidelity worth paying for, and how to state honestly what your staging does not cover.→
- What you cannot test before productionLoad, real traffic, real hardware, real neighbours, real data volume, real operators — and the conclusion the part builds to: testing is not something you complete, so the canary is the final test rather than a formality.→
Part XXVII
Collections, Galaxy and Dependency Trust
8 checks
- Collections, namespaces and the FQCNWhat a collection actually is on disk, what namespace.collection.plugin buys you, and a demonstration of two collections shipping the same short name where the order of a list decides which one runs.→
- Proving which artefact you installedansible --version does not tell you what you can run. The four commands that establish a controller manifest, and why the manifest is what you compare when a playbook works on one machine and not another.→
- Where collections live and which copy winsFour places a collection can be on disk, the order they are searched, and a demonstration of a collection that a playbook loads happily while ansible-doc insists it does not exist.→
- requirements.yml and version pinningWriting the file, the specifier syntax that is actually accepted, and the demonstration that matters: a pinned collection whose own dependency is unpinned, so the pin does not hold.→
- Reading a collection before you run itA repeatable intake review for third-party automation: publisher, maintenance, source, transitive dependencies, release history, privilege required, and what the modules do with the credentials you hand them.→
- Checksums, GPG signatures and what they proveansible-galaxy collection verify demonstrated on a tampered install, the --offline distinction, GPG signature options, and the sentence people get backwards: this is the artefact the author published is not this artefact is safe.→
- Private Galaxy, mirrors and air-gapped installBuilding an offline bundle with ansible-galaxy collection download, the generated requirements file that only works from its own directory, and configuring GALAXY_SERVER_LIST so a production controller never reaches the public internet at run time.→
- Shipping your own internal collectionWhen a pile of roles should become a versioned artefact: galaxy.yml, build, install, and semantic versioning applied to automation whose consumers are your own colleagues.→
Part XXVIII
Plugins, Lookups and Filters
8 checks
- The plugin types and how they loadFifteen plugin types, what each one hooks into, and the count of each on a bare ansible-core controller - so that when you want to change a behaviour you know which kind of plugin governs it.→
- Lookups execute on the controllerNot asserted - demonstrated. A play against two hosts that cannot be reached at all, where the lookup task succeeds and returns data and the next task fails UNREACHABLE.→
- lookup, query and loopslookup() joins its results into a comma-separated string and query() does not - demonstrated with type_debug, and with the loop error that catches it in 2.21 but did not always.→
- Lookups that touch secretsenv, file, unvault, password and pipe - including the two facts nobody expects: the password lookup writes a plaintext file to your controller, and pipe is a shell, so merge rights on a playbook are shell access on the controller.→
- Filters that make data readableThe collection-shaping filters - map, select, selectattr, dict2items, items2dict and recursive combine - each verified against ansible-doc, and a worked refactor of an expression nobody could review.→
- Jinja tests and conditionals that fail safedefined, truthy, match, search, subset and version - plus two verified 2.21 behaviours: a non-boolean when: is now a hard error, and a when: wrapped in braces is deprecated for removal in 2.23.→
- Callback plugins and run outputChanging what a run reports without editing a playbook: result_format yaml, the one-stdout-callback rule, JUnit XML for CI, and an honest note about which shipped callbacks are deprecated for removal in 2.23.→
- Writing a filter plugin (and when not to)One filter, complete: the Python, the DOCUMENTATION block that makes ansible-doc work, a test playbook that runs in CI, and the judgement call about what a filter must never do.→
Part XXIX
Dynamic Inventory
8 checks
- Inventory plugins, not inventory scriptsWhy plugins replaced executable inventory scripts, how the auto plugin dispatches on the plugin: key, and the file-naming rules that decide whether your config is read at all.→
- Building and proving a dynamic sourceStanding one dynamic inventory source up end to end and proving exactly what it returns with ansible-inventory --list before any playbook is pointed at it.→
- Turning provider metadata into targetable groupskeyed_groups, groups, compose, leading_separator and strict - and how to design group names you would be willing to type after a --limit at two in the morning.→
- Inventory caching and stalenesscache, cache_plugin, cache_timeout and cache_connection; --flush-cache; and why a cached inventory is a stale picture of the fleet whose staleness and freshness carry different risks.→
- Authenticating an inventory sourceWhere an inventory plugin gets its credentials, why they must be read-only, what an attacker gains from them, and why inventory identity and connection identity should not be the same account.→
- When new hosts appear on their ownThe blast-radius hazard unique to dynamic inventory: somebody else creates instances with the right tag and your unchanged playbook configures them. Diffing the host list, alerting on the delta, and opting in rather than out.→
- Merging static and dynamic inventoryMultiple -i sources, merge and precedence behaviour, how group_vars bind to dynamically created groups, and keeping a static never-automate list authoritative over anything the API says.→
- When the inventory source failsAPI outage, partial results, rate limiting and expired credentials - why a partial inventory is more dangerous than a missing one, and how to make "no hosts matched" a failure instead of a silence.→
Part XXX
Host Targeting and Blast Radius
8 checks
- Blast radius as a design constraintThe definition the whole course is organised around - the number and criticality of systems a mistake can affect - and the four questions asked before every run.→
- Host pattern syntaxEvery pattern form ansible-core 2.21 actually supports - all, globs, union, intersection, exclusion, regex, group indexing and slices - verified by execution rather than recalled from memory.→
- How Ansible evaluates a patternOperator position in the string does not matter. Unions are collected, intersections applied, exclusions removed - which is why an exclusion cannot be cancelled by a later term and is therefore a reliable safety mechanism.→
- --limit and limit filesHow --limit narrows a play and can never widen it, the @file form for a generated host list, and using a limit file as the reviewable artefact of a change ticket.→
- Verifying the effective host list before you runThe central skill of this part: determining exactly which hosts a command will change, before running it - and the named trap that ansible-inventory --graph ignores --limit.→
- Zero hosts: warning or errorA bare pattern matching nothing is a warning and exit 0; an unmatched --limit is a hard error and exit 1. Why the first silently turns a CI green into "nothing happened", and how to make zero hosts always fail.→
- ansible_play_hosts, ansible_play_batch and friendsThe magic variables that let a play inspect its own scope - which hosts are still active, which are in this batch, which were in the play at the start - and how to assert on them.→
- Guardrails: refusing to run too wideA reusable pre-flight block that aborts on an oversized target, an unacknowledged production run or a protected host - plus the order: keyword and why shuffle destroys canary reasoning.→
Part XXXI
Serial Execution and Failure Tolerance
8 checks
- serial: batching a playserial as an integer, a percentage and a list of increasing batch sizes. What the repeating PLAY banner means, why a percentage rounds down, and what the play restarting does to pre_tasks and set facts.→
- Handlers flush per batchUnder serial, handlers flush at the end of every batch rather than once at the end of the play. This is what makes a rolling restart possible, and it is why a handler that touches a shared dependency fires once per batch.→
- max_fail_percentage, preciselyEvaluated per batch, and the threshold must be exceeded rather than equalled. The 33-versus-34 boundary demonstrated by execution, and the surviving hosts an abort leaves half-finished.→
- any_errors_fatal and what fatal meansThe failing task completes on every host in the batch, then the play stops for everyone. Verified: no later batch runs, and a subsequent play in the same playbook does not run either.→
- Choosing a failure policyThree mechanisms, three different questions. Which of no policy, max_fail_percentage and any_errors_fatal fits which class of change, and why the canary ramp is the default answer.→
- Unreachable is not failedUnreachable hosts are accounted separately and ignored by both failure keywords. Why a network blip looks like a fleet-wide failure, how to tell them apart from the recap, and what ignore_unreachable actually changes.→
- Ending a batch or a play deliberatelymeta: end_batch, end_host, end_play, end_role and clear_host_errors. Stopping on a condition you detected yourself, and the difference between a deliberate stop and a silent one.→
- Stopping a rollout at 02:00What Ctrl-C leaves half-done, how to work out which batch was in flight, how to establish the fleet actual state afterwards, and how to decide between hold, forward and reverse.→
Part XXXII
Rolling Deployments
8 checks
- The shape of a rolling deploymentThe six-step loop as a named pattern: drain, verify drained, deploy, restart, health-check, return to service. Why every step needs a verification and not just an action.→
- Draining a host from the load balancerdelegate_to the load balancer, and the step most implementations skip: proving the host actually drained. wait_for state drained, exclude_hosts, and why TIME_WAIT makes the naive version hang.→
- Health checks that actually assert somethingThe difference between the port is open, the process is up and the service is correct. uri with status_code and until, wait_for with search_regex, and why a --check run cannot prove your health gate works.→
- Restart, reload and handler timingWhen a reload is legal and when only a restart will do, why handlers are the right mechanism inside a rolling play, and why the end-of-batch flush is too late to verify the batch you just changed.→
- Return to service and the soak intervalRe-enabling in the load balancer, verifying real traffic is arriving rather than that the API accepted the call, and the deliberate pause between batches that lets a slow failure surface.→
- When the rollout fails halfwayHalf the fleet runs the new version and half the old. Establishing the true split, the three options and the criteria for each, and the drained hosts that are the urgent part.→
- Version skew: what must be true to roll at allDuring a rolling deploy old and new run simultaneously. Database schema compatibility in both directions, API and message formats, session affinity and shared caches — and when a rolling deploy is the wrong pattern.→
- The complete rolling playbookThe whole pattern in one annotated file: guardrails, reachability pre-flight, serial and failure policy, drain, deploy, health gate and return-to-service, with a note on every block explaining which failure it prevents.→
Part XXXIII
Delegation and Controller-Side Execution
7 checks
- Which machine does this task run on?Every task has an execution context - the managed node, the controller, or a third host - and the confusing bugs in this part all start with an author who pictured the wrong one. How to prove which applies instead of assuming.→
- delegate_to and the variables that follow itThe rule people get wrong in both directions: the connection is made with the delegated host settings, while the task template still resolves against the original host. Both halves demonstrated, because assuming either one alone produces a bug.→
- delegate_facts: whose facts are these?Facts produced by a delegated task are stored against the original host unless you say otherwise. That default quietly writes one machine measurements into another machine record, and the run reports ok throughout.→
- Four ways to run something locallydelegate_to localhost, connection local, local_action and a separate hosts localhost play look interchangeable and are not. What each one changes about the host loop, the variables in scope, and where become escalates.→
- run_once means once per batchVerified on 2.21.3: run_once with serial 2 over six hosts executed three times, not once. It runs on the first host still available, its results propagate only within the batch, and a when: on it is evaluated on that one host alone.→
- Making "exactly once" actually mean onceThree constructs that survive serial: a condition on the play host list, throttle for serialising without singling anyone out, and a separate play for genuinely global work. Chosen by stated intent, not by habit.→
- Delegating to a host that is not in inventoryYou can delegate to a bare address, and upstream warns it might cause issues. Measured here: no hostvars entry, no group membership, no facts - but group_vars/all still reaches the connection, which is the part nobody expects.→
Part XXXIV
Concurrency, Strategies and Performance
7 checks
- Measure first, tune secondprofile_tasks is not in ansible-core, and most slow playbooks are slow for one identifiable reason that is not parallelism. How to get a baseline, what the numbers mean, and why a single wall-clock reading is not a measurement.→
- forks and the controller real limitsMore forks is faster right up until it is not. Measured on a 12-core controller: near-linear to forks 8, then flat, with run-to-run noise larger than any further gain. Where the ceiling comes from, and why exhaustion presents as target-side faults.→
- linear, free and host_pinnedExactly four strategy plugins ship in ansible.builtin, and the difference between them is which synchronisation guarantee they remove. free is faster and abandons the barrier that makes a staged change meaningful.→
- throttle: protecting shared dependenciesOne package mirror, one licence server, one API. throttle lowers concurrency for the tasks that touch them, applies at play, role, block, task and handler level, and - measured here - cannot raise concurrency above forks.→
- Pipelining, ControlPersist and connection costConnection overhead is paid per task per host, so it scales with the product. What the defaults actually are on 2.21.3, what enabling pipelining requires of every managed node sudoers file, and how to measure the difference honestly.→
- Fact gathering is usually the billThe largest saving available on a large fleet, and the only one that costs nothing operationally - provided you audited which facts your roles actually read. The audit, the per-subset costs measured, and gathering on demand.→
- Fact caching and the risk of stale factsThe last lever in the tuning sequence, and the only one whose failure mode is a wrong value rather than a slow run. Which plugins core actually ships, what --flush-cache clears, and how to decide what is safe to cache by how fast it changes.→
Part XXXV
Large Fleet Architecture
7 checks
- Waves: ordering a fleet by consequenceDividing thousands of hosts into waves by criticality and dependency rather than alphabetically — which wave proves the change, which wave is allowed to fail, and why a wave boundary is a decision point with a human at it.→
- Inventory structure that survives growthGroup hierarchy for thousands of hosts: role, environment, location and lifecycle as orthogonal dimensions, the group-per-fact trap that produces four hundred groups nobody can target, and the same-depth precedence rule your naming scheme silently decides.→
- Where the controller livesOne central controller or several regional ones: what latency does to a fleet run, why the controller holds the most privileged network path in the estate, and the uncomfortable fact that the controller is inside its own blast radius.→
- Scheduling and the thundering herdWindows, staggering and jitter: why three thousand hosts pulling from one package mirror at 02:00 is an outage you caused, and how serial, throttle and per-host jitter each shape a different part of the load.→
- Partitioning a run so it can be resumedA four-hour run has a different failure profile from a four-minute one. Splitting a fleet-wide run into independently restartable units, recording where it got to, and why the retry file and --start-at-task both let you down.→
- Inventory performance and truth at scaleInventory generation is a real cost at thousands of hosts and it runs before every command. Caching large sources, generating per-wave limit files, and keeping the inventory an accurate description of a fleet that changes daily.→
- Estimating runtime before you commit to a windowComputing expected wall-clock from host count, task count, per-task cost and effective concurrency; why the linear strategy makes it a sum of maxima rather than a max of sums; and what to cut when the change does not fit.→
Part XXXVI
Drift and Convergence
7 checks
- Drift, and what convergence actually meansDrift is the gap between declared and actual state. Its four sources, why an idempotent playbook does not mean a converged fleet, and the limit that governs everything else in this part: a clean run only proves it checked what it manages.→
- Check mode as a drift detectorRunning the production playbook with --check --diff on a schedule and reading the result as a drift report — including the awkward fact that a check run reporting changes still exits 0, and what core actually gives you to parse.→
- What check mode cannot tell youThe verified behaviour that breaks drift reports: a skipped command still registers rc=0, so the obvious gate opens in check mode; check_mode false makes a task execute for real during a dry run; and command support is partial rather than absent.→
- Reading diff output properlyWhat --diff shows for templates, files and other modules, where it shows nothing at all, what the before and after labels actually point at, and why no_log turns your best review artefact into a blank.→
- Detect, decide, enforceSeparating the audit from the enforcement without duplicating the code, the triage step between them that decides what drift even is, and the honest trade-off in running unattended convergence on a schedule.→
- Recording an approved deviationThe mechanism nobody builds until it hurts: an exception expressed as inventory data carrying a reason, an owner and an expiry, so the audit reports an expected deviation instead of fighting the same host every night.→
- Snowflakes and the uniqueness budgetWhen host_vars proliferate, automation stops being automation. Measuring uniqueness across the fleet as a health metric, and knowing when a genuinely unique host should be graduated out of the standard rather than special-cased inside it.→
Part XXXVII
Environments and Repository Architecture
7 checks
- Environments are inventories, not variablesAn environment expressed as a variable is a guess the play makes; an environment expressed as an inventory tree is a fact the loader enforces. Measured on 2.21.3, including the reserved name that makes the variable approach fail in the worst possible direction.→
- The reference repository layoutA production layout walked directory by directory, and an honest comparison of the alternatives it rules out: branches per environment, one repository or several, roles vendored or pulled from requirements.yml.→
- Layering group_vars across environmentsThe three layers this part controls - role default, group_vars/all, group_vars/<group>, host_vars - the lexicographic rule inside a group_vars directory, and the one command that shows which layer a value came from.→
- One role, many environmentsA role that reads the environment name makes production the least-tested code path in the estate. Parameterising instead, and the technique that lets staging exercise the production parameter set.→
- Promoting a change from development to productionWhat "promoted" means concretely - the same commit, the same pinned dependencies, a different inventory - and how to record which version of the automation ran against which environment.→
- Secrets separated by environmentSeparation comes from who holds which password file, not from the vault id labels. What a wrong vault id actually does to a production run, measured on 2.21.3, and the static gate that does not catch it.→
- Conventions, ownership and the 03:00 READMENaming that makes grep an investigation tool, CODEOWNERS scoped to the paths that cause outages, and a README written for the person who was paged and has never opened this repository.→
Part XXXVIII
Git Workflow and CI for Ansible
7 checks
- This repository can take down the fleetA merge here has a larger blast radius than most application deploys. What a reviewer of an Ansible diff should actually look for, and why attached check-mode output beats reading YAML carefully.→
- Branch protection and what production runs fromProtected main and short-lived branches are the easy half. The operational rule is that production runs from a tag or an explicit commit, never from whatever main happened to contain at 02:00.→
- Syntax check, yamllint and ansible-lint as CI stagesThree static gates ordered so each catches a class the previous one structurally cannot - measured on 2.21.3 and ansible-lint 26.6.0, including the profile that reports a passing playbook with eleven violations.→
- Keeping secrets out of the repositoryScanning at commit time and in CI, an Ansible-specific gate for files that should have been vaulted, and the response when one gets through: rotate the credential first, because rewriting history does not un-disclose it.→
- Check mode as a merge gateRunning --check --diff against a staging inventory in CI and attaching the output to the pull request, the exit codes a naive gate gets wrong, and the ways a clean dry run means nothing ran at all.→
- Role testing with Molecule in the pipelineWhich Molecule sequence a CI job should run and why, the idempotence step as the highest-value automated check a role can have, and choosing the test environment from what the role actually touches.→
- The complete pipelineEvery gate in order with the class of production failure it exists to prevent, what each costs in wall-clock time, which ones block a merge, and how the pipeline degrades so a slow test does not hold up an urgent security fix.→
Part XXXIX
Automation Platforms, RBAC and Event-Driven
7 checks
- What a platform adds over the command lineThe honest inventory of what AWX and Automation Controller give you - run history, central credentials, RBAC, scheduling, an API - and the things they do not give you at any price.→
- Projects, inventories, credentials, job templatesThe platform object model with the command line equivalent of each object side by side, and how to structure your own runs the same way without deploying anything.→
- RBAC as a blast-radius controlThe four permissions that matter - edit, run, target, authenticate - why separating them is the organisational form of --limit, and what the equivalent looks like with no platform at all.→
- Credentials the operator never seesHow a platform injects a secret into a run without disclosing it, why use-without-read is a genuine security property, and the concentration risk you accept in exchange.→
- Schedules, workflows and unattended runsScheduled jobs, workflow branching, approval nodes and surveys - and the failure modes specific to automation that changes production while nobody is watching.→
- Event-driven Ansible: sources, rules, actionsThe rulebook model - event sources, conditions and the eleven actions - where event-driven automation genuinely fits, and how it relates to the playbooks the rest of the course teaches.→
- Automated response without automating the outageHow an event loop that remediates can amplify an incident, and the controls that stop it: throttle, lock, a host cap per firing, an alert-only stage, approval above a threshold, and a kill switch a human can reach in seconds.→
Part XL
Patch and Reboot Management
8 checks
- Sizing the patch run before you write itChoosing the batch shape for a fleet patch run: the canary host, serial as a ramp, max_fail_percentage evaluated per batch and exceeded rather than equalled, and why forks is throughput and never safety.→
- Patching Debian and Ubuntuansible.builtin.apt in production: update_cache with cache_valid_time, what safe, full and dist actually mean, only_upgrade, lock_timeout, and what the default dpkg_options silently decides about your modified config files.→
- Patching RHEL, Rocky and Almaansible.builtin.dnf and dnf5, use_backend, security and bugfix filters and what upgrade-minimal semantics mean, update_only, exclude, repository scoping, allowerasing, and download_only to split a transaction across two windows.→
- The package module and the portability trapansible.builtin.package selects the package manager but never translates package names, its check-mode support depends on whichever plugin it dispatches to, and it documents only the intersection of options. When that abstraction earns its place and when it costs more than it saves.→
- Deciding whether a reboot is requiredDetecting reboot requirement without rebooting: stat on /var/run/reboot-required and its .pkgs companion on Debian-family, the inverted exit status of dnf needs-restarting -r on RHEL-family, comparing the running kernel against installed kernels, and recording the decision as evidence.→
- The reboot module actual contractEvery real parameter and default of ansible.builtin.reboot on 2.21.3: reboot_timeout evaluated twice, pre and post reboot delays, test_command, the boot_time_command that proves a boot happened, search_paths ignoring PATH, and wait_for_connection for reboots the module did not initiate.→
- Proving a host came back, not just answeredWhy SSH responding is not health: wait_for_connection, systemctl is-system-running and its eight states, a service_facts degraded check, mount verification, cluster membership, and an application probe with uri plus until and retries — arranged so a validation failure stops the next batch.→
- Rolling kernel upgrades across a fleetThe end-to-end rolling workflow — drain, patch, reboot, reconnect, prove the new kernel is running, return to service, next wave — and the argument for why a simultaneous fleet reboot is unacceptable rather than merely slow.→
Part XLI
Service and Application Deployment
8 checks
- What deploy means to a convergent toolDefining the deployable unit as package, configuration, service and validation converged together, and why a deploy step that is not rerun-safe turns every subsequent run into a gamble.→
- Service state, enablement and restart disciplinesystemd_service against the generic service module, state and enabled as two independent decisions, daemon_reload after a unit file changes, masked, restart driven only by handler notification, and meta flush_handlers before a validation gate.→
- Validating configuration before it can break a servicetemplate and copy with validate and the %s placeholder, the real validators for a deployment stack, backup: true and what to do with the backup, and the precise boundary between what a syntax check proves and what a running service will accept.→
- Health gates that actually gateuri with status_code and until, retries and delay and the time budget they really produce, wait_for on a port against a real readiness probe, asserting on response content rather than HTTP 200, and the difference between a gate and a monitor.→
- Rolling a deployment through a load balancerComposing serial with delegate_to for drain and undrain, throttle for a step that contends on a shared resource, and the run_once subtlety that makes a one-time migration run once per batch instead of once per run.→
- Deploying a version, not latestWhy state: latest in a deployment role makes a fleet non-reproducible, version-specifier installs for apt and dnf, get_url with a checksum as a converged artifact fetch, recording the deployed version as durable evidence, and expressing an upgrade as a reviewable change.→
- Why databases get different rulesSchema migrations are neither idempotent nor freely reversible, check mode lies about them, and run_once is mandatory and insufficient. Checking a node role before acting, separating converging the server from changing the schema, a verified backup, and a human gate automation must not bypass.→
- The limits of Ansible as a deployment toolWhat a push-based convergence tool structurally cannot provide — no atomic cut-over, no traffic control, no automatic revert, no continuous reconciliation — where the hand-off to a load balancer, orchestrator or scheduler belongs, and the signals that you have outgrown a push-based deployer.→
Part XLII
Ansible Beyond Linux Servers
7 checks
- Why network modules run on the controllerA switch cannot run Python, so Ansible inverts its own execution model for network devices: the module runs on the control node and only the resulting commands cross the wire. Connection plugins, ansible_network_os, enable-mode escalation and controller-side backups all follow from that one fact.→
- Changing a device you are connected throughThe only place in this course where a successful task can remove your ability to fix it. Out-of-band access as a precondition, diff before apply, confirmed commit where the platform offers it, and why serial: 1 stops being a preference here.→
- Automating APIs instead of hostsWhen the target has no SSH, the host loop becomes a liability: 300 inventory hosts issue 300 identical POSTs. run_once, throttle, controller-side execution, idempotency when the remote API is the state, and the verified fact that a uri task is silently skipped under --check.→
- Ansible and Docker hosts: drawing the lineAnsible converges the host and delivers the compose file; Docker owns runtime, restart policy and health. The collection can do far more than that, and the lesson is about why doing more creates two owners for one piece of state.→
- Ansible and Proxmox VE: drawing the lineProxmox owns the virtualisation platform; Ansible owns what runs inside the guests. The community.proxmox collection, API-token authentication, the proxmox inventory plugin, and the rule that a guest built by Ansible must not know which hypervisor it landed on.→
- Provisioning versus configurationWho owns the existence of the resource? Provisioning tools keep a record of what they created and can therefore destroy what you stopped declaring. Ansible has idempotency but no such record — using it to provision works, and gives up the property that makes provisioning safe to re-run.→
- A decision table for mixed estatesOne reviewable artefact covering Linux hosts, container hosts, hypervisors, network devices, cloud infrastructure and appliances: who owns each, what Ansible role is, and the specific cost of forcing Ansible into a slot a purpose-built tool already fills.→
Part XLIII
Observability and Auditing of Automation
7 checks
- Reading a PLAY RECAP like an operatorWhat each recap field actually asserts, verified field by field on 2.21.3: why changed=0 across a converged fleet is the healthiest possible result, why skipped hides more bad news than failed, and why hosts that never ran are absent rather than zero.→
- Changing what a run tells youCallback plugins decide what a run says. One stdout callback, many aggregate and notification callbacks. Verified on 2.21.3: which callbacks ansible-core actually ships, which two are deprecated, why the yaml stdout callback no longer exists, and why ad-hoc commands load none of them by default.→
- The controller log nobody turned onlog_path defaults to None, so by default no durable record of any run exists. Turning it on is one line and has three consequences: the file grows without bound, it is created with your umask, and it contains everything a task printed that no_log did not cover.→
- Machine-readable run artefactsA log is a rendering; an artefact is a structure. Which structured callbacks exist, what set_stats can stamp onto a run, and the metadata that turns a file nobody can interpret into evidence: the commit, the inventory, the limit, the vault ids and the operator.→
- "Who changed three hundred servers?"Reconstructing a change end to end by joining the controller run artefact to managed-node evidence. The attribution problem when every host sees one automation account, the joins that actually work, and the controls that put a human name back on a change.→
- Metrics that predict an automation failureFour measurements worth trending across runs: drift rate, run-duration distribution, unreachable-host trend and per-task failure hotspots. Each identifies rotting automation before the run that breaks production, and each is destroyed by a different kind of sloppiness.→
- Reporting outward without making it mandatoryNotification callbacks, external run recorders and platform job history, framed as options with honest costs. Callbacks run in the controller process and send module results outward; a production change must never depend on a chat service being up.→
Part XLIV
Failure Modes and Partial Fleet Failure
8 checks
- The seven things that can happen to a hostok, changed, skipped, failed, unreachable, rescued and ignored are seven different statements about what you know about a host. Verified on 2.21.3: ok silently includes changed and ignored, and rescued hides a real failure behind a green recap.→
- Three hundred hosts, two hundred and twenty successesThe judgement exercise this whole course is built around. 300 targeted, 220 ok, 30 failed, 50 unreachable — and the honest answer is that the estate now runs three different configurations, not that the change passed or failed.→
- Why unreachable hosts are the dangerous onesThe fifty hosts that said nothing. Why ignore_unreachable turns an unknown into a silent one — verified on 2.21.3, a play that never contacted a host reports ok=2 unreachable=0 and exits 0 — and why the quarantine list is the required output.→
- When the connection dies mid-taskThe one case where the host state is genuinely unknown: the module was delivered, the result never came back. Why that makes idempotency a safety property rather than a style preference, and what async with poll 0 does and does not fix.→
- Fighting the automatic updater for the package lockThe commonest cause of a mixed-result patch run. Verified on 2.21.3: apt lock_timeout defaults to 60, dnf to 30, and dnf5 documents its lock_timeout as a no-op. Why raising the timeout is not the fix and why this failure clusters rather than scattering.→
- Is it safe to run it again?Rerun safety is a per-task property you designed in weeks ago, not a question you answer at 03:00. Building the target list without .retry files — verified on 2.21.3, RETRY_FILES_ENABLED defaults to false — and the tasks that make a second run worse than the first.→
- Retry, roll back, quarantine or investigateThe four-way decision at 03:00, with explicit criteria, the evidence each branch requires, who is authorised to make the call, and the cost of choosing wrong. Written into the change plan before the run, because nobody judges this well with a half-converged fleet in front of them.→
- Stopping a bad change at host twelveThe mechanical controls that turn a 300-host disaster into a 12-host incident: serial as the unit of damage, max_fail_percentage which must be exceeded rather than equalled, any_errors_fatal which stops the playbook and not just the play, and force_handlers.→
Part XLV
Debugging and Troubleshooting
7 checks
- Verbosity is a disclosure decisionWhat each -v adds, measured on 2.21.3 — and the finding that shapes the part: the module result prints at -v, so a secret passed as a module argument is in your scrollback from the first level. Plus the 2.21 change that -vvvv no longer implies ssh -vvv.→
- Proving which hosts you are about to touchansible-inventory --graph, --list, --host and --yaml as a debugging instrument: answering why is this host in this group and why is this host not, before the run rather than after it.→
- Finding out which variable wondebug var versus msg (and the templated-var mistake that now fails outright on 2.21.3), reading hostvars and group_names, ansible_facts versus the injected ansible_ variables, and ansible-config dump --only-changed as the first question rather than the last.→
- Reading Jinja and templating errorsThe recurring error shapes on 2.21.3 and what each really means. The new caused-by chain points at the line that defined the value, not just the line that used it — and a string false in a when clause is now a hard error rather than a silent truthy.→
- Diagnosing transport failuresThe habit that resolves most of these: reproduce with plain ssh -vvv before changing anything in Ansible. Plus the settings that actually matter — ssh_args default, timeout 10, persistent timeouts 30, HOST_KEY_CHECKING true — and the agent-forwarding difference between a laptop and a controller.→
- Diagnosing privilege escalation failuresSeparating a sudo problem from an Ansible problem with one command, the unprivileged become_user case and the world_readable_temp option it pushes you toward, and why ANSIBLE_KEEP_REMOTE_FILES is a debugging tool and a security hazard in the same breath.→
- Reducing three hundred hosts to one reproductionThe narrowing toolkit in the order it should be used: --limit, tags, --start-at-task, --step, --check --diff and the debug strategy plugin. Including the trap verified on 2.21.3 — a --start-at-task run produces a recap indistinguishable from a full one.→
Part XLVI
Python and Interpreter Discovery
6 checks
- What actually runs on a managed nodeA module is a Python program copied to the target, executed there, and read back as JSON. What that requires on the target, what it does not require, and why controller Python and target Python are two different numbers.→
- How Ansible chooses a PythonThe discovery mechanism as implemented in ansible-core 2.21.3: where it runs in the play, the exact shell probe it sends, the fact it caches, and the definitive answer on auto_legacy read from the source rather than the documentation.→
- Pinning the interpreter as a production decisionWhen to stop relying on discovery: per-group pins, targeting a virtualenv that carries a module dependency, and the mixed-fleet case where identical hosts quietly run different Pythons.→
- Bootstrapping a host with no PythonThe one legitimate use of raw: installing an interpreter on a host that cannot run modules yet. Making a bootstrap play idempotent when raw has no idempotency of its own, and handing off cleanly once Python exists.→
- Targets that cannot run PythonAppliances, minimal images and embedded systems where Python is absent by design. Why command and shell are not an escape hatch, what raw and script can honestly do, and what automating these hosts actually looks like.→
- Windows in brief, and why it is differentWindows managed nodes need no Python: modules are PowerShell, delivered over winrm, psrp or ssh. A deliberate orientation rather than a Windows curriculum, and an honest statement of where this course stops.→
Part XLVII
Controller Security
7 checks
- Controller compromise is fleet compromiseThe controller holds credentials that are root on every managed node, and no agent-side authorisation stands in the way. An attacker with the controller does not need an exploit - they run a playbook.→
- Building a controller worth trustingA dedicated host with a minimal package set, no browsing and no development work, encrypted at rest, with restricted egress and its own patching schedule - and why "the controller is just my workstation" is the most serious version of this failure.→
- Who is allowed to run whatTurning a shared controller into an accountable one: named accounts rather than a shared login, sudo rules on the controller itself, per-environment credential separation, and the accountability collapse that follows from everyone having the vault password.→
- Keys that are only good for automationDedicated automation keys with no human use, the passphrase-versus-unattended trade, constraining a key at the far end with authorized_keys options, and certificate-based SSH as the scalable answer.→
- Rotating a key across a fleet without locking yourself outThe ordering that makes rotation survivable: distribute, verify on every host, only then remove. What to do about hosts that were unreachable during step one, and why break-glass access is the precondition for attempting rotation at all.→
- Vault identity custody and rotationWho holds the vault password, what that implies, and how to change it. Vault IDs from prompt, file or script, the empty DEFAULT_VAULT_IDENTITY_LIST default, and ansible-vault rekey with --new-vault-id.→
- Every collection you install runs as root everywhereThird-party content as a supply-chain risk: pinned requirements, what ansible-galaxy collection verify does and does not check, reading a role before its first run, and offline installs from a vetted internal mirror.→
Part XLVIII
Maintenance Windows and Rollback
7 checks
- Ansible has no undoAnsible provides no universal automatic rollback. What re-running an older revision actually converges, what it silently leaves behind, and why every change must be planned around that.→
- The reverse is part of the changeRollback is an artefact written before the window opens: the exact reverse procedure, who runs it, how long it takes, what triggers it, and the point after which going back costs more than going forward.→
- A catalogue of rollback patterns, and what each is forFive rollback patterns - config restore, package downgrade, artefact redeploy, snapshot revert, git revert and re-apply - each with the failure it answers, what it cannot undo, and when it is the right choice.→
- Running the windowPre-check, change, validation, rollback trigger, reporting - the five phases of a maintenance window, and the difference between a window that is a control and a window that is a ritual.→
- Canary first, with criteria written downProgressive exposure with serial, and abort criteria enforced by max_fail_percentage rather than by a tired operator noticing - demonstrated with executed output from an aborted run.→
- A backup you have not restored is a hypothesisWhat to capture before an automation change, where it must live, how the pre-flight play asserts its freshness, and why backup: true produces a file nobody has ever restored from.→
- Reconciling a fleet after a partial rollbackAn aborted change leaves three populations, not two. Building the reconciliation plan, proving convergence with check mode, choosing the reference state, and the report the change board is owed.→
Part XLIX
Compliance, Validation and Certificates
7 checks
- Reporting drift versus correcting itOne role, two modes: an audit run that tells you the estate has drifted, and an enforce run that changes three hundred hosts. Designing for both, and staging remediation so the control does not become the outage.→
- What check mode actually proves in an auditModule support varies, a skipped probe registers a plausible-looking result, read-only gathering needs check_mode: false, and --diff in an audit run prints secrets into the evidence package.→
- Producing evidence an auditor can readTurning a run into durable evidence: package_facts, service_facts and a scoped setup, written as one machine-readable file per host per run with the timestamp, the policy version and the host identity.→
- Validation plays that fail loudlyassert, fail and failed_when as a validation play run after every change - including the executed proof that a list of failed_when conditions is joined with an implicit and, so a compliance check written that way can essentially never fire.→
- Certificates, end to endcommunity.crypto in sequence - private key, CSR, certificate - then deployment with the right owner, group and mode, a handler that reloads rather than restarts, and expiry read as a fact so renewal is scheduled instead of triggered by an outage.→
- Rotating a secret across three hundred hostsA shared secret cannot change atomically across a fleet, so rotation needs a dual-acceptance window, an ordered rollout, no_log throughout, a verification step proving the old value no longer works, and a plan for the hosts that were unreachable.→
- Compliance on a schedule, read by someoneRunning audit playbooks on a timer, where results go, alerting on drift rate rather than individual diffs, tuning out expected noise, and the failure this part cares about most - an audit nobody reads manufactures the belief that someone is looking.→
Part L
Automation Disaster Recovery
6 checks
- The controller is permanently lostThe disaster scenario the whole part answers - the automation host is destroyed, not restorable, and the fleet still needs managing tomorrow morning.→
- What belongs in the repository, and what must notThe recoverable set - playbooks, roles, inventory sources, ansible.cfg, requirements.yml and vault-encrypted variables - and the set whose presence turns a repository theft into a fleet compromise.→
- A controller you can rebuild byte-for-byte enoughPinning every layer - ansible-core, Python, collections, roles and Python dependencies - and proving the rebuild produced the same versions rather than assuming it.→
- Backing up the things you must not lose or leakVault passwords and automation private keys have to survive the controller being destroyed without becoming stealable - escrow, split custody, break-glass, and periodic proof that the escrowed material still works.→
- Execution environments as a reproducibility answerThe containerised answer to "works on my controller" - execution-environment.yml version 3, ansible-builder and ansible-navigator - presented as one legitimate option rather than a mandate.→
- The timed rebuild drillThe exercise that converts a recovery plan into evidence - rebuild the controller on a fresh host from git plus escrow, run a read-only play against the fleet, and record how long it took.→
Part LI
Custom Modules and Tool Selection
6 checks
- Exhaust the alternatives firstThe ordered list of things to try before writing module code - an existing collection, uri against the API, command with an honest changed_when, a plugin, or a well-shaped role - and the maintenance cost you are choosing to take on.→
- Writing a module that behaves like a real oneThe concrete API - a file in library/, AnsibleModule, argument_spec, supports_check_mode, module.params, exit_json and fail_json - built and executed against ansible-core 2.21.3.→
- The contract a module owes its usersSix properties a custom module must have before it is safe against a fleet - genuine idempotency, an accurate changed value, real check mode, no_log on secrets, failure messages that name the cause, and documented returns - each tied to the failure it prevents.→
- Testing it, then shipping it properlyFast local iteration with a JSON args file, ad-hoc runs via ANSIBLE_LIBRARY, real sanity tests with ansible-test, and packaging into a versioned private collection instead of copying library/ between repositories.→
- Filter, lookup, module or action pluginThe decision turns on one question - does this code need to run on the controller or on the target? - and choosing wrong produces automation that works on the developer machine and nowhere else.→
- When Ansible is the wrong toolFive problems Ansible is structurally unsuited to - high-frequency events, very large data transfers, stateful provisioning, complex orchestration loops and real-time monitoring - each with the failure you get if you ignore it.→
Part LII
Anti-Patterns
8 checks
- Anti-pattern: shell everywherecommand and shell used where a module exists - no idempotency, no check mode, no diff - so a rerun re-performs the action, --check proves nothing, and a partially applied change reports success.→
- Anti-pattern: secrets in GitPlaintext credentials in the repository, or a vault password committed beside the vault - an irrevocable leak whose remediation is a fleet-wide credential rotation, not a git rm.→
- Anti-pattern: no limit, no canaryhosts: all with no --limit, no serial and no canary host - a bad template reaches 300 machines in ninety seconds and there is no healthy population left to diagnose against or roll back from.→
- Anti-pattern: ignore_errors as a green-build buttonBlanket ignore_errors: true and its cousin ignore_unreachable, applied until the run is green - automation that reports success during an outage it caused.→
- Anti-pattern: tasks that always change, or never doA changed result that does not correspond to a change - producing restart storms on a converged fleet, and turning the drift signal into noise so real drift becomes invisible.→
- Anti-pattern: encoding the inventory in whenDeep when chains, hostname regexes and hard-coded hostnames doing the job of groups and group variables - so nobody can answer "which hosts does this task touch?" without executing it.→
- Anti-pattern: latest everywherestate: latest in roles, unpinned collections, no requirements.yml and no controller pinning - so two hosts converged a week apart are not the same host, and a rebuild does not reproduce the environment.→
- Anti-pattern: automation nobody tests, and automation nobody dares runTwo ends of the same anti-pattern - production as the first test environment, and its endpoint, automation too frightening to run so hosts get fixed by hand and the repository starts lying.→