Git, CI/CD & GitOps · Self-assessment
Knowledge checks
Every knowledge check in this course, in curriculum order. Each link opens the page at its quiz. The questions are auto-graded in the browser and nothing is recorded — a wrong answer costs you only the explanation, which is the part worth reading.
- Knowledge checks
- 716
- Parts covered
- 119
- Of all lessons
- 100%
Part I
Version Control Foundations
6 checks
- Version control foundations — snapshots, history, and auditabilityWhy version control exists; what a snapshot is; how history, collaboration, reproducibility, and auditability change the operational picture for an infrastructure team.→
- Snapshots and the history model — what a commit really recordsWhat a Git snapshot is, how content-addressed storage makes commits tamper-evident, and how the snapshot model differs from a diff-based model.→
- Collaboration and conflict — concurrent edits and the merge boundaryHow Git handles concurrent edits, when automatic three-way merge succeeds and when it surfaces a conflict, and why the conflict boundary is the design point of the whole model.→
- Reproducibility and immutable references — the commit hash as the unit of reproducibilityWhy a commit hash is the only safe unit of reproducibility, why mutable references like `latest` and branch names break reproducibility, and how pinning works in an infrastructure repository.→
- Auditability and chain of trust — from production state back to a commitWhat an audit trail requires, why a commit hash alone is not enough, and how the chain of trust connects production state to a reviewer and a rationale.→
- Infrastructure-as-code implications — why IaC has a stricter version-control bar than application codeWhy IaC has a stricter version-control bar than application code: wider blast radius, narrower rollback, stronger audit requirement, and the practical discipline that follows from each.→
Part II
Git Architecture
6 checks
- The working tree, the index, and the repository — the three areas Git operates onWhat the working tree, the index, and the repository each represent; how a commit moves state through them; how to read git status and git diff with the three-area model in mind.→
- The three-trees model — HEAD, index, and working tree as a navigation systemHow HEAD, the index, and the working tree form a three-trees model; how git diff with no arguments compares different pairs; how reset, restore, and checkout move state between the three trees.→
- Plumbing versus porcelain — the low-level commands Git is built onWhat plumbing commands are and how they differ from porcelain; why Git exposes them; how to use cat-file, hash-object, write-tree, and update-ref for forensics and scripted operations.→
- The .git directory layout — what lives where under the repositoryWhat the .git directory contains; what objects/, refs/, HEAD, config, hooks/, and packed-refs each store; how to navigate the repository on disk for forensics and operational tasks.→
- Environment variables and config files — how Git finds its repository and its settingsHow Git locates the repository, the index, and the working tree; the role of GIT_DIR, GIT_WORK_TREE, GIT_OBJECT_DIRECTORY, GIT_INDEX_FILE, and GIT_DIR; how the config cascade works; when to override each one.→
- Git as a content-addressed store — why hashes are the identityWhat "content-addressed" means for Git; how a hash collision would imply the same content; how SHA-1 and SHA-256 work for Git objects; why objects are immutable; and what this means for forensic trust.→
Part III
Git Objects
6 checks
- The three object types — blob, tree, commit, tagWhat Git stores on disk: blob, tree, commit, and tag objects. How each type tags its own payload, how they reference each other by OID, and why understanding the object graph is required before reading refs, signing, or transfer protocols.→
- Object IDs and hashing — what the SHA covers and why collisions matterHow Git object IDs are computed, what exactly is fed into the SHA, why SHA-1 is being replaced by SHA-256, what a hash collision would imply, and how to choose the right algorithm for a new repository.→
- Blob objects — content only, no filename, deduplicated by hashWhat a blob is, why blobs carry no filename or path, how identical content produces a single shared blob, how to create blobs with git hash-object -w, and how to inspect them with git cat-file.→
- Tree objects — directory entries, mode bits, and recursive sub-treesWhat a tree object is, how mode bits and OID references make up a tree entry, how trees nest to represent directories, how to write a tree with git write-tree, and how index entries become tree entries via git update-index.→
- Commit objects — parents, tree pointers, author versus committer, signaturesWhat a commit object contains, how parents chain into a history DAG, how merges produce two parents, how author differs from committer, and how GPG signatures are embedded in the commit payload.→
- Tag objects — annotated tags, lightweight tags, and signed tagsHow annotated tags differ from lightweight tags, what a tag object contains, how signed tags embed a GPG signature, and how git verify-tag validates the chain of trust.→
Part IV
Commit Graph and History
6 checks
- The commit DAG — what Git is really a graph ofWhat a Directed Acyclic Graph is, why Git models history as a DAG and not a linear list, why acyclicity is a non-negotiable invariant, and how this single decision shapes every log, merge, and traversal command.→
- Parent references — first parent versus all parentsHow a commit's parent lines encode history topology, what the first parent means by convention, why the --first-parent flag in git log produces a stable trunk view, and how merge and octopus commits encode multiple parents.→
- Ancestry and reachability — what `reachable` actually meansWhat it means for commit B to be reachable from commit A, how `git merge-base` finds the common ancestor, how `git log A..B` and `git log A...B` differ, and why reachability is the foundation of every history traversal command.→
- History traversal — log options that actually change the graph walkHow git log traverses the DAG, what each of --graph, --oneline, --decorate, --all, --topo-order, --date-order, --reverse, --follow, and --merges does, and which combinations serve which operational question.→
- Graph topology and merges — what merge commits actually encodeWhat a merge commit looks like in the DAG, how a three-way merge produces two parents, how the merge base and the recursive strategy resolve conflicts, and how octopus merges encode many parents.→
- The octopus and its cost — when multi-parent merges help and when they hurtWhen octopus merges are the right tool, why they are the wrong tool for active conflict resolution, the cost of deep merge chains on log traversal and merge-base computation, and how an octopus in the wrong place warps the entire DAG downstream.→
Part V
Branches, Refs and HEAD
6 checks
- Refs and the refs namespace — refs/heads, refs/tags, refs/remotesWhat a Git ref is, where refs live in .git/refs, the on-disk file format of a loose ref, the three top-level namespaces (heads, tags, remotes), and why a ref is just a 40-character SHA-1 pointing at a single object.→
- Branches are pointers — what a branch really isA branch is a moving ref under refs/heads, not a directory of files. Creating, listing, deleting, and pointing a branch; what git branch does and what it deliberately does not (it does not switch the working tree).→
- HEAD and current state — what HEAD actually isHEAD is a symbolic ref pointing at a branch ref, which points at a commit. The difference between HEAD and refs/heads/main; inspecting HEAD with git rev-parse; what HEAD means for the working tree and the index.→
- Detached HEAD state — what it means, why it is dangerous, and how to recoverA detached HEAD points directly at a commit rather than at a branch. How you get there (checkout by OID, tag, or remote ref), why commits on a detached HEAD can be lost, and how the reflog is the recovery path.→
- Tags versus branches — what tags are for, annotated versus lightweight, and why branches are not releasesTags are intended to be immutable release markers; branches are moving pointers. Annotated versus lightweight tags; why a branch should not be used as a release marker; what tools depend on tag immutability (git describe, signed tags, CI pinning).→
- Packed refs and the reflog — when refs are packed, and how the reflog records every ref changeWhen Git packs refs into a single file under .git/packed-refs to reduce filesystem overhead; how the reflog records every ref change as a time-stamped entry; reflog retention, expiration, and interaction with garbage collection.→
Part VI
Index / Staging Area
6 checks
- The index explained — what the staging area is and why Git has oneWhat the index actually is on disk, why Git separates the staging area from the working tree, how the three-trees model collapses into a commit, and why every commit is a deliberate reviewable event.→
- git add mechanics — what staging actually does under the hoodWhat git add writes to the index, how the . and -A flags differ, when --intent-to-add and --chmod are useful, how renames detection works, and how to stage a single file deliberately.→
- Partial staging with -p — hunk-level staging for clean commitsHow git add -p splits a file into hunks and lets you stage some hunks while leaving others in the working tree, when hunk-level staging is the right tool, and how to drive the interactive prompts.→
- Staging versus skipping — git add, .gitignore, and the untracked bucketThe difference between a file that is not staged and a file that is ignored, how .gitignore patterns work, when git add --force is appropriate, and why infrastructure repositories need a deliberate ignore policy.→
- The index as a commit preview — git diff --cached and the review before commitHow git diff --cached turns the index into a preview of the next commit, why every CI pipeline and pre-commit hook reads from the index, and how to build a commit preview discipline into the workflow.→
- Resetting and restore on the index — git reset, git restore, and choosing the right undoHow git reset and git restore move state between the three trees, the difference between reset --soft, --mixed, and --hard, when to use git restore --staged versus git restore, and the production-safe alternatives to git reset --hard.→
Part VII
Repository Inspection
6 checks
- git status decoded — staged, unstaged, untracked, and the porcelain contractWhat every line of git status means, how the short status codes map to movements between the three trees, and why --porcelain is the only stable contract for scripts.→
- git log fundamentals — reading history, decoding the default outputWhat every line of git log means, how the commit hash, author, date, and subject are formatted, and how --oneline, --graph, --decorate, and --all rewrite the default view for production use.→
- git log formatting — pretty formats, custom output, and machine-readable contractsHow --pretty=format placeholders work, how --date formats interact with the format string, and how to build a machine-readable output that is safe to parse in CI scripts.→
- git show and the commit object — inspecting one commit in fullWhat git show displays for a single commit, how to use the rev:path syntax to extract one file at one commit, and how show differs from log and diff when investigating a historical change.→
- git diff three ways — working tree, index, and HEADThe three diffs git can produce in the working tree, how --cached and HEAD modify the scope, how --stat summarises and --word-diff highlights changes, and the production use of each form.→
- git blame and annotation tracking — per-line authorship and code archaeologyHow git blame maps every line of a file to its introducing commit, how to use -L ranges to restrict blame to a region, and how --ignore-rev and --ignore-rev-file exclude noisy commits from the annotation.→
Part VIII
Branching
6 checks
- Branch creation and switching — git switch, git checkout, and the three-tree updateHow a branch is created with git switch -c or git checkout -b; what happens to HEAD, the index, and the working tree when you switch branches; why a clean working tree is required for a safe switch.→
- The branch lifecycle — listing, sorting, filtering, and auditing branchesgit branch, -v, -a, --no-color, --merged and --no-merged: how to read the local branch list, the remote-tracking mirror, and the merged-vs-unmerged boundary that drives safe deletion.→
- Divergence and shared history — the commit graph when two branches splitWhat "divergence" means in the commit DAG; how two branches share a common ancestor; why reachability, not chronology, determines whether a branch can be fast-forwarded or must be merged.→
- Tracking and upstream — what `@{u}` means and how branches are linked to remotesHow local branches track remote branches; what git branch --set-upstream-to does; the @{u} and @{upstream} shorthands; the difference between a remote-tracking ref and an upstream configuration.→
- Branch naming and organization — patterns that signal intent to the teamConventional branch prefixes (feature/, bugfix/, release/, hotfix/); what a branch name tells the team about lifetime, owner, and merge target; slash-namespaces and the ref storage they imply.→
- Branch deletion and recovery — git branch -d, git branch -D, and the reflog resurrection pathThe safe delete (-d) versus force delete (-D); what happens to orphaned commits; how the reflog makes recovery possible; the window during which a deleted branch can be resurrected.→
Part IX
Merging
6 checks
- Fast-forward merges — when a merge is just a pointer moveWhen a merge can be a pointer move instead of a new commit; how Git decides between fast-forward and true merge; why fast-forward happens; the --ff-only flag; the operational consequences of a linear history.→
- Three-way merges — when fast-forward is not possibleWhen fast-forward cannot resolve a merge; what the merge base is; how Git uses the base plus the two tips to construct a merged tree; why three-way merges can produce conflicts; how recursive merge differs from the older resolve strategy.→
- Merge commits and --no-ff — forcing topology for auditWhy a team might want a merge commit even when fast-forward is possible; what --no-ff does to the graph; the audit and rollback trade-offs of merge commits versus linear history; how --no-ff interacts with merge.ff configuration.→
- The merge process — what Git actually does during a mergeThe end-to-end steps Git takes during a merge: reading tree objects, finding the merge base, running the recursive strategy, staging the result in the index, writing MERGE_HEAD and the merge commit; what the working tree looks like at each phase; what `git status` reports during a merge.→
- Aborting a merge — when to abort, and how the state is restoredWhen to use `git merge --abort` versus `git merge --quit`; what state is restored by abort; the difference between abort and reset; how to detect an in-progress merge; when NOT to abort; the relationship between abort and the reflog.→
- Merge strategies — recursive, resolve, octopus, ours, subtreeThe five built-in merge strategies in Git: recursive (default), resolve, octopus, ours, subtree; when each is the right choice; how --strategy-option modifies behaviour (patience, diff-algorithm, ignore-all-space); how to choose a strategy for an infrastructure repository with criss-crossed history.→
Part X
Merge Conflicts
6 checks
- When merges conflict — the three cases the algorithm cannot resolveThe three cases that produce a merge conflict in a three-way merge: both sides modify the same line, one side deletes a file the other side modifies, and both sides modify a binary file. What each case looks like in `git status`, what it means for the merge, and why each case is a semantic ambiguity only a human can resolve.→
- The conflict markers — reading <<<<<<<, =======, and >>>>>>>What each conflict marker means, how the file is staged during a conflict (the three-stage index), how `git diff`, `git diff --cached`, and `git diff --base/--ours/--theirs` show different views of the same conflict, and how the staged file moves from "unmerged" to "resolved" the moment `git add` touches it.→
- Resolving by hand — opening files, reading hunks, choosing sidesHow to resolve a textual conflict manually: reading the marker block, understanding what each side intended, choosing the combined content, removing the markers, verifying with `git diff`, and committing with `git merge --continue`. When manual resolution is the right choice over a merge tool, and how to express a "neither side, both sides, or new combination" resolution in the working tree.→
- Resolving with tools — git mergetool, VS Code, and merge tool configurationHow to launch `git mergetool` during a conflict, what merge tools show (three-way layout, ours, theirs, base, merged result), how to configure a merge tool (`merge.tool`, `mergetool.<tool>.path`), and how VS Code, vimdiff, meld, kdiff3, and beyond compare are integrated. When a merge tool is worth setting up versus when the conflict is small enough to resolve by hand.→
- IaC conflict examples — Terraform state, Ansible inventory, Kubernetes YAMLConcrete merge conflicts in infrastructure repositories: Terraform state files (binary, case 3, requires not committing state); Ansible inventory conflicts (YAML host lists, case 1 and case 2 mixes); Kubernetes manifest conflicts (case 1 with semantic JSON shape). Why Terraform state is uniquely dangerous to merge, why inventory files produce recurring case-2 conflicts, and why K8s YAML conflicts often look clean but are semantically broken.→
- Conflict prevention — short branches, small commits, rerere, and the human layerHow to reduce the rate of merge conflicts before they happen: short-lived branches, small and atomic commits, frequent rebases against main, communication between engineers, file ownership via CODEOWNERS, and `git rerere` to remember past resolutions. Why "fewer conflicts" comes from workflow shape rather than from any single Git setting.→
Part XI
Rebasing
6 checks
- What rebase does — replaying commits on a new baseWhat `git rebase` does to a branch; how it replays commits one at a time on top of a new base; why every replayed commit gets a new OID; the difference between rewriting local history and rewriting shared history.→
- Rebase versus merge — when linear history is worth the rewriteThe trade-off between merge (preserves topology, creates a merge commit) and rebase (preserves linearity, rewrites OIDs); when each is the right verb; how to choose for a team policy; the local-vs-shared boundary that constrains the choice.→
- Interactive rebase — the editor interface and the six verbsHow `git rebase -i` opens the todo list; the verbs pick, reword, edit, squash, fixup, drop; how the editor rearranges the commit sequence; what each verb does to the resulting history.→
- Rebase execution and stoppoints — when rebase pauses and how to resumeWhen `git rebase` stops mid-replay; the three exit verbs `--continue`, `--abort`, `--skip`; how conflict resolution works during a rebase; the `.git/rebase-merge` and `.git/rebase-apply` state directories; using `git rebase --exec` to run commands between commits.→
- Autosquash and fixup — folding review-fixup commits automaticallyHow `git commit --fixup=<sha>` and `git commit --squash=<sha>` mark a commit for autosquash; how `git rebase --interactive --autosquash` (or `git rebase -i --autosquash`) reorders the todo list; the cleanup workflow for review-fixup commits before merging.→
- Shared history risks — why rewriting pushed commits is dangerousThe golden rule of rebasing; why rewriting commits that have been pushed breaks teammates, CI, artifact registries, and signed tags; the difference between `--force` and `--force-with-lease`; how `--force-with-lease` catches remote-tip divergence.→
Part XII
Merge vs Rebase
6 checks
- The merge versus rebase trade-off in detail — preservation versus rewriteWhy merge preserves shared history and rebase rewrites it; the consequences for the audit trail, the rollback path, and the recovery procedure; what each verb commits the repository to; how to read the resulting graph for forensic purposes.→
- When to rebase — local branches, cleanup, and pre-merge replayThe four scenarios where rebase is the right verb: local unpushed branches, feature branches before review, history cleanup with interactive rebase, and pre-merge replay onto the trunk tip; the boundaries that contain each scenario safely.→
- When to merge — shared branches, integrations, and release topologyThe scenarios where merge is the right verb: anything pushed and consumed, integration branches, long-lived shared branches, and release branches; why --no-ff forces a merge commit for the audit trail; how merge commits record the integration event in the DAG.→
- Team policy and consistency — why the team must pick one verb and stick to itThe cost of an inconsistent merge-rebase policy across a team; why the choice must be made explicit in CONTRIBUTING; the configuration keys that enforce the policy; how to migrate a repository to a consistent policy without rewriting history.→
- IaC and the merge-rebase question — Terraform state, configuration drift, and forced mergesWhy Terraform state and configuration drift force the merge verb on protected branches; how state file references interact with OID rewrites; when merge is the only correct verb for IaC repositories; the special cases of state file pinning and module consumption.→
- The hybrid workflow — feature rebase, fast-forward into main, and trunk-based developmentHow teams combine rebase and merge in a single workflow; the feature-branch-rebase-then-fast-forward pattern; how trunk-based development uses short-lived branches and fast-forward merges; the operational rules that keep the hybrid safe.→
Part XIII
Cherry-Pick
6 checks
- What cherry-pick does — replaying a single commit onto another branchHow `git cherry-pick <commit>` applies a single commit to the current branch as a new commit; why the new commit has a different OID; what `git cherry-pick -x` records in the message; the boundary between cherry-picking and merging.→
- Cherry-pick conflicts — when the replay does not apply cleanlyWhat happens when the cherry-picked commit and the current branch have diverged; how to read the conflict markers; how to use `git cherry-pick --continue`, `--abort`, and `--quit`; when `--no-commit` (`-n`) is the right choice.→
- Backporting hotfixes — moving a fix without merging everythingHow production release and maintenance branches use cherry-pick to move targeted fixes between branches; the release/maintenance workflow; what to backport and what to leave; how to record the move with `-x` and release notes.→
- Cherry-picking multiple commits — ranges and batchesHow `git cherry-pick A..B` picks a range of commits; the inclusive/exclusive behaviour of the range; how `--no-commit` (`-n`) combines a batch into one commit; ordering and stop-on-conflict semantics; the production use case of a multi-commit backport.→
- Cherry-pick versus merge — choosing between targeted and combined historyWhen cherry-pick is the right tool and when merge is; the duplicated-history cost of cherry-pick; the long-term-divergence cost of selective cherry-pick; the operational decision tree between the two for an infrastructure team.→
- When not to cherry-pick — changes that must travel as a unitChanges that should not be cherry-picked: shared abstractions, refactors that cross file boundaries, configuration that diverges by environment, security-sensitive changes whose audit trail depends on a single OID. The discipline of saying \"merge, do not cherry-pick\" before the cost of getting it wrong is paid.→
Part XIV
Revert
6 checks
- What revert does — producing a new commit that undoes a changeHow `git revert <commit>` produces a brand-new commit that inverts the changes of an earlier commit; the difference between an undo (history-preserving) and an erase (history-rewriting); the role of revert in shared-history workflows.→
- Revert versus reset — additive undo versus history rewritingThe fundamental contrast between `git revert <commit>` (adds a new commit that undoes the change) and `git reset <commit>` (moves the branch tip backward, rewriting the OIDs of every commit that comes after); why revert is safe on shared branches and reset is not; the soft/mixed/hard variants of reset and what each keeps.→
- Reverting a merge commit — `git revert -m 1 <merge>` and the mainline parentHow `git revert -m 1 <merge>` specifies which parent of a merge commit is the mainline; why `-m 1` is the right choice when reverting a feature merge into `main`; the difference between reverting a merge and reverting a non-merge; what happens to the original branch and the second parent.→
- Revert without committing — staging the inverse with `git revert -n`How `git revert -n` (also `--no-commit`) stages the inverse changes in the index without creating a commit; when staging without committing is the right workflow; how `--no-commit` interacts with multi-commit reverts and conflict resolution; the discipline of combining multiple inverses into a single reviewable commit.→
- Revert and the history audit trail — why every revert is visibleWhy `git revert` produces an explicit, visible, named commit that says what was undone; the audit value of explicit reversals over silent fixes; how `git log --grep=^Revert`, `git show`, and `git blame` treat revert commits; the discipline of keeping revert messages intact for the auditor.→
- Revert versus redeploy — when to roll back, when to fix forwardThe choice between `git revert` (a rollback commit that undoes the change in production) and a redeploy (a new release that fixes the problem forward); when each is appropriate; how GitOps controllers, image tags, and deployment pipelines make the choice; the production framing of "revert for rollbacks, redeploy for fixes".→
Part XV
Reset
6 checks
- Reset modes explained — --soft, --mixed, --hard, --merge, --keepThe five reset modes in git reset and how each one moves HEAD, resets the index, and resets the working tree; the mental model that turns reset from a single overloaded command into five distinct operations with different blast radii.→
- Soft reset — moving HEAD and keeping staged changesHow `git reset --soft <commit>` rewinds the branch tip while leaving the index and working tree intact; the canonical use cases — fixing the last commit message, splitting a commit, adding a missed file to the last commit.→
- Mixed reset — moving HEAD and resetting the indexHow `git reset --mixed <commit>` (the default) rewinds HEAD and resets the index to match while leaving the working tree intact; the canonical use — unstage files without losing edits — and the relationship to `git restore --staged`.→
- Hard reset — the destructive mode and when it is acceptableHow `git reset --hard <commit>` rewinds HEAD, resets the index, AND rewrites the working tree; the destructive blast radius; the only acceptable use cases — local-only mistakes on unpushed branches; why the recovery path is the reflog.→
- Reset with file paths — `git reset <commit> -- <path>` and the reset-vs-restore distinctionThe path-scoped form of `git reset <commit> -- <path>`; the three argument slots in a reset command; why path-scoped reset never moves HEAD; the reset-vs-restore distinction for the index and the working tree.→
- Reset safety and recovery — when not to use --hard, reflog-based recovery, --merge and --keepThe production rules for `git reset` — when --hard is acceptable and when it is forbidden; the reflog-based recovery path; the two conditional modes --merge and --keep and how they differ from --hard; the relationship between reset, revert, and restore.→
Part XVI
Restore and Switch
6 checks
- The Git 2.23 command split — why git checkout was overloaded and how switch and restore replaced itWhy git checkout did too many things; the August 2019 Git 2.23 release that introduced git switch and git restore; the design intent of the split; what stayed inside git checkout and why.→
- git switch in detail — branch switching, creation, detachment, and orphan branchesThe full flag set of git switch: positional branch name, -c and -C for creation, -d and --detach for detached HEAD, --orphan for branchless histories, --disjoint-checks for safety, and - for the previous branch.→
- git restore in detail — restoring files from the index, from HEAD, and from any refThe full flag set of git restore: restoring the working tree from the index, restoring the index from HEAD with --staged, restoring both with --staged --worktree, pulling from a named ref with --source, and the common defaults.→
- Restore and the three trees — sources, destinations, and which combinations are validMapping git restore onto the three-tree model: HEAD, index, and working tree as sources and destinations; valid combinations; which combinations are destructive; the symmetric view of the command.→
- Switch versus checkout — when each is appropriate, behaviour differences, and the migration storyA side-by-side comparison of git switch and git checkout; behaviour differences that go beyond a simple rename; the cases where checkout is still the right choice; the migration path from checkout to switch and restore.→
- Migrating team habits — muscle memory, training, CI scripts, and what to update when migratingHow to roll out git switch and git restore across a team: documentation, training, alias strategy, CI script audits, and the difference between a policy and a culture change; what to update in onboarding, runbooks, and dashboards.→
Part XVII
Reflog
6 checks
- What the reflog records — every ref update, every time, with a reasonThe reflog is a per-clone log of ref updates, not of operations or commits. Every time a named ref changes value, the reflog records the old OID, the new OID, the ref name, the timestamp, and a reason. HEAD is always logged; branch and remote-tracking reflogs exist when those refs move.→
- Reflog locations and scopes — per-ref logs, the .git/logs tree, and enabling logging for tagsWhere reflog files live under .git/logs; the difference between the HEAD reflog and per-branch reflogs and per-remote reflogs; how to opt lightweight tags and custom refs into the reflog with core.logAllRefUpdates; the .git/logs/HEAD special case.→
- Navigation with reflog — @{N}, @{date}, and using reflog as a history of HEADHow to use reflog selectors @{0}, @{1}, HEAD@{N}, HEAD@{yesterday} to navigate to a previous state of HEAD; the difference between numeric and date-based reflog selectors; combining reflog with reset, switch, and checkout for navigation and recovery.→
- Recovering from a hard reset — the precise reflog recipe and the boundaries of recoveryThe four-step reflog recipe for recovering from a mistaken git reset --hard; what survives (the reflog entry, the orphan commit objects) and what does not (the branch pointer, the relationship to upstream); the 90-day retention boundary; the case where recovery is not possible.→
- Recovering from a bad rebase — reflog, ORIG_HEAD, and the cherry-pick recipeHow a bad rebase loses reflog entries for the rebased branch but preserves them in HEAD’s reflog; the role of ORIG_HEAD; the cherry-pick recovery recipe that replays the lost commits; the branch-from-orphan alternative.→
- Reflog expiry and gc — git reflog expire, gc.reflogExpire, and the choreography with garbage collectionHow git reflog expire prunes reflog entries by age and reachability; the two configuration knobs gc.reflogExpire and gc.reflogExpireUnreachable; the relationship with git gc and git gc --prune; the gc.pruneWorktrees option; the audit-clone discipline.→
Part XVIII
Git Recovery
6 checks
- The recovery mindset — stop, inspect, locate, recoverThe four-step discipline for recovering from any Git accident: stop and do nothing else, inspect the reflog before typing any history-rewriting command, locate the target OID, and recover via a non-destructive command. Why panic wastes time and why the first minute after an accident is the most expensive.→
- Recovering a deleted branch — the reflog is the recovery pathHow to recover a branch that has been deleted with git branch -D or git push --delete; the reflog is the recovery path; the precise recipe (git reflog, git branch <name> <oid>); the 90-day retention boundary; the case where the reflog has expired and the recovery path is fsck or another clone.→
- Recovering a dropped stash — git stash list, the reflog, and the .git/logs/refs/stash fileHow to recover a stash that has been dropped with git stash drop; the two recovery paths (git stash list and .git/logs/refs/stash); the precise recipe (git stash apply or git stash branch from the OID); why a dropped stash is recoverable longer than a deleted branch.→
- Recovering an amended commit — the previous commit is in the reflogHow to recover a commit that has been amended with git commit --amend; the previous commit is still in the reflog at HEAD@{1}; the original OID becomes a dangling commit; the recovery recipe (git cherry-pick or git branch from the OID); what the original OID becomes after the amend.→
- Recovering a commit after a shared rebase — the original is still in your local reflogHow to recover a commit that has been force-pushed away by a teammate's rebase; the original commit is still in your local reflog even after the remote has been rewritten; the recovery recipe (git cherry-pick or git branch from the OID); the difference between the local reflog (preserved) and the remote ref (rewritten); the --force-with-lease discipline.→
- The recovery decision tree — a flowchart for any "I lost X" scenarioA flowchart-driven approach to any Git recovery scenario: identify what was lost (branch, stash, commit, working tree), identify the recovery path (reflog, fsck, remote ref, backup), execute the recipe. The decision tree covers the four common accident types and the cross-cutting recovery paths (other clones, fsck, remote refs, backups).→
Part XIX
Tags and Releases
6 checks
- Lightweight versus annotated tags — what each stores and why the audit trail differsWhat a lightweight tag stores on disk (just a commit OID); what an annotated tag stores (a tag object with tagger, date, message, optional signature); why annotated tags are content-addressed; the audit trail implications for release pipelines.→
- Creating and listing tags — git tag, -a, -m, -d, -l with patterns, and git tag -nHow to create lightweight tags (git tag <name>), annotated tags (git tag -a <name> -m msg), tag a specific commit (git tag -a <name> <commit>), delete a tag (git tag -d <name>), list tags with patterns (git tag -l "v1.*"), and show tag annotations (git tag -n[<num>]).→
- Tag pushing and fetching — why git push does not push tags by default, and what --tags and --follow-tags dogit push does not push tags by default; the explicit invocations are git push origin <tag>, git push origin --tags (all tags), and git push origin --follow-tags (annotated tags reachable from pushed commits). On fetch, git fetch --tags pulls every tag; a plain git fetch pulls the tags the remote advertises for the fetched refs.→
- Signed tags — git tag -s, git tag -u, and verification with git verify-tagHow to sign annotated tags with git tag -s (default key) or git tag -u <keyid> (specified key); how git verify-tag validates the signature; the difference between signing and verifying; the gpgsig block in the tag payload; the role of signed tags in the supply-chain chain of trust.→
- Tag protection and releases — repository settings, release notes, and immutability via signingHow to protect release tags in GitHub/GitLab settings (prevent deletion, require signed tags, restrict who can create); the release-notes workflow that turns a tag into a published release; how signing plus tag protection produces a chain of trust from commit to published artifact.→
- Release workflows — tag-on-merge, release branches, semantic versioning, release artefacts, and the deploy-after-tag patternThe end-to-end release workflow for an infrastructure repository: tag-on-merge to the default branch, semantic versioning rules (MAJOR.MINOR.PATCH and pre-release suffixes), release branches for backports, the release artefact as the unit of deployment, and the deploy-after-tag pattern that pins production to the tag.→
Part XX
Remotes
6 checks
- What a remote is — a named pointer to another repositoryWhat a Git remote actually is on disk; the difference between the remote name, the remote URL, and the remote-tracking refs; how the four common URL schemes (https, ssh, git, file) affect authentication and transport.→
- Adding and removing remotes — the lifecycle of a remote entryHow to add a remote with git remote add, remove it with git remote remove, rename it with git remote rename, change its URL with git remote set-url, and constrain which branches are fetched with git remote set-branches.→
- Remote-tracking branches — the local cache of remote stateWhat refs/remotes/origin/main actually is; how git fetch writes it; why it is read-only; how it differs from a local branch; and how the namespace separates cached state from working state.→
- Upstream relationships — the per-branch link to a remoteHow branch.<name>.remote and branch.<name>.merge encode the upstream; the @{u} shorthand; setting with --set-upstream-to, removing with --unset-upstream, and the practical effects on push, pull, status, and @{u}.→
- Multiple remotes — origin, upstream, and forks in one cloneHow a single clone can talk to many remotes; the typical origin + upstream + personal-fork pattern; how git fetch --all and per-remote refs work; cross-repo workflows for open-source contributions and multi-environment deployments.→
- Remote pruning and cleanup — keeping the local cache honestHow git fetch --prune and git remote prune remove stale remote-tracking refs; what gets removed and what does not; how to inspect stale refs before pruning; and the safety mechanisms that prevent accidental data loss.→
Part XXI
Fetch vs Pull
6 checks
- What fetch does — downloading remote state into the local cacheWhat git fetch actually performs on disk; how it negotiates objects and refs with the remote; how it writes them into refs/remotes/<name>/; and why the local branches under refs/heads/ are never touched by a fetch alone.→
- What pull does — fetch plus merge (or fetch plus rebase)How git pull is a convenience wrapper around git fetch plus git merge; the two modes (merge and rebase); why the wrapper hides two distinct operations; when the convenience helps and when it hurts a team workflow.→
- git pull --rebase versus git pull --merge — linear history versus merge commitsThe two modes of git pull; how pull --rebase rewrites local commits on top of the fetched upstream; how pull --merge integrates them with a merge commit; when each is appropriate for an infrastructure workflow.→
- git pull --ff-only — refusing a pull that would require a mergeHow git pull --ff-only restricts a pull to fast-forward updates only; the failure mode when the remote has commits the local branch does not have; when ff-only is the right safety net for shared infrastructure branches and CI clones.→
- Configuring pull to rebase by default — pull.rebase and branch.<name>.rebaseHow pull.rebase true and branch.<name>.rebase true change the default of git pull for an entire repository or a single branch; the precedence between global, repo, and branch-level settings; the team implications of setting them.→
- The IaC and team pull policy — choosing one rule and enforcing itThe cost of letting every engineer choose their own pull strategy; the operational benefits of a single team-wide pull policy; the four questions a team must answer to set one; how the policy is encoded in repo config and verified in CI.→
Part XXII
Force Push
6 checks
- What force push does — overwriting the remote tip with your local tipWhat `git push --force` actually does on the wire; how the remote replaces its branch ref with your local one; which commits become unreachable and which downstream consumers break.→
- The destructive default — what `git push --force` actually destroysThe concrete loss when a force-push replaces the remote tip: commits on the remote that are not local, the teammates work built on those commits, and the audit trail that connected production to a specific change.→
- `git push --force-with-lease` — the safe force-pushHow `--force-with-lease` reads the expected remote tip from the remote-tracking ref, sends it to the server, and refuses the push if the remote has moved; why it is the safe default for solo feature-branch work.→
- The reflog as safety net — what is recoverable after a force-pushHow the local reflog and the server-side reflog together cover the post-force-push recovery window; what is recoverable inside the 90-day window; what is not recoverable after the window expires.→
- Branch protection and force-push — server-side enforcementHow branch protection rules on the remote reject force-pushes to protected branches; the difference between non-admin pushes, admin enforcement, and bypass actors; why force-pushing `main` is a structural impossibility in a protected repository.→
- Force-push incident response — when a teammate has rewritten your workThe incident-response procedure when a teammate has force-pushed a shared branch and broken local clones, CI caches, artifact pins, or signed tags; how to recover inside the reflog window; how to prevent the next occurrence.→
Part XXIII
Worktrees
6 checks
- What a worktree is — multiple working trees sharing one .git directoryWhat a Git worktree is; why one repository can have many working trees; how each working tree has its own HEAD and index but shares the object database with the others; when a worktree is the right tool for an infrastructure engineer.→
- Creating and removing worktrees — git worktree add, -b, --detach, --forceThe mechanics of creating a worktree (with or without a new branch, in detached HEAD state, or forcefully); the mechanics of removing one (clean or forceful); how each flag changes the on-disk state and what each flag is for in production.→
- Multiple worktrees and shared git — how .git/ is sharedHow multiple worktrees share a single .git directory; the .git/worktrees/$NAME/ metadata structure (HEAD, commondir, gitdir); what is and is not safe to share across worktrees; the per-worktree .git file that points back to the shared metadata.→
- Worktrees and branches — same branch in two worktrees; detached HEADWhy Git refuses to check the same branch out in two worktrees; what happens on detached HEAD in a worktree; how worktrees interact with branch protection and ref updates; the rule that worktrees give every branch its own working copy without duplicating history.→
- Infrastructure use cases — comparing IaC branches side by side; CI in a worktree; long-running checkoutsThe three production use cases for worktrees in infrastructure engineering: comparing two IaC branches side by side without context-switching loss; running CI builds in a worktree to reuse the object database; pinning a long-running checkout to a known commit for incidents, migrations, or slow rollouts.→
- Worktree cleanup and pruning — git worktree prune; stale metadata; the lock fileHow Git cleans up worktree metadata through `git worktree prune`; how stale metadata (a deleted worktree directory with no matching `git worktree remove`) is handled; how `git worktree lock` and `git worktree unlock` protect long-running checkouts; when each cleanup operation is appropriate.→
Part XXIV
Bisect
6 checks
- What bisect does — binary search through commit history for the offending commitWhat `git bisect` is for; why it exists; how it walks commit history using a bad/good marking protocol; the four phases of a bisect session; why it is the canonical tool for finding which commit introduced a regression.→
- The binary search mental model — O(log n) and the step counts for typical historiesWhy bisect is a binary search; what O(log n) means for the number of steps; the step count for histories from 100 to 100000 commits; why some sessions take more than the minimum number of steps; the cost of skip when a midpoint cannot be tested.→
- Automated bisect with `git bisect run` — exit codes, test scripts, and the shell wrapperHow `git bisect run <cmd>` automates the midpoint-by-midpoint test loop; the exit-code protocol that distinguishes good from bad; what makes a good test script; the `sh -c` form for multi-step scripts; when automated bisect is the right tool and when it is not.→
- Bisect log and visualize — recording, inspecting, and replaying a sessionWhat `git bisect log` records; how `git bisect visualize` shows the candidate set and the markings; how `git bisect replay <logfile>` re-runs a session from a saved log; when to use each; what to look for in the log when the conclusion seems wrong.→
- Bisect and build artefacts — finding the commit that broke the buildHow to bisect against build artefacts (binaries, container images, packages) by writing a test script that builds and asserts; the canonical build-as-test pattern; minimising the test script; bisecting binary artefacts when source-level tests are unavailable; how the pattern maps to a CI pipeline.→
- Bisect pitfalls and recovery — flaky tests, dependencies, side effects, and `bisect reset`The four failure modes of `git bisect run`: flaky tests, missing build dependencies, side-effecting test scripts, and dirty working trees; how each manifests as wrong conclusions or session aborts; the recovery procedures (`git bisect reset`, `git bisect replay`, log editing) that restore the session to a known state.→
Part XXV
Hooks
6 checks
- What Git hooks are — scripts Git invokes at lifecycle eventsWhat a Git hook is; how Git invokes a hook at a specific lifecycle point; the structural split between client-side hooks (developer machine) and server-side hooks (hosting server); why hooks exist as the standard integration point for policy, lint, and workflow glue.→
- Client-side hooks — where they live and how to enable themWhere client-side hooks live in the .git directory; how Git ships them as non-executing .sample files; how to enable a hook by removing the .sample suffix and chmod +x it; the full list of client-side hooks and the lifecycle point each fires at; the role of core.hooksPath for sharing hooks across clones.→
- Pre-commit and pre-push — the two most-used client-side hooksWhy pre-commit and pre-push are the two most-used client-side hooks; what pre-commit can block (format, lint, secret-scan, staged-file policy) by reading the index; what pre-push can block before the push leaves the machine; the difference between blocking on staged content (pre-commit) and blocking on the outgoing pack (pre-push).→
- Server-side hooks — pre-receive, update, post-receive, post-commitThe four server-side hooks (pre-receive, update, post-receive, post-commit), what each reads, and what each is used for; the role of post-receive as the canonical deployment hook; why hosted Git platforms (GitHub, GitLab, Bitbucket) usually disable custom server-side hooks and expose a hosted policy API instead; when a self-hosted bare repository is the right choice.→
- Hooks and policy enforcement — the limits of client-side controlWhy client-side hooks cannot enforce policy: --no-verify bypasses pre-commit and commit-msg, --no-verify (or -n) bypasses pre-push, and a client with hooks uninstalled has no enforcement at all. The structural difference between advisory client-side hooks and enforcement server-side hooks; the role of CI as the actual enforcement tier; the discipline of designing policies that survive client-side bypass.→
- Hooks and supply chain — secret scanning, dependency review, and the role of hooks versus CIThe supply-chain attacks a hook can catch (committed secrets in pre-commit, vulnerable dependencies in pre-push), the tools that implement those checks (gitleaks, trufflehog, npm audit, dependency-review-action), and the structural distinction between what hooks catch (developer-machine, fast feedback) and what CI catches (server-side, mandatory, broader scope).→
Part XXVI
Git Configuration
6 checks
- Config scopes — system, global, local, worktree, and where each value livesThe five scopes Git reads configuration from; the precedence order between scopes; where each scope writes on disk; how to inspect the merged view and the origin of every value.→
- User name and email identity — authorship, DCO, and why real identities matterWhat user.name and user.email do; why they are recorded in every commit object; how Signed-off-by lines and DCO enforcement depend on the email; the production discipline of a real, stable, professional identity.→
- Aliases — what they are, what they cost, and which ones to keepHow git config alias.* defines a shortcut that expands to a command; the most common aliases for status, log, diff, and branch; the productivity gain from aliases; the maintenance cost when aliases depend on flags or behaviour that change across Git versions.→
- Include and conditional configs — one identity per repo, one set of aliases per teamHow include.path pulls another config file into the merged view; how includeIf.<condition>.path applies a config only when a condition matches; the gitdir: condition for per-repository config and the hasconfig: condition for chained includes; the trade-off between conditional and per-repo local config.→
- Credential helpers and secure storage — what each one stores, where, and at what riskHow credential.helper works; the difference between the cache, store, osxkeychain, wincred, manager, and libsecret helpers; the security trade-off between plaintext storage and OS-provided secure storage; the production discipline of choosing the least-exposed helper for the deployment.→
- Signing configuration — keys, formats, and what production commits and tags should look likeHow user.signingkey, gpg.format, gpg.ssh.program, commit.gpgsign, and tag.gpgsign combine to produce signed commits and signed tags; the difference between openpgp, ssh, and x509 signing formats; the production discipline of signing every commit and every tag in production repositories.→
Part XXVII
Infrastructure Repository Architecture
6 checks
- IaC repository types — single-tool versus multi-tool repositoriesThe trade-offs between one repository per IaC tool and a single multi-tool repository; the operational signals that should force a split; the boundary rules for keeping shared code shared and tool-specific code isolated.→
- Terraform repository layout — modules, environments, and root modulesThe HashiCorp-recommended layout for a Terraform repository; the distinction between root modules, child modules, and reusable modules; what belongs at the repository root versus under a module directory; how the layout interacts with state and remote backends.→
- Ansible repository layout — roles, playbooks, inventories, and group_varsThe role-based organisation of an Ansible repository; the split between roles/, playbooks/, inventories/, group_vars/, and collections/; what belongs in a role versus a playbook; how the layout maps to Galaxy best practices.→
- Kubernetes repository layout — base plus per-environment, Kustomize versus HelmThe two layouts for a Kubernetes manifest repository (overlays versus directories); the base-plus-per-environment pattern; the trade-offs between Kustomize and Helm for templating and environment-specific configuration; what to commit and what to render.→
- Network and policy repositories — separate repositories for network and policy configurationWhy network configuration and policy configuration belong in repositories separate from the application and platform repositories; the security boundary the separate repository enforces; the review and approval pattern that matches the blast radius.→
- Documentation and runbook repositories — keeping docs in version control alongside codeWhy documentation and runbooks belong in a version-controlled repository alongside the code they describe; the layout for a docs repository; the cross-reference pattern that ties a runbook to a commit hash and an alert; the trade-off between co-located and separate docs repositories.→
Part XXVIII
Monorepo vs Multi-Repo
6 checks
- The trade-off — coupling versus independence, and what a single commit can affectThe fundamental trade-off between one repository and many: how much a single commit can change, who can see what, and the coordination boundary before it is set in stone.→
- Monorepo architecture — one repository, many projectsHow a monorepo physically and operationally holds many projects under one trunk; the build-system, ownership, and CI patterns that make it work; the real-world examples at Google, Facebook, and Microsoft.→
- Multi-repo architecture — one repository per service or projectHow a multi-repo physically and operationally separates each service or project into its own trunk; the ownership, release-cadence, and isolation patterns that make it work; the real-world examples at Netflix, Amazon, and smaller teams.→
- Hybrid and middleware — polyglot repos, submodules, partial clones, and sparse-checkoutThe hybrid patterns that sit between a pure monorepo and a pure multi-repo: submodules for versioned reuse, partial clones for bandwidth, sparse-checkout for working-tree size, and the operational rules that keep each pattern correct.→
- Ownership and access control — CODEOWNERS at the repository and directory levelHow CODEOWNERS scopes ownership in a monorepo versus a multi-repo; the blast radius of a misconfigured grant; the operational rules that keep ownership aligned with the repository boundary.→
- The decision criteria — team size, coupling, CI performance, blast radius, security boundaries, and tool supportA decision framework that maps the operational signals to the repository shape: team size, coupling, CI performance, blast radius, security boundaries, and tool support. The criteria are not aesthetic; they are forced by the operational state of the team and the codebase.→
Part XXIX
Branching Strategies
6 checks
- Trunk-based development — committing to the trunk every dayWhat trunk-based development means for an infrastructure team; how feature flags replace long-lived feature branches; the discipline that makes trunk-based safe.→
- Short-lived feature branches — small commits, frequent merges, and the rebase trade-offThe shape of a short-lived feature branch; the rebase-versus-merge trade-off for keeping it current; the cost of letting a branch live longer than a day.→
- Release branches — when stable releases matter and what maintenance costsWhat a release branch is; when LTS-style maintenance makes a release branch worthwhile; the cost of maintaining a release branch over time; the merge-back discipline.→
- GitFlow contextually — what it is, when it fits, and why it does not fit modern CIThe five branch types of GitFlow; the historical context in which GitFlow made sense; the modern CI assumptions GitFlow violates; the GitOps-specific reasons GitFlow does not fit.→
- Environment branches — branch per environment and why it is an anti-pattern in GitOpsWhat environment branches are; the historical reason they existed; why they are an anti-pattern in GitOps; the cases where they are still useful.→
- The team-policy decision — choosing, documenting, enforcing, and changing the branching strategyHow a team chooses a branching strategy that fits its release cadence and deployment model; how to document the choice; how to enforce it; how to change it when the situation changes.→
Part XXX
Pull Requests and Merge Requests
6 checks
- Pull request fundamentals — proposing, reviewing, mergingWhat a pull request is; the review-then-merge pattern; head ref and base ref; what the merge commit represents; why the PR is the unit of change in a GitOps team.→
- The diff and the review — what reviewers look at and why small PRs matterWhat the PR diff represents; the two-dot and three-dot diff; what a reviewer is checking for; the cognitive cost of large PRs; the discipline of keeping PRs small.→
- Approvals and required reviewers — the gate, CODEOWNERS, and the auditRequired approvals and how the count is set; CODEOWNERS and the ownership map; bypass actors and what they leave behind; the approval audit trail.→
- Status checks — the automated gate and the cost of flakinessWhat status checks are; required versus optional checks; the difference between CI checks, forge-side checks, and external integrations; the cost of flaky checks and how to retire them.→
- PR lifecycle and merge strategies — squash, merge commit, and rebaseThe PR lifecycle from draft to merged; the three merge strategies and what each does to history; the trade-off between readable history and atomic history; which strategy fits which repository policy.→
- PR quality and best practices — templates, checklists, and the review contractSmall PRs, descriptive titles, linked issues, the PR template; the review checklist that prevents review-by-rubber-stamp; the production discipline of treating the PR as a contract.→
Part XXXI
CODEOWNERS and Ownership Controls
6 checks
- What CODEOWNERS is — a file in the repo that maps paths to ownersA file in the repository that maps path patterns to owners; how GitHub, GitLab, and Bitbucket interpret it as a path-aware reviewer assignment.→
- Syntax and patterns — the line format, glob rules, and the order trapThe CODEOWNERS line format; glob semantics including leading-slash anchoring and double-star; comments and optional section headers; the rule that the last matching pattern wins.→
- Team and individual owners — @user, @org/team, and the security boundaryThe three forms of CODEOWNERS owners: individual users, org/team handles, and email addresses; why team membership is the security boundary that makes CODEOWNERS work.→
- CODEOWNERS and required reviews — how forges turn ownership into a gateHow platforms use CODEOWNERS to require reviewers per path; the wiring between the file and branch protection; how the requirement is satisfied in practice.→
- CODEOWNERS and bypass — who can override the rule and what is left behindHow bypass is configured on GitHub, GitLab, and Bitbucket; the bypass actors and the conditions under which they can override CODEOWNERS; the audit trail that bypass leaves behind; the limit of CODEOWNERS as a control.→
- CODEOWNERS as an operating system — composing ownership, branch protection, and status checksCODEOWNERS plus branch protection plus required status checks plus signing plus audit log equals the governance operating system of the repository; the composition is the control, not any single piece.→
Part XXXII
Protected Branches
6 checks
- What branch protection is — server-side controls on a branchBranch protection as a server-side rules engine on a named branch; what the server enforces, what clients cannot bypass, and the boundary between forge-side controls and client-side trust.→
- Direct push restrictions — disallow direct writes, require pull requestsHow to configure a protected branch so that all changes arrive via pull request; the difference between disallowing direct pushes and disallowing force pushes; the practical consequences for force-push-only-on-feature-branches and the recovery paths.→
- Required approvals and status checks — the merge-block conditionsHow approval minimums, dismissal of stale reviews, CODEOWNERS-required reviews, and required status checks work together as the merge-gating stack; configuration on the major forges; what each rule refuses and what each rule does not refuse.→
- Bypass and bypass actors — who can override the rule and what is left behindThe role-based and explicit-list-based bypass mechanisms in GitHub, GitLab, and Bitbucket; the conditions under which bypass is permitted; the audit trail that bypass leaves behind; the limit of bypass and the role of the bypass event in incident response.→
- Tag protection — preventing tag deletion, restricting tag creation, enforcing signed tagsHow the forge protects tags: refusing deletions, restricting who can create or move a tag, and the signing requirement that ties a tag to a key; the relationship with signed-tag enforcement and the operational consequences for release artefacts.→
- Protected branches and policy — branch protection as the enforcement layerBranch protection as the place where policy becomes enforceable: the relationship between the rule set, CODEOWNERS, signed commits, audit logs, and the team policy; the discipline of codifying branch protection; the limits of the layer and what it cannot replace.→
Part XXXIII
Commit and Tag Signing
6 checks
- Why sign commits — the chain of trust and what signing actually provesWhat commit and tag signing extend over unsigned commits; the chain of trust from commit hash to cryptographic identity; what a signature proves (identity assertion at a moment in time) and what it does not (intent, review, code quality, key custody guarantees).→
- GPG signing setup — generating a key, configuring Git, signing with -SHow to generate a GPG key for commit signing with gpg --gen-key, export the public key with gpg --armor --export, point user.signingkey at the key, sign commits with git commit -S, and skip hooks with --no-verify while keeping the signature intact.→
- SSH signing setup — the Git 2.34+ approach using an existing SSH keyHow to enable SSH commit signing with gpg.format ssh, point user.signingkey at an SSH public key path, generate a dedicated signing key with ssh-keygen -t ed25519, configure gpg.ssh.program, and verify SSH-signed commits against an allowedSigners file.→
- Signing tags versus commits — both can be signed; tags survive rebasesThe operational difference between signing commits and signing tags; why tags are more useful for release identity (they are not rebased); the gpgsig block on tags versus commit objects; combining signed commits on a branch with a signed tag at the release point.→
- Verifying signatures — git verify-commit, git verify-tag, --show-signature, and forge UIsHow to verify commit and tag signatures locally with git verify-commit, git verify-tag, git log --show-signature, and git log --pretty=%G?; how to interpret the G/B/U/X/Y/R/E/N status codes; how the GitHub and GitLab UIs render verified commits; how to wire verification into CI.→
- Signing policy and enforcement — branch protection, required signed commits, and the failure modesHow branch protection enforces signed commits as a precondition for merging; the failure modes when engineers bypass signing; the policy decisions around commit.gpgsign enforcement, allowed signers file maintenance, and key rotation; the integration with CODEOWNERS and required reviewers.→
Part XXXIV
Git Security
6 checks
- Authentication options — SSH vs HTTPS and the trade-offs that decideThe two transport-and-authentication options Git offers (SSH and HTTPS); the operational trade-offs between key-based and token-based authentication; when SSH is the right default, when HTTPS is the right default, and how to decide between them for a workstation, a runner, and a deploy host.→
- SSH keys and deploy keys — personal keys, deploy keys, host keys, and the read-only distinctionHow SSH authentication works for Git; the three classes of SSH key (personal account keys, deploy keys, and host keys); the read-only vs read/write distinction for deploy keys; how to generate, install, and rotate each kind; the failure modes when the wrong key class is used.→
- HTTPS tokens and personal access tokens — PATs, fine-grained tokens, and OAuth appsThe three HTTPS-credential models Git forges offer (classic PATs, fine-grained PATs, and OAuth apps); how each scopes the token to repositories and permissions; the expiry and rotation properties of each; the production discipline of scoping every token to the narrowest permissions its use case requires.→
- Credential storage and rotation — where the credential lives, how often it changes, and the leak surfaceHow Git credentials are stored once they reach the client; the credential-helper architecture (cache, store, osxkeychain, wincred, manager, libsecret); the rotation cadence that limits the window of exposure for any credential that has ever been written to disk; the leak surface of plaintext storage, CI logs, shell history, and backup snapshots.→
- The compromised account — what happens, what to do first, and how to contain the blastThe operational playbook for a compromised Git account, SSH key, or HTTPS token; the first-hour containment steps; the forensic steps that determine what the attacker did; the remediation steps that close the door; the structural changes that prevent recurrence.→
- The least-privilege credential — scoping tokens to minimum scope, ephemeral credentials, and the principleThe principle of least privilege applied to Git credentials; how to scope a token to one repository, one permission, and the shortest useful lifetime; ephemeral credentials (per-job tokens, short-lived OAuth tokens) and why they are the strongest form of the principle; the operational discipline of issuing narrow credentials on demand rather than broad credentials in advance.→
Part XXXV
Secrets in Git
6 checks
- The secret leak fallacy — why deleting the file does not delete the secretThe mental-model error that makes engineers believe a secret is safe because the file is no longer in the working tree; how secrets persist in Git history, pack files, reflogs, and forks; why removal is forensic, not preventive.→
- Detection with secret scanning — gitleaks, truffleHog, and the CI gateThe detector landscape for secrets in Git: gitleaks, truffleHog, GitHub native secret scanning, GitLab native secret scanning; the CI gate that blocks the merge; the false-positive and entropy trade-offs; the pre-commit and pre-receive positions.→
- The rotation response — rotate first, then clean history, the order that survives the incidentThe operational playbook for a confirmed secret leak: the rotation, the revocation, the consumer update, the history rewrite, the audit log. The order is enforced by the physics of the situation; the cost of getting the order wrong is the answer to the wrong question.→
- History cleanup tools — git filter-repo, BFG, and the limits of the rewriteThe rewriting tools for Git history: git filter-repo (the modern replacement for filter-branch), BFG Repo-Cleaner (the fast credential scrubber), the replace-text and invert-paths modes, the reflog expiry and gc pattern, and the hard limits of the rewrite (forks, clones, backups, archives).→
- Forks, clones, and mirrors — the persistence of leaked secrets and the impossibility of perfect recallThe second-order leak channels: every fork, every clone, every mirror, every CI cache, every backup that has the original history. Why the rewrite cannot reach them; the practical impossibility of perfect recall; the operational response when recall is impossible; the posture that prevents the leak from happening in the first place.→
- Prevention by design — pre-commit hooks, CI gates, and secrets that never reach the repositoryThe structural changes that prevent the next secret leak: pre-commit hooks that block the commit, CI gates that block the merge, secret managers that never put the secret in a file, environment variables that never reach the working tree, and the layered posture that keeps the secret out of the team trust boundary.→
Part XXXVI
Git History Rewriting
6 checks
- git filter-repo — the modern replacement for git filter-branchWhy git filter-repo replaced git filter-branch; the four modes that matter (--invert-paths, --replace-text, callbacks, --mailmap); the safety contract (fresh clone, reflog expire, gc, --force-with-lease).→
- BFG Repo-Cleaner — the fast credential scrubberWhat BFG Repo-Cleaner is (a Java tool focused on the credential-scrub use case); the --replace-text, --delete-files, --strip-blobs-bigger-than modes; the trade-off (blob-only, no callbacks, faster than filter-repo on large repos).→
- Removing files from history — when a path must not existWhen a file must be removed from history (accidentally-committed binary blob, large artefact, credential file, internal document); the exact procedure with git filter-repo --invert-paths; verification before the force-push; the channels the path-removal does not reach.→
- Replacing content — when a string must change across historyWhen text must be replaced across history (email address, sensitive value, name, organisation); the exact procedure with git filter-repo --replace-text and --name-callback; the process-substitution one-shot pattern; verification before the force-push; the difference between --replace-text and BFG --replace-text.→
- Re-signing after rewrite — restoring the chain of trustWhy every GPG/SSH signature becomes invalid after a history rewrite (the commit bytes changed, the signature no longer matches); the two re-signing procedures (filter-repo --re-sign with --gpg-key, or git rebase --exec git commit --amend --no-edit -S); the verification step.→
- The force-push aftermath — communication, coordination, auditWhat the team must do after a history-rewrite force-push: pre-push communication (announce the window, freeze the affected branches, prepare the recovery message); post-push coordination (notify every clone owner, distribute the recovery commands, handle concurrent pushes); the audit step (forks, clones, mirrors, CI caches, backups).→
Part XXXVII
CI Fundamentals
6 checks
- What CI is and is not — automation, not gatekeepingWhat continuous integration actually is; the misconception that CI guarantees change safety; the line between automation (CI) and policy enforcement (branch protection, signed commits, OPA); what CI does not replace.→
- The trigger to result pipeline — what fires a CI runWhat triggers a CI run (push, pull request, tag, schedule, manual); how the trigger becomes a runner checkout; how the result is published back to the commit; the four canonical trigger categories.→
- The runner and its environment — hosted, self-hosted, ephemeral, persistentWhat a CI runner is; the difference between hosted and self-hosted runners; ephemeral versus persistent runners; the trust boundary the runner represents; how to choose for an infrastructure team.→
- Jobs and steps — the units of work and how they relateWhat a job is in a CI pipeline; the difference between jobs and steps; how jobs relate to each other (parallel, sequential, dependent); designing the job graph for an infrastructure pipeline.→
- Status checkout — what gets reported back, and who reads itWhat a commit status is; which systems consume it (branch protection, GitOps controllers, PR decoration, downstream pipelines); how the status is published against the SHA; the canonical status states.→
- CI versus the developer laptop — why "it works on my machine" is the bugWhy the developer laptop is not a sufficient execution environment; what drifts between laptop and CI; how CI provides reproducibility, isolation, and auditability the laptop cannot; the laptop-to-CI handoff.→
Part XXXVIII
CI Architecture
6 checks
- The three-plane model — control plane, runner, and environmentThe three planes every CI run touches; the trust boundaries between them; what crosses each boundary; why understanding the planes is the prerequisite for every architectural decision in CI.→
- Control plane isolation — why orchestration is separate from executionWhy the control plane is architecturally separate from the runner; what that separation buys in security; what it costs in operational complexity; how OIDC, short-lived tokens, and ephemeral runners extend the isolation.→
- Runner network and internet — egress controls, private runners, and the cloud-only caseWhat egress means for a CI runner; how to control outbound network access; private runners in private networks; the cloud-only case where runners have no internet at all; why network egress is the highest-leverage CI security control.→
- Jobs and concurrency — concurrency groups, cancel-in-progress, and queue limitsHow CI systems schedule jobs against a runner pool; what concurrency groups do; what cancel-in-progress means and when to use it; queue limits and what happens when a pool is exhausted.→
- Environment variables and config — secrets, env, vars, and the scopes that bind themThe three configuration mechanisms in a CI job: secrets, env, and vars; the scopes each mechanism lives in; how the runner passes values between steps; the boundary between GITHUB_ENV and GITHUB_OUTPUT.→
- Artifacts, caches, and outputs — three distinct mechanisms for moving data through a jobThe three mechanisms for moving data in a CI job: artifacts (durable, stored, downloadable), caches (reusable, key-addressable, best-effort), and outputs (per-job, structured, ephemeral); the differences and when each fits an infrastructure pipeline.→
Part XXXIX
Pipelines
6 checks
- Pipeline as code — the workflow file is committedWhy a CI/CD pipeline is a versioned file in the repository; what the workflow-as-code model gives an infrastructure team; the costs it imposes; where it is and is not the right tool.→
- Stages and jobs — the two-level hierarchy and when each model appliesWhat a stage is and what a job is in GitHub Actions versus GitLab CI; the two-level hierarchy; when a flat jobs model is correct; when stages are required; the trade-off between DAG flexibility and grouping.→
- Job dependencies and DAG — needs, requires, dependencies, fan-in, fan-outHow to express the dependency graph of a CI pipeline; what fan-in and fan-out mean; how the DAG is validated against cycles; how the system chooses the next job to start; the difference between explicit and implicit dependencies.→
- Parallel execution and fan-out — matrix builds, the use case, and the costHow matrix builds parallelise a pipeline across an axis of variables; the cost in runner minutes, API calls, and log noise; when a matrix is the right tool and when it is over-engineering.→
- Pipeline status and observability — what the run page shows and how to read the logsHow a CI run is presented: the run page anatomy, status badges, job list, step logs, artifact list, and timing; what the log lines mean; how to correlate log timestamps with runner wall-clock time; the four signals that predict a problem before it fails.→
- Reusable workflows — DRY at the workflow level, what they enable, what they costHow reusable workflows extract a pipeline into a callable unit; the workflow_call trigger; inputs, secrets, and outputs; the four patterns reusable workflows replace; the cost in coupling, debugging, and review surface.→
Part XL
Runners
6 checks
- Hosted runners — convenience, isolation, and the control you give upWhat GitHub-hosted runners are; what they install by default; how they are isolated per job; what you do not control on them; when to use them and when not to.→
- Self-hosted runners — control, operational cost, and the security costWhat a self-hosted runner is; the operational responsibility it carries; the persistent-host security model; how to register, configure, and run a self-hosted runner with the official config.sh and run.sh scripts.→
- Ephemeral runners — clean state every job, and the cost of throwing away stateWhat ephemeral runners are; the safety they provide; the cost of rebuilding per job; the patterns that make ephemeral runners practical: just-in-time provisioning, immutable images, job-scoped secrets.→
- Runner autoscaling and Actions Runner Controller — scaling on KubernetesWhat autoscaling means for CI runners; how Actions Runner Controller (ARC) scales runner pods on Kubernetes based on job queue depth; minRunners, maxRunners, and scale-down behaviour; the controller-listener pattern.→
- Runner labels and selection — matching workflows to runners, and the trust modelHow runner labels work in GitHub Actions; how workflows select runners with runs-on; how labels combine to form a routing language; the trust model that labels express and the risks of misconfiguration.→
- Runner tooling and actions — what is installed, GitHub-managed actions, and the marketplaceWhat the runner image provides; what GitHub-managed actions do and why they are safer than marketplace actions; how to pin actions by SHA; the risk model of third-party actions; the build vs buy decision for actions.→
Part XLI
Runner Security
6 checks
- The runner threat model — who attacks, how, and with what accessWho attacks the CI runner, what access each attacker class starts with, what they can reach from the runner, and how the attack tree from initial access to production compromise is structured. The discipline is to write the threat model before the controls, not after.→
- Arbitrary code execution — every CI run executes attacker-controlled code if the trigger is a PR from outsideWhy a pull_request trigger from a fork or first-time contributor is untrusted code execution; what a malicious workflow step can do to the runner filesystem, environment, and network; the structural separation between fork PRs and trusted PRs in GitHub Actions, GitLab CI, and Jenkins.→
- Production credentials on runners — why long-lived credentials on shared runners are catastrophicWhy static long-lived credentials stored on a shared CI runner are the worst-case secret posture: the credential lifetime is the runner lifetime, the credential scope is whatever the runner can reach, and a single compromised job discloses every credential the runner holds. The structural fix is OIDC short-lived federation.→
- The Docker socket risk — docker.sock mounted into a runner = root on the host; the escalation pathWhy /var/run/docker.sock mounted into a runner container is host-equivalent root; the escalation path from arbitrary code execution to host compromise via the daemon API; the audit commands that detect the exposure; the alternatives (rootless daemon, socket proxy, DinD on a separate host) and their trade-offs.→
- Privileged containers and host mounts — what privileged means; the kernel surfaceWhat --privileged actually grants (capabilities, devices, seccomp, AppArmor, /proc and /sys), the kernel surface each exposes, why --privileged --cap-drop=ALL is a contradiction, and the host-mount patterns that are equivalent to --privileged: docker.sock, /, /proc, /sys, /var/lib/docker.→
- Persistence and lateral movement — what happens after the initial compromise; how to containWhat an attacker does after the initial runner compromise: persistence (cron, systemd units, SSH keys, IAM role assumption), lateral movement (cluster access, internal network pivot, cloud API access via OIDC token replay); the containment procedure (isolate, audit, revoke, rebuild); the structural changes that prevent recurrence.→
Part XLII
CI Secrets
6 checks
- Secret variables fundamentals — how CI platforms store and serve secretsHow CI forges store secrets at rest, how the runtime injects them into the job, and why the secret variable is the smallest unit of trust in a CI pipeline. The encrypted-at-rest model, the access pattern, and the boundary between secret storage and secret usage.→
- Masking and its limits — what the log masker catches and what it does notHow the CI log masker identifies secret values and replaces them with asterisks; what the matcher pattern can and cannot catch; the production consequences of trusting masking as a control.→
- Log leakage risks — failure modes that bypass the maskerThe concrete failure modes that put a secret into a log the team reads; the side channels the masker does not cover; the production consequences of a leaked credential that the team thought was masked.→
- Environment scopes — repository, environment, and organisation granularityHow CI forges partition secrets by scope; the rules for repository, environment, and organisation-level secrets; how environment protection rules and required reviewers gate a deploy job to a specific identity.→
- Secret rotation cadence — when to rotate, what triggers rotation, and the disciplineThe rotation cadence for a long-lived CI secret; the triggers that force an out-of-cycle rotation; the operational discipline that keeps rotation predictable rather than reactive.→
- The short-lived credential ideal — OIDC, dynamic secrets, and the end of long-lived keysWhy long-lived CI credentials are a structural smell; how OIDC federation and dynamic secrets from HashiCorp Vault replace them; the operational rule that a credential lifetime must be no longer than the workload lifetime.→
Part XLIII
OIDC and Short-Lived Credentials
6 checks
- Why long-lived credentials fail — rotation cost, leak surface, blast radiusThree structural failures of long-lived CI credentials — the rotation cost compounds over time, the leak surface grows with the credential lifetime, and the blast radius of a leak is whatever IAM grants. Static keys fail by construction; OIDC succeeds by construction.→
- OIDC federation basics — what OIDC is; the trust relationship between CI and cloudOpenID Connect is a signed JWT identity layer. In CI/CD, the forge (GitHub Actions, GitLab CI) issues an OIDC ID token per job; the cloud validates the token against a trust policy and exchanges it for a short-lived STS session. The trust relationship is two configurations that must agree.→
- GitHub Actions OIDC in practice — id-token: write permission; the JWT issuanceHow a GitHub Actions workflow requests an OIDC ID token, what the token contains, and how the token is exchanged for a cloud credential. The id-token: write permission is required; the token is bound to the workflow, repository, branch, and job; the cloud validates and exchanges.→
- OIDC in AWS — provider, audience, role, trust policy; aws-actions/configure-aws-credentialsConfiguring OIDC federation in AWS: register the GitHub OIDC provider, create an IAM role with a trust policy that gates repository, branch, and workflow path, and use aws-actions/configure-aws-credentials@v4 to exchange the OIDC token for a short-lived STS session. The four pieces — provider, audience, role, trust policy — must agree.→
- OIDC in Azure and GCP — Workload Identity Federation; the federation poolConfiguring OIDC federation in Azure and GCP. Azure uses Workload Identity Federation with a federated credential on an app registration. GCP uses Workload Identity Federation with a workload identity pool and provider. Both differ from AWS in the trust relationship model but converge on the same short-lived credential outcome.→
- OIDC trust policy deep dive — sub claims, job_workflow_ref claims, the immutable subject claims changeThe anatomy of an OIDC trust policy: the sub claim gates repository + branch + ref, the job_workflow_ref claim gates workflow file path, and GitHub made the sub claim format immutable on November 2021 — workflows using the legacy format permanently fail. Reading the claims is the prerequisite to writing the policy.→
Part XLIV
Artifacts
6 checks
- What an artifact is — the output of a job, named and stored for laterThe artifact as the durable output of a CI job: a name and a path, uploaded to the artifact store, downloadable by other jobs in the same workflow and by humans with workflow read permission. The storage model and the audit invariant.→
- Build outputs and binary artefacts — what gets uploaded, the upload limits, the retentionBuild outputs as artifacts: compiled binaries, packed archives, signed bundles. The upload size limits per artifact and per run, the retention window, and the production rules for binary artifacts — verify the digest, set the retention, and never rebuild a binary from source if the binary is the artifact.→
- Terraform plans as artifacts — the plan file as a review surface, plans across jobsThe Terraform plan file as a review-grade artifact: produced in the plan job, uploaded with actions/upload-artifact@v4, downloaded by the apply job with actions/download-artifact@v4, reviewed by humans in the PR UI behind a comment that links the run. The plan is the contract; the artifact is the binding.→
- Reports and junit — test results, coverage, scan results as structured artifactsStructured reports as artifacts: JUnit XML for test results, Cobertura or LCOV for coverage, SARIF for static analysis, JSON for inventory. The artifact format is the contract between the tool that produced the report and the consumer that aggregates it. The PR UI surfaces pass/fail counts and failed test names.→
- Images and manifests — OCI image artifacts, manifest artifacts, the production patternOCI images and Kubernetes manifests as artifacts: the image built and pushed to a registry, the manifest rendered and uploaded as a YAML bundle, the deployment job downloading the manifest and applying it. The artifact is the image digest; the manifest is the contract between the image and the cluster.→
- Artifact retention and storage — retention policies, the cost, the production rulesRetention policies as a cost/audit tradeoff: short retention saves storage but loses audit; long retention preserves audit but inflates cost. The four production rules: set retention explicitly, archive to long-term storage for compliance windows, periodically audit storage usage, and never store secrets in artifacts.→
Part XLV
Artifact Immutability
6 checks
- The immutable identity principle — every artifact has a content-addressed identityWhy immutability is the foundational property of an artifact: the same bytes always yield the same identity, the identity survives relocation and renaming, and only a content-addressed identity can support promotion, rollback, and audit.→
- Digests and content-addressing — sha256:abc..., the artifact’s true nameWhat a digest is, how the registry uses it to store and retrieve content, why the manifest digest is what production deployments pin to, and how to read it with crane.→
- Promotion across environments — same artifact, different environmentsPromotion is moving the same artifact from one environment to the next without rebuilding it. The bytes that survive staging are the bytes that survive production; the digest that passed tests is the digest that runs.→
- The build versus rebuild trap — why rebuilding per environment destroys provenanceThe assumption that rebuilding an artifact per environment produces an equivalent artifact is wrong: same source does not mean same bytes. Toolchain drift, OS package versions, and timestamp layers make each rebuild a different artifact with a different digest.→
- Immutable tags and digest pinning — why :latest is a contract that resolves to whoever pushes lastTags are mutable pointers; digests are immutable identities. The :latest tag resolves to whoever pushed last; the :v3.2.7 tag is reassignable too. Digest pinning via @sha256:... makes the deployment immune to tag reassignment.→
- Provenance and the build identity — SLSA, attestations, and the chain from source to bytesProvenance is the signed attestation that records who built an artifact, from what source, with what tools. SLSA defines the levels; the build identity is the part that names the pipeline, the commit, and the builder.→
Part XLVI
Caching
6 checks
- What a cache is — a content-addressed, key-matched, best-effort blob storeA cache is a key-addressed blob store the runner reads and writes to skip redundant work. The key is derived from a hash of the inputs (typically the lockfile), the scope is the repository and branch, and the lifetime is best-effort by design.→
- Cache versus artifact — best-effort acceleration versus durable recordThe cache and the artifact solve different problems: the cache is a best-effort accelerator that may be evicted at any time; the artifact is a durable, retention-bounded, auditable record. Mixing them is the source of silent failures and unbounded storage cost.→
- actions/cache@v4 — inputs, behaviour, and a real workflowThe `actions/cache@v4` step is the standard mechanism for caching dependencies in GitHub Actions. Its three required inputs - path, key, restore-keys - determine what is cached, how it is addressed, and how partial-match fallback works.→
- Cache keys and restore keys — exact match, prefix fallback, partial reuseThe exact-match key and the prefix-match restore-keys are two different lookup mechanisms. The exact-match key is the identity of the cache entry; the restore-keys are an ordered list of fallbacks the runner tries when the exact key is absent. Used together, they let a workflow reuse a near-match entry when the lockfile has changed slightly.→
- Cache poisoning and staleness — the failure modes of trusting a cacheA cache is best-effort and trustable only to the extent the cache key is unique. Staleness is a correctness failure: the lockfile has changed but the key has not, and the runner restores outdated bytes. Poisoning is a security failure: an attacker pre-populates the cache with malicious content the workflow restores.→
- Cache cost and retention — what caches cost, how they are evicted, and when to delete themCaches have a per-repository size limit (10 GB), are evicted under LRU pressure, and can be deleted explicitly with `gh cache delete`. The retention policy is not user-controllable; the cost is included in GitHub Actions pricing for private repositories and is free for public repositories.→
Part XLVII
Pipeline Dependencies
6 checks
- The pipeline as a graph — why DAG, not stages, is the right mental modelA pipeline is a directed acyclic graph of jobs, not a linear sequence of stages. The graph is what the scheduler walks; every edge is an ordering guarantee, every missing edge is an opportunity for parallelism.→
- needs and depends-on — declaring edges between jobsThe `needs:` and `dependencies:` clauses declare edges between jobs. The listed jobs must complete before the current job starts; the list can be a single job or many; the default semantics is to wait for successful completion. Both GitHub Actions and GitLab CI use the same concept with slightly different syntax.→
- Fan-in and fan-out — parallelism at job boundariesFan-out is one job whose completion enables many parallel successors. Fan-in is many predecessors whose completion enables one successor. The diamond shape is fan-out followed by fan-in. These three shapes are the vocabulary for reading and writing pipeline DAGs.→
- Pipeline triggers and chains — workflow_run and workflow_callPipelines can chain across workflows. `on.workflow_run` triggers a workflow after another workflow completes; `on.workflow_call` makes a workflow callable as a reusable step. The two mechanisms are inter-workflow edges; they complement intra-workflow `needs:`.→
- Outputs as inputs — the typed contract between jobsJobs can declare outputs and pass them to dependents via `needs.<job_id>.outputs.<name>`. Outputs are typed strings, scoped to a single run, and lost at run end. They are the typed contract between jobs; artifacts are the durable contract.→
- Pipeline failure propagation — what happens when a dependency failsWhen a `needs:` predecessor fails, the dependent job is skipped by default. The overrides are `if: always()`, `if: failure()`, `if: success()`, and `continue-on-error`. Designing failure propagation is part of pipeline design; the default is not always the right choice.→
Part XLVIII
Conditional Execution
6 checks
- Conditional fundamentals — when to skip, when to run, and the cost of always-runningA conditional is a guard that decides whether a job or step runs. Guards prevent wasted compute, prevent accidental production actions, and prevent noisy logs. The cost of running unconditionally is real and cumulative.→
- Branch and path filters — narrowing triggers by branch name and changed filesBranch filters and path filters are the two most common guards in any non-trivial pipeline. They decide whether an event is worth running for, before any expression evaluation or runner allocation happens.→
- Tag and environment conditions — controlling which events touch which environmentsTag triggers release events. Environment names identify deployment targets. A tag or environment condition is the boundary between "this is a build event" and "this is a deployment event". The condition is what stops the second from running when the first was intended.→
- Expressions and context — `${ }`, `github.event`, `github.ref`, and the operatorsThe `${ }` syntax evaluates an expression against a context object. The context is the live state of the run - the trigger, the actor, the ref, the event payload. The operators are a small DSL; the discipline is to keep expressions readable and to test them before relying on them.→
- Matrix strategies — when a matrix is right, when it is overused, and the cardinality costA matrix strategy expands one job into many parallel jobs. The expansion is parameterised by a set of variables. The discipline is to keep the matrix small, the variables orthogonal, and the cardinality bounded by runner cost and signal-to-noise.→
- Environment protection rules — required reviewers, wait timers, and branch restrictionsProtection rules are platform-side guards attached to a named environment. They evaluate after the workflow has decided to run, but before the deployment job starts. Required reviewers, wait timers, and branch restrictions are the three rules that prevent a deploy from racing past the team.→
Part XLIX
Infrastructure CI
6 checks
- The infrastructure pipeline pattern — eight stages from commit to auditWhy infrastructure CI is a multi-stage pipeline and not a single job; the eight canonical stages (format, lint, security, tests, plan, review, apply, audit); what each catches and what it costs.→
- Format stage — terraform fmt, ansible-lint format, kubeconformWhat the format stage is for; why whitespace is a pipeline concern; the format tools for Terraform, Ansible, and Kubernetes; what format catches and what it deliberately ignores.→
- Lint and static analysis — tflint, ansible-lint, kubeconform, conftest, OPAHow the lint stage differs from the format stage; the canonical linters for Terraform, Ansible, and Kubernetes; how conftest and OPA extend lint to policy; what lint catches and where its blind spots are.→
- Security scanning — tfsec, checkov, trivy, kics, snykWhy security scanning is a separate stage; the canonical scanners for Terraform, Kubernetes, container images, and dependencies; what scanners catch and how their rule sets differ; the limits of static security analysis.→
- Test and validate — terratest, Molecule, kyverno tests and the cost of testingWhy testing is a separate stage from plan; the canonical test frameworks for Terraform, Ansible, and Kubernetes; the cost of tests in time and cloud spend; what unit tests can and cannot prove.→
- Plan and review — terraform plan as the review artefact; the comment-on-PR patternWhy terraform plan is the review artefact; how to capture a plan as a binary and JSON for posting; the comment-on-PR pattern for surfacing plans on pull requests; why the plan that is applied must be the plan that was reviewed.→
Part L
Terraform CI
6 checks
- The Terraform CI discipline — why plan-on-PR is the right patternWhy the pull request is the right place to run terraform plan; what runs in CI versus what must wait for a human; the boundary between plan and apply; why apply-on-PR is the wrong default for production infrastructure.→
- fmt and validate — what Terraform built-ins catch and what they do notWhat terraform fmt checks (whitespace, block formatting) and what it does not (semantics); what terraform validate checks (HCL syntax, internal references, provider schemas) and what it does not (variable values, external behaviour, runtime correctness).→
- tflint and fmt deep — the .tflint.hcl ruleset and what fmt should have caughtHow tflint differs from terraform fmt and validate; the .tflint.hcl configuration file; selecting rulesets per provider; running tflint --init to download rulesets; running tflint --recursive across a monorepo; what deep formatting rules the canonical fmt style leaves undecided.→
- tfsec and checkov — what security scanners actually check, and the false positive rateHow tfsec and checkov approach Terraform security scanning; the categories of finding each tool produces; the false-positive rate and how to triage it; why security scanners are necessary but never sufficient; the difference between static policy and runtime reality.→
- Terratest and integration tests — Go-based testing of real infrastructureHow Terratest executes real Terraform applies against a real cloud account to verify behaviour; the cost of integration tests in money and time; what to test and what to mock; the Go-based test harness; the defer-and-destroy cleanup pattern.→
- Plan as artifact and PR comment — saving the plan, posting it, and the review contractHow to capture terraform plan as a binary artifact; how to convert it to JSON for parsing; how to post the plan as a pull-request comment; why the plan that is applied must be the plan that was reviewed; the consequences of re-planning between review and apply.→
Part LI
Ansible CI
6 checks
- The Ansible CI discipline — what "tested" means for AnsibleWhy "tested" is a claim that needs unpacking for Ansible code; the difference between static checks (lint, syntax) and runtime checks (check mode, Molecule); the boundary between a passing CI run and a safe apply.→
- YAML and playbook linting — yamllint and ansible-lint as the first gateWhy YAML syntax is the first thing to break in an Ansible change; what yamllint catches (indentation, quoting, truthy values, document structure); what ansible-lint catches (task shape, FQCN, naming); how to choose a ruleset that fits the team rather than the tool defaults.→
- ansible-lint and FQCN rules — fully qualified collection names as a conventionWhy bare module names are ambiguous in modern Ansible; what a fully qualified collection name (FQCN) looks like; the ansible.builtin.* and community.* namespaces; how ansible-lint enforces FQCN with the fqcn rule family; the migration path from bare names.→
- Ansible playbook syntax check — what --syntax-check proves and what it does notHow ansible-playbook --syntax-check parses a playbook against the local collection cache; what it catches (YAML parse errors, undefined variables, missing modules, Jinja errors) and what it does not (task success, idempotency, runtime behaviour); why it is the cheapest post-lint gate before runtime checks.→
- Molecule and integration testing — scenarios, drivers, and the verify stageWhy Molecule is the canonical Ansible integration test framework; what a scenario is; the default scenario lifecycle (dependency, lint, cleanup, destroy, side-effect, syntax, converge, idempotence, verify, cleanup, destroy); drivers (delegated, docker, podman, vagrant) and what they prove; testinfra and ansible.builtin.assert as the verify layer.→
- Staged validation and idempotency — check mode, diff mode, and the production patternWhy ansible-playbook --check produces a plan and ansible-playbook --diff produces a change list; how to combine the two for the production validation pattern; why idempotency is the property that makes repeated applies safe; how the staged pipeline (lint, syntax, --check, Molecule, apply) makes the apply step a routine execution rather than a leap of faith.→
Part LII
Kubernetes CI
6 checks
- The Kubernetes CI discipline — render, validate, packageWhy Kubernetes CI is a three-stage pipeline; what each stage catches that the others miss; the boundary between manifest validation, template validation, and package publication; why a render artifact is the right handoff to GitOps.→
- Manifest validation with kubeconform — schemas, strict mode, version pinningWhat kubeconform checks against the Kubernetes OpenAPI schema; the difference between strict mode and summary mode; how to pin the schema version so the check is reproducible; what kubeconform catches that helm lint does not.→
- Helm and Kustomize validation — lint, template, and buildWhat helm lint checks; what helm template --validate catches that lint misses; what kustomize build verifies; the difference between source-time validation and render-time validation; how the two tools compose with kubeconform.→
- Policy with Conftest and Kyverno — Rego and CEL, and what policy catchesWhat policy validation catches that schema validation cannot; the difference between Rego (OPA, Conftest) and CEL (Kyverno); how to write a policy that fails on missing labels, forbidden images, or wildcard RBAC; the boundary between CI policy and admission-time policy.→
- Security scanning with Trivy and Kubescape — vulnerabilities and misconfigurationsWhat Trivy catches (CVE in images, misconfigurations in manifests, vulnerable dependencies); what Kubescape catches (CIS benchmarks, NSA hardening, MITRE ATT&CK); the difference between vulnerability scanning and misconfiguration scanning; how the two compose with kubeconform and policy.→
- Render and package as OCI — the artifact handoff to the GitOps controllerWhy the package stage produces an OCI artifact rather than raw YAML; how helm package pushes to an OCI registry; how Kustomize output is wrapped into an OCI artifact; the role of cosign signatures; how the GitOps controller pulls by digest.→
Part LIII
Container CI
6 checks
- The container supply chain — source to registryWhat every step between a commit and a registry tag introduces; why the chain is the unit of trust for a container artifact.→
- BuildKit and the build cacheHow BuildKit turns a Dockerfile into a content-addressable layer graph; how the build cache works; why cache mounts and registry caches change the cost of a CI build.→
- Multi-stage builds and distroless imagesHow multi-stage Dockerfiles shrink the final image; what distroless removes; the security argument that motivates both.→
- Image testing in CIWhat to test about a container image before tagging it: vulnerability, size, structure, and behaviour; the gates that turn a build into a deployable artifact.→
- SBOM generation in CIWhy an SBOM is part of the artifact; SPDX versus CycloneDX as formats; syft and cyclonedx-bom as the two dominant generators; what an SBOM cannot do.→
- Image signing in CIHow cosign signs container images; keyless signing with OIDC and Fulcio; the production discipline that turns a signature from a checkbox into a chain of trust.→
Part LIV
Infrastructure Testing Strategy
6 checks
- The testing pyramid for IaC — five layers and what each one costsThe five-layer testing pyramid for infrastructure-as-code; the cost and runtime of each layer; what each layer catches and what it is scoped to miss; how the layers compose into a single change-gate strategy.→
- Static checks and policy — the cheapest layer and what it catchesWhy the static and policy layers belong at the base of the testing pyramid; the tools that implement them (terraform fmt, terraform validate, ansible-lint, kubeconform, tflint, tfsec, checkov, conftest); what each tool catches; why policy violations that are syntactically valid are still free to catch.→
- Unit and module tests — terraform test and Molecule in the pipelineThe third layer of the testing pyramid: unit and module tests; terraform test (the 1.6+ native test framework) for Terraform modules; Molecule for Ansible roles; what each framework catches; the cost in seconds and minutes; how to keep the corpus small and high-value.→
- Disposable integration tests — ephemeral environments and what they catchThe fourth layer of the testing pyramid: disposable integration tests; ephemeral environments that ask the live cloud or cluster; Terratest, Molecule with cloud drivers, kubectl apply --dry-run=server; the cost in money and time; the discipline of keeping the corpus small.→
- Staging and production validation — the final layer and what it should catchThe fifth and final layer of the testing pyramid: staging and production validation; smoke tests against a deployed environment; canary analysis; drift detection; what this layer should be scoped to catch and what it deliberately leaves to the layers below.→
- The test strategy decision — choosing depth against the blast radiusHow to choose the depth of the testing pyramid for a given change; the trade-off between the cost of testing and the blast radius of the change; when to invest in the expensive layers and when the cheap layers are enough; the strategy decision as a recurring operational call.→
Part LV
Continuous Delivery versus Continuous Deployment
6 checks
- Continuous Delivery vs Continuous Deployment — the actual distinctionThe single real difference between continuous delivery and continuous deployment; the presence or absence of a human approval gate between the artifact-ready stage and the production apply; why the terms are routinely conflated in vendor marketing and how to read past the marketing.→
- Continuous integration recap — the foundation and its limitsWhat continuous integration actually guarantees; what it does not guarantee; why CI is the necessary foundation for both continuous delivery and continuous deployment but is not sufficient for either; the line between automation and gatekeeping.→
- Continuous delivery requires approval — what the gate meansWhat a continuous delivery pipeline looks like; what the approval gate is and is not; the common forms of approval - click-through in environment protection, change advisory board records, manual workflow steps; why the gate is the safety mechanism that distinguishes delivery from deployment.→
- Continuous deployment — no approval, what that impliesWhat continuous deployment means in production; what discipline must compensate for the absence of human approval; the maturity prerequisites - comprehensive automated tests, observability, progressive delivery, canaries, automated rollback; the trade-off the team is making.→
- When deployment without approval fails — failure modes and incident classesThe classes of incidents that occur when continuous deployment is implemented without the supporting discipline; the false sense of safety that a green CI run creates; the specific failure modes - silent regressions, performance regressions, semantic breaks, data-shape changes - and the runbook responses.→
- The decision framework — choosing between continuous delivery and continuous deploymentA framework for choosing between continuous delivery and continuous deployment; the criteria - test coverage, observability, blast radius, regulation, on-call burden; what each option requires from the team; how to move from one to the other as the team acquires the compensating disciplines.→
Part LVI
Deployment Environments
6 checks
- Environments and promotion — the model and the boundaries between environmentsWhat an environment is as a deployment target; the boundaries between environments; how the promotion model treats the artifact as invariant across environments; what changes between environments versus what stays the same.→
- Development environment — fast feedback and the failure modes that are acceptableWhat the development environment is for; why it is the inner loop of the team; the failure modes that are acceptable in development but unacceptable anywhere else; the discipline that lets a fast-moving environment stay safe.→
- Test and staging environments — closer to production and the data questionWhy staging should mirror production in topology but differ in data; the three options for staging data - synthetic, sanitized, or fresh; the failure modes that are acceptable in staging but not in production; the boundary between staging and production as a pre-production rehearsal.→
- Production environment — the boundary and what production means in this courseWhat production means as a deployment target; the boundary at the production edge; what changes about identity, observability, and blast radius at the production boundary; the operational discipline that the production environment exists to enforce.→
- Identity isolation per environment — separate OIDC identities and the blast-radius disciplineWhy each environment gets its own OIDC identity and IAM role; how the platform issues short-lived credentials scoped per environment; the blast-radius discipline that identity isolation enforces; the failure modes that identity isolation prevents.→
- Ephemeral and preview environments — per-PR environments and the costWhat ephemeral and preview environments are; how a pull request gets its own short-lived environment; the cloud cost, identity complexity, and lifecycle management overhead they introduce; the workloads they serve well and the workloads they serve badly.→
Part LVII
Approval Gates
6 checks
- When approval adds safety — the scenarios where a human checkpoint prevents real incidentsWhat approval gates are for; the categories of failure a human checkpoint catches that automated verification cannot; the scenarios where adding an approver changes the production outcome; how to recognise a safety gate versus a checkbox gate.→
- When approval adds bureaucracy — the scenarios where a gate is theatreWhat approval theatre looks like; the conditions under which an approval gate adds latency without adding safety; the cost of ceremony on engineering velocity; how to recognise the difference between a substantive gate and a checkbox and how to remove the checkbox without removing the discipline.→
- Protected environments and required reviewers — the platform-side mechanism that enforces the gateHow protected environments and required reviewers work in GitHub Actions and GitLab CI; the canonical protection rules (reviewers, wait timer, branch restriction); the real command to add required reviewers; how the rules compose into a deployment gate; the trade-off between friction and signal.→
- Pull request approvals — N approvals, CODEOWNERS, dismiss stale, the auditHow pull request approvals work as the change-side counterpart to the deploy-side gate; the N-approvals rule, CODEOWNERS integration, dismiss-stale behaviour, and the audit trail that the PR produces; how the change-side and deploy-side gates compose.→
- Multi-party approval and segregation of duties — financial-grade controls and when they matterWhat multi-party approval means as a discipline; the financial-grade controls that mandate it; when segregation of duties matters for an infrastructure repository; the cost of the control; how to apply it without breaking the deploy cadence.→
- Approval fatigue and bypass risks — the failure mode and the warning signsWhat approval fatigue is and how it produces bypass behaviour; the warning signs that a team has crossed from substantive review into rubber-stamping; the failure modes that emerge when the gate is bypassed; the discipline of catching the drift before the gate stops being a gate.→
Part LVIII
Deployment Strategies
6 checks
- The deployment pattern taxonomy — five patterns, four trade-off axesThe five standard deployment patterns (recreate, rolling, blue-green, canary, A/B) and the four axes along which they trade off (availability, rollback speed, cost, blast radius); how to read the trade-off space before choosing a pattern.→
- Rolling update — incremental replacement, surge and unavailabilityHow a rolling update replaces old pods with new pods incrementally; the role of maxSurge and maxUnavailable; when rolling update fits; how to observe and roll back a rolling update in Kubernetes.→
- Canary and progressive delivery — traffic splitting, metric-based promotionHow canary deploys route a small fraction of traffic to the new version and promote it based on metrics; the role of Argo Rollouts and Flagger in the Kubernetes toolchain; the metric gates, the analysis templates, and the rollback-on-regression behaviour.→
- Blue-green deployment — two environments, atomic switch, instant rollbackHow blue-green deploys run two identical environments and switch traffic atomically; the load balancer / DNS / Kubernetes Service mechanics; the data question; the rollback simplicity; when blue-green is the right pattern and when it is not.→
- Recreate deployment — the downtime cost and the cases that justify itHow a recreate deployment stops the old version before starting the new version; the downtime cost in availability terms; the workloads where the downtime is acceptable (development, breaking migrations, stateful single-instance services); how to minimise the downtime window.→
- Choosing a strategy — the decision matrix and the production disciplineThe decision matrix that maps workload profile and change type to deployment pattern; the production discipline for evaluating, adopting, and retiring a pattern; the metrics that signal a pattern mismatch; how the pattern choice evolves as the workload matures.→
Part LIX
Rollback
6 checks
- Rollback across artifact boundaries — the five mechanisms and what each can undoWhy rollback is not a single operation but five: application image, Kubernetes Deployment, Terraform state, configuration management, and database. The artifact boundary each mechanism crosses, and the failure mode each one cannot recover from.→
- Application rollback — promoting the previous digest, the registry as the source of truthHow to roll back a containerised application by promoting the previous image digest; why the registry is the source of truth for what can be rolled back to; the role of digest pinning; the failure mode when the previous digest is no longer available.→
- Kubernetes rollback via Deployment history — kubectl rollout undo and the revision modelHow Kubernetes Deployments keep one ReplicaSet per revision within revisionHistoryLimit; how kubectl rollout undo reverts to a previous revision; how kubectl rollout history and --to-revision give precise control; the boundary the controller does not cross.→
- Terraform rollback via state and applied — terraform state list and the targeted applyHow Terraform rollback reverts cloud resources through state and the applied configuration; the role of terraform state list as the inventory of what Terraform owns; git revert plus a fresh reviewed plan as the primary rollback, with terraform apply -target reserved for exceptional error recovery; the boundaries of state-driven recovery.→
- Configuration management rollback — Ansible idempotency and the next-run restorationHow Ansible rolls back not by an explicit command but by the next idempotent run; why declared-state automation does not need a separate rollback procedure; the failure mode when the previous state itself was wrong; the discipline of testing rollback with the next run.→
- Database rollback and the data question — forward-fix vs backward-restore, the migration disciplineWhy database rollback is fundamentally different from the other four mechanisms; the data question (the data the new schema wrote cannot be unread by the old schema); the forward-fix discipline; when a backward-restore from backup is the right answer; the migration framework.→
Part LX
Forward Fix versus Rollback
6 checks
- The decision framework — what makes rollback safe and what makes forward-fix the only optionA framework for choosing between rollback and forward-fix based on four properties of the change; the decision matrix; why the reflex answer "always roll back" is wrong as a default.→
- When rollback is the right answer — config regressions, deployment bugs, image issuesThe three categories of change where rollback is the safe response; the operational pattern for confirming the rollback is right; what to verify after the rollback completes.→
- When forward-fix is the right answer — schema migrations, breaking changes, data issuesThe three categories of change where forward-fix is the only safe response; the operational pattern for writing and applying a forward-fix; what to verify after the fix lands.→
- The decision cost and time — MTTR trade-offs and the production disciplineHow cost and time determine the on-call decision when both rollback and forward-fix are technically possible; the MTTR budget; pre-staging the forward-fix; the runbook pattern that lets the on-call engineer choose without re-deriving the answer.→
- Rollback and data integrity — when rollback cannot be undone and the cascading data effectsChanges whose rollback is technically possible but operationally destructive; the cascading data effects across replicas, caches, queues, and downstream consumers; the discipline of accepting the forward-fix even when the rollback is faster.→
- Post-rollback investigation — the audit trail, the fix-forward, and the lessonThe post-rollback investigation that turns a rollback or forward-fix into an organisational lesson; the audit trail; the fix-forward that prevents recurrence; the blameless post-mortem.→
Part LXI
Pipeline Failure Handling
6 checks
- The failure types — flaky tests, real failures, infrastructure failures, race conditionsThe four categories of pipeline failure and the immediate response each requires; why a single "retry on failure" rule is wrong; how to classify a failure before deciding on the response.→
- Retries and backoff — automatic retry vs manual retry, and the exponential backoff patternWhy automatic retry is a shield, not a default; when exponential backoff is the correct pattern; the difference between a retry budget and a retry stampede; the discipline of deciding per-job what retry, if any, is correct.→
- Cleanup on failure — the cleanup path when a job fails midwayThe cleanup step that must run whether the job succeeded or failed; using if: always() to guarantee execution; the difference between cleanup and rollback; the discipline of treating cleanup as a first-class pipeline stage.→
- Partial deployment and resumability — when a deploy gets four of five changes doneThe failure mode of an interrupted deploy; the distinction between resumable and non-resumable operations; how to make a deploy resumable by tracking completed sub-changes; why "complete or revert" is the alternative when resumability is not possible.→
- Idempotency and the safe retry — why retry only works if the operation is idempotentThe mathematical definition of idempotency and its operational consequence; the difference between a safe retry and an unsafe retry; how to recognise and fix a non-idempotent deploy; the discipline of treating idempotency as a contract.→
- The postmortem and the fix — from a failed pipeline to a fix-forwardThe discipline that turns a failed pipeline into an improved pipeline; the difference between blame and learning; what an effective postmortem documents; the forward-fix that prevents the failure from recurring.→
Part LXII
Concurrency
6 checks
- Concurrent deployments and isolation — what "concurrent" means and the failure modesWhat "concurrent" means in a deployment pipeline; the four failure modes that concurrency introduces - interleaved writes, state corruption, lost updates, split-brain - and the operational cost of each.→
- Environment locking and mutexes — concurrency groups, cancel-in-progress, and the implementationHow GitHub Actions concurrency groups work; the cancel-in-progress flag and its trade-offs; how to scope a group to an environment; the difference between a workflow-level and a job-level concurrency group.→
- Terraform state locking — why state lock is mandatory; backend lock typesWhy Terraform state locking is not optional in a shared environment; the four backend lock types - local, S3+DynamoDB, Consul, GCS - and what each protects against; how the lock acquisition and lock timeout work; when to use terraform force-unlock.→
- Concurrency limits and throttling — runner concurrency, queue depth, and the queue effectHow runner concurrency limits create queues; the relationship between runner pool size, job arrival rate, and queue depth; why throttling is sometimes the right answer; the difference between runner concurrency and job concurrency.→
- Queueing and serialization — when to serialise a deploy and the cost it carriesWhen to fully serialise a deploy instead of partially throttling; the cost of serialisation in latency and developer experience; the queue disciplines that make serialisation visible; why "one at a time" is sometimes the only safe answer.→
- The concurrency discipline — when to allow concurrency, when to prevent itThe decision framework for concurrency: target resource shape, lock availability, recovery cost, blast radius; when to allow concurrency, when to throttle, when to serialise; the rules that synthesise lessons 1-5 into a production policy.→
Part LXIII
CI/CD Observability
6 checks
- The CI observability question — what gets measured, what doesn't, and the failure modesWhat "CI observability" actually means; what metrics a pipeline naturally emits and what it doesn't; the four failure modes when a team treats the green tick as evidence of health; why a passing pipeline can mask a degrading system.→
- Queue and runtime metrics — what they reveal and the alert thresholds that matterWhat queue depth and runtime duration actually measure; the alert thresholds that distinguish a healthy pipeline from a saturating one; why runtime is a leading indicator of failure and queue depth is the hidden cost of serialisation.→
- Success rate and flake rate — the distinction and how to interpret a 5% flake rateWhat success rate measures versus what flake rate measures; why a 95% success rate can mask a 30% per-attempt failure rate; how to compute flake rate from retry counts; the alert threshold that catches a flaky suite before it becomes a broken one.→
- Deployment frequency and lead time — the first pair of DORA metrics and what they meanWhat deployment frequency and lead time for changes measure; why these are the first pair of DORA metrics; the operational meaning of each metric for an infrastructure team; the relationship between the two.→
- Change failure rate and MTTR — the second pair of DORA metrics and how to interpret themWhat change failure rate and mean time to restore (MTTR) measure; why these are the second pair of DORA metrics; the operational meaning of each for an infrastructure team; the relationship between throughput metrics and stability metrics.→
- Observability without employee ranking — the ethical boundary and what metrics actually measureThe ethical boundary of CI metrics: what they measure (systems) and what they must never measure (people); why per-engineer dashboards are a misuse of observability; the structural patterns that produce healthy teams; how to design metrics that improve the system without ranking the engineers.→
Part LXIV
Auditability
6 checks
- The audit chain — the six links between a commit and a running serviceThe six links of the audit chain; what each link records; why each must be machine-readable; the difference between a green deploy and an auditable deploy.→
- The deployment claim — what the record asserts and what it omitsWhat a deployment record claims to assert; the difference between the claimed and actual provenance; what a deployment record deliberately omits; the falsifiability test for a deployment claim.→
- The change question — who, what, why, when, where, and howThe five questions a deployment record must answer; the difference between trigger and decision identity; why "why" is the hardest to record; the audit-grade answer to each.→
- Deploy evidence and provenance — artefact identity and the chain to sourceWhat deploy evidence is; the difference between identity and integrity; the artefact identity as a content-addressed digest; the chain from a running container back to a commit.→
- The deployment receipt — what the audit-grade record containsWhat a deployment receipt is; the required fields of an audit-grade record; the difference between a deployment record and a deployment receipt; the in-cluster evidence that makes the receipt durable.→
- Six months later — reconstruction as the test of an audit trailReconstruction as the practical test of an audit trail; the six queries a real auditor runs; the time-to-answer as the operational metric; the gap analysis when reconstruction fails.→
Part LXV
Software Supply Chain Security
6 checks
- The supply chain trust boundaries — six stages, each with a threat modelSoftware supply chains are pipelines with trust boundaries. Each boundary is a place where an attacker can substitute bytes, identities, or artifacts. Six stages, six threat models, one rule: verify at every boundary.→
- Source trust and repository attestation — branch protection, signed commits, the trust chainSource trust begins at the commit. Branch protection, signed commits, and repository attestation define who can introduce code and under whose identity. The first link of the supply chain is the human-to-commit link.→
- Dependency trust and the attack surface — what is pulled in, what could be maliciousA build pulls in libraries, base images, and tools from external sources. Every pull is a trust decision. The dependency tree is the largest attack surface in the supply chain and the one most often left unverified.→
- CI trust and the runner as an actor — the runner is an attacker if compromisedThe CI runner is the actor that executes the build. The runner has access to the source, the dependencies, the secrets, and the publishing credential. A compromised runner is a build-stage attacker with all of the production trust delegated to it.→
- Artifact trust and content-addressing — digests, signing, the registry as trust storeThe registry is the trust store between the build and the deployment. An artifact identified by digest is verifiable; an artifact identified by tag is mutable. Signing the artifact binds the bytes to the publisher identity. cosign verify is the gate.→
- Deployment trust and the runtime boundary — what happens when an artifact reaches productionThe runtime boundary is the last trust boundary. The deployer identity, admission control, and the policy that gates the cluster are the controls. The deployment does not end when the artifact starts; it ends when the artifact is verified at the runtime boundary.→
Part LXVI
Third-Party Actions and Plugins
6 checks
- Third-party actions are code — they execute on the runnerA third-party GitHub Action is arbitrary code, downloaded at job start and executed on the runner with the runner's privileges. The trust you extend to the marketplace is the trust you extend to the action author, their dependencies, and their build pipeline.→
- Mutable references and the tag-swap attackA version tag is a pointer, not a value. The runner resolves @v2 to whatever commit the tag points at today, which is not necessarily the commit it pointed at yesterday. A tag-swap attack moves the pointer; the workflow does not change; the code that runs does.→
- Compromised maintainers and the supply-chain attackSupply-chain attacks against package registries — npm, RubyGems, PyPI — have repeatedly demonstrated that the attacker does not need to compromise your repository. The attacker compromises the maintainer, the registry account, or the build pipeline, and your runner pulls the malicious version on the next resolve.→
- Abandoned actions and the frozen actorAn abandoned action is an action whose maintainer has moved on. The action is not malicious; the action is unmaintained. The code does not change, but the world around it does — new CVEs in transitive dependencies, new runner images, new APIs the action calls. The frozen action becomes a frozen attack surface.→
- Pinning to commit SHA — the only safe referenceThe defence against the tag-swap attack is to bind the workflow to an immutable content hash. Pinning to a full 40-character commit SHA turns the action reference into a content address; the runner downloads exactly the bytes your team reviewed, regardless of what the maintainer publishes later.→
- Allowlisting and policy — controlling which actions are permittedPinning binds the workflow to specific bytes; allowlisting decides which actions are permitted to be pinned in the first place. An organisation-level allowlist with named owners and required review is the policy layer that turns pinning from a per-workflow choice into an enforceable organisation rule.→
Part LXVII
Dependency Pinning
6 checks
- The pinning discipline — why mutable references are a vulnerabilityMutable references — tags, branch tips, range constraints — can resolve to different content over time. Pinning binds a reference to a content address and turns the supply chain from a chain of trust into a chain of evidence.→
- Action and plugin pinning — GitHub Actions, GitLab CI, Jenkins pluginsCI actions, templates, and plugins are pulled from external registries on every run. Pinning binds the runner to bytes the publisher cannot move; the syntax differs across systems but the discipline is the same.→
- Container image pinning — digests over tagsA container tag is a publisher-chosen string that resolves to different bytes when the publisher re-tags. A digest is a content hash that resolves to fixed bytes forever. Production pipelines pin by digest, not by tag.→
- Terraform provider and module pinning — version constraints and the registry lockTerraform providers and modules are pulled from registries at every init. Version constraints narrow the selection; the dependency lock file (`.terraform.lock.hcl`) freezes the selection to specific checksums. The lock file is the source of truth for reproducible plans.→
- Ansible collection and role pinning — requirements.yml with version constraintsAnsible collections and roles are pulled from Galaxy, Git, or a private automation hub. Version constraints in `requirements.yml` declare intent; exact pins freeze selection. Production playbooks pin collections and roles deliberately and review the upgrade.→
- Package pinning and lockfiles — npm shrinkwrap, pip-tools, go.sum, Cargo.lockApplication and tooling packages are pulled from registries at build time. Range constraints let the registry publish a new satisfying version between two consecutive builds. Lockfiles pin every direct and transitive dependency to a specific version and checksum; the lockfile is committed to the repository.→
Part LXVIII
SBOM
6 checks
- What an SBOM is — the bill of materials for softwareA Software Bill of Materials is the inventory of what is inside an artifact. The structural analogy to manufacturing BOMs, the three core fields every SBOM captures, and the operational questions an SBOM answers.→
- SPDX and CycloneDX — the two formats, the trade-offs, the toolingSPDX and CycloneDX are the two SBOM formats with mature tooling. They model components differently; the choice between them is a downstream-consumer decision. The trade-off table, the format overview, and the conversion trap.→
- SBOM generation in CI — syft, cyclonedx-bom, cdxgenSBOM generation belongs in the CI pipeline at build time, against the final artifact. syft is the dominant generator; cyclonedx-bom is the format-native CLI; cdxgen is the application-source specialist. The integration is a CI step that scans, emits, uploads, and prepares for attestation.→
- SBOM distribution and attestation — SBOM as an in-toto attestationThe SBOM is durable only when it is anchored to the artifact digest as a signed attestation. cosign attach sbom, the in-toto attestation envelope, the OCI referrer model, and the verification side that makes the signed SBOM load-bearing.→
- Vulnerability matching against SBOMs — VEX, Grype, TrivyThe SBOM is the input; the matcher is the tool that turns components into CVE matches. Grype and Trivy scan SBOMs offline; VEX (Vulnerability Exploitability eXchange) is the document that distinguishes a CVE match from a CVE that is actually exploitable.→
- SBOM as an operational input — incident response, license audit, dependency changeThe SBOM is consumed by three production operations beyond vulnerability matching: incident response (what changed since the last known-good build), license audit (does the artifact comply with policy), and dependency change (what lands in the next build). The SBOM is the input to all three.→
Part LXIX
Artifact Signing
6 checks
- Why sign artifacts — the supply chain integrity argumentA signature is the link between an artifact digest and the identity that produced it. Without it, the registry is a wall of mutable tags; with it, the verifier can answer "was this produced by the build pipeline I trust?". The supply chain integrity argument and what an unsigned deployment costs.→
- Cosign and the Sigstack — what cosign does, the broader Sigstore ecosystemcosign is the signing CLI of the Sigstore project. The Sigstore ecosystem extends cosign with Fulcio (certificate authority for OIDC-based short-lived certs), Rekor (transparency log), and the gitsign and policy-controller components. Mapping the parts to the chain of trust.→
- Keyless signing with Fulcio — OIDC-issued ephemeral keysThe keyless flow uses an OIDC token from CI as the signer identity. Fulcio issues a short-lived X.509 certificate naming the OIDC subject; the per-run ephemeral key never leaves the CI runtime. The verifier trusts the certificate, not a stored key.→
- Self-managed keys and KMS — when to use your own keys, KMS integrationThe keypair flow re-enters when the team cannot rely on a public Sigstore instance, runs self-hosted CI without OIDC, or has compliance requirements that mandate a customer-managed key. KMS integration with AWS KMS, GCP KMS, HashiCorp Vault, and PKCS#11 HSMs is the production path for keypair signing.→
- Rekor and the transparency log — what Rekor does, the public logRekor is the append-only transparency log that records every signed public statement. The log is the tamper-evident record; a signature without a Rekor entry has lost the tamper-evidence property. The public-good instance, the self-hosted instance, and the verifier integration.→
- Verify on deploy — the cluster admission controller, policy-controllerThe signature is the data structure; the verifier is what makes the data structure load-bearing. The cluster admission controller (Sigstore policy-controller, kyverno, Connaisseur) is the production pattern that makes a signature reject an unsigned image at deploy time. The verification policy, the ClusterImagePolicy object, the failure mode.→
Part LXX
Provenance
6 checks
- What provenance is — the attestation of where an artifact came fromProvenance is the verifiable, signed claim that an artifact was built from a specific source, by a specific builder, in a specific step. This lesson defines provenance, distinguishes it from a build log, a signature, and an SBOM, and shows where it sits in the SLSA, in-toto, and Sigstore stack.→
- SLSA build levels — L0 through L3 and what each level guaranteesSLSA build levels describe incremental guarantees about the build process. L0 has no provenance; L1 has provenance but does not sign it; L2 adds signed provenance and an isolated build; L3 adds a hardened build platform and two-party review. The levels are the yardstick the audit team uses to score the supply chain.→
- in-toto attestations — the standard format and predicate typesin-toto is the envelope format for provenance, SBOMs, VEX, and other supply-chain claims. The Statement binds a subject to a predicate under a typed predicate type; the DSSE envelope wraps the signed payload. This lesson walks the envelope, the predicate types, and the verifier that consumes them.→
- SLSA source track — source control integrity and the producer side of provenanceThe SLSA source track grades the source control system that produced the artifact. L0 has no source integrity; L1 records the source commit; L2 adds signed commits and branch protection; L3 adds two-party review and a hardened source platform. The source track is the producer side of the provenance chain.→
- GitHub Actions artifact attestations — actions/attest and the platform-native provenanceGitHub Actions produces SLSA build provenance and SBOM attestations natively via actions/attest-build-provenance and actions/attest-sbom. The attestations are signed with the OIDC identity of the workflow, stored in the GitHub attestations API and as OCI referrers, and exposed in the GitHub UI for audit. This lesson walks the two actions, the output, and the workflow identity.→
- Consuming and verifying attestations — the verifier side and the policyThe attestation is load-bearing only when the verifier checks it. gh attestation verify, the Sigstore policy file, the OIDC subject pin, and the policy-as-code artifact the audit team consumes. The verifier side is where the SLSA level is actually earned.→
Part LXXI
CI/CD Threat Modelling
6 checks
- The CI/CD threat model — who attacks, what they want, how they get inA CI/CD pipeline is a high-value target because it is a single actor with delegated trust across source, dependency, artifact, and deployment boundaries. This lesson defines the attacker classes, the assets they want, and the six attack surfaces a defender has to think about.→
- The malicious commit attack — the insider or compromised PR scenarioA pull request that lands a malicious commit is the canonical CI/CD attack. The attacker is the developer (insider) or the developer (compromised). The PR passes review because the change looks legitimate; the build succeeds because the code compiles; the artifact is published because the pipeline ran. The audit trail shows a clean PR, a green build, and a signed artifact — and a backdoor.→
- The account-compromise attack — the stolen-credentials scenarioA stolen personal access token, a phished OIDC session, a hijacked SSH key, or a leaked deploy credential. The attacker who holds a developer or runner identity holds the trust the team has delegated to that identity. This lesson walks the credential-theft attack, the audit gap, and the controls that make a stolen credential short-lived and revocable.→
- The dependency-compromise attack — the upstream-package scenarioA typosquatted package, a compromised maintainer account, a hijacked mirror, or a poisoned base image. The attacker who compromises an upstream the team pulls from compromises the team's build. This lesson walks the dependency-compromise attack, the detection signals in the dependency tree, and the controls that pin and verify every upstream.→
- The runner-compromise attack — the persistent-access scenarioA self-hosted runner with a long-lived filesystem, a long-lived credential, and a permissive workflow is a runner the attacker can persist in. The attacker who compromises the runner today has access tomorrow, the day after, and every day until the runner is rebuilt. This lesson walks the runner-compromise attack, the persistence signals, and the controls that make a runner a one-shot environment.→
- The secret-leak and registry-compromise attack — credential-exfiltration scenariosA secret in a workflow log, a credential in an artifact, a registry token in an environment file. The CI pipeline produces a lot of output; the output goes to a lot of places; the attacker who reads the output reads the team's secrets. This lesson walks the secret-leak attack, the registry-compromise attack, and the controls that keep credentials out of the build output and the artifact registry.→
Part LXXII
GitOps Foundations
6 checks
- The GitOps principles — declarative, versioned, pulled, reconciledThe four OpenGitOps principles that define what makes an operational model GitOps; how declarative desired state, versioned immutability, automatic pull, and continuous reconciliation differ from the older imperative-and-push patterns.→
- Declarative desired state — describing what should be, not how to get thereWhy GitOps starts from a declarative description of the desired state; how YAML, HCL, Jsonnet, and CUE differ in their expressiveness and their cost; what "convergence to a description" means operationally.→
- Versioned and immutable — Git as the canonical record of desired stateWhy the desired state must live in a version-control system with immutable history; what protection, signed tags, signed commits, and content addressing buy the GitOps model; how immutability is the trust anchor.→
- Pulled automatically — the controller reaches out to GitWhy GitOps inverts the direction of change; how a controller inside the cluster pulls from Git instead of CI pushing to the cluster; what the inverted trust boundary buys in security and reliability.→
- Continuously reconciled — observed versus desired, and the loopHow a GitOps controller observes the actual state, computes the delta against the desired state, and converges; what reconciliation level (managed, sync-wave, automated) means; how drift is detected and corrected.→
- GitOps versus traditional CD — the push model, the pull model, and the trust differenceA side-by-side comparison of GitOps and traditional CI/CD: credentials and blast radius, drift correction, audit trail, deployment latency, and operational failure modes. When each model is the right tool.→
Part LXXIII
Push versus Pull Deployment
6 checks
- The push model — CI holds credentials and reaches out to the clusterHow the traditional CI/CD push model works: the runner authenticates to the cluster and applies state. Where the credentials live, what the runner does, and what the trust boundary looks like in production.→
- The pull model — a controller inside the cluster reaches out to GitHow the GitOps pull model works: a controller inside the cluster authenticates to Git, reads the desired state, and applies it. Where the credentials live, what the controller does, and why the trust boundary inverts.→
- Credential boundaries — where the secrets live in each modelA side-by-side map of every credential each deployment model requires: cluster creds, Git creds, cloud-provider creds, registry creds. Where they live, how long they live, and what their compromise enables.→
- Blast radius — what a CI compromise affects, what a controller compromise affectsA precise accounting of the blast radius of a credential compromise in each model. What can a CI runner do if compromised? What can a controller do if compromised? What cannot either do?→
- Hybrid architectures — when neither the pure push nor the pure pull model fitsCases where neither pure push nor pure pull is right: bootstrapping clusters, multi-cluster fan-out, systems without a controller, mixed-tenant pipelines, and the operator pattern. How teams combine both models without losing the security properties of each.→
- The trust decision — choosing the right model for your organisationA decision framework for choosing between push and pull: cluster topology, repository trust, team capability, regulatory environment, blast-radius tolerance, and migration cost. How to make the choice defensible rather than dogmatic.→
Part LXXIV
Reconciliation
6 checks
- The reconciliation loop — observed, desired, difference, actionThe reconciliation loop in one diagram and one sentence; the four phases every tick performs; the cycle that turns a Git commit into cluster convergence on a continuous cadence.→
- Observed versus desired — what "actual" and "what should be" meanWhat the observed state really is; what the desired state really is; where each one comes from; the asymmetries that make the diff non-trivial, including defaulting, normalisation, and ownership annotations.→
- Reconciliation intervals — how often, and the trade-offsHow the reconciliation interval is set per controller and per resource; the trade-offs between responsiveness and API-server load; how to size the interval to the workload and the operational cost of a tick.→
- Argo CD reconciliation mechanics — the three components, the cache, the diffHow Argo CD splits reconciliation across the repo server, the application controller, and the API server; how the cache keeps the controller fast; how the diff is computed and what the sync window for apply looks like.→
- Flux reconciliation mechanics — GitRepository, Kustomization, the source controllerHow Flux splits reconciliation across the source controller, the kustomize-controller, and the helm-controller; the GitRepository and Kustomization CRDs; how the source-controller produces artifacts the other controllers consume.→
- Failure modes in reconciliation — what stops the loop, and how to detect itThe failure modes that stop the reconciliation loop: read failures, observe failures, apply failures, render failures, RBAC denials, and stuck caches. The signals each one produces and the remediation for each.→
Part LXXV
Drift
6 checks
- What drift is — the actual diverging from the desiredDefine drift as the divergence between the live observed state and the rendered desired state; explain why drift is the central failure mode of a GitOps system; introduce the four categories — manual, accidental, emergency, and cascading.→
- Manual drift — the operator edits the cluster directlyWhy operators edit the cluster directly; how the reconciliation loop detects the edit; the race between hand-applies and self-heal; why `kubectl edit` is the canonical example and what its signature looks like in the Application status.→
- Accidental drift — controllers, schedulers, and side effectsHow controllers, schedulers, and mutating webhooks change resources without an operator touching them; why the diff engine must classify these as drift; and how to tell accidental drift apart from manual drift from the diff alone.→
- Emergency drift — when incident response breaks the modelWhy incident response sometimes requires breaking the GitOps model; the break-glass question; how to record the break so the post-incident review can repair the model; and why the controller should pause, not fight, during a declared incident.→
- Drift detection and alerting — what metrics show drift and what the alerts look likeWhat to measure to detect drift early; the metrics the GitOps controller exposes; the alerts that page; the dashboards that show drift over time; and the difference between detection and remediation in the alert chain.→
- Self-heal versus control — when automatic reconciliation fights the operatorWhy self-heal can fight a legitimate operator action; the per-Application toggle; the production rule that self-heal is on by default but suspendable per incident; and how to design an Application so that operator actions never need to fight the controller.→
Part LXXVI
GitOps Repository Architecture
6 checks
- Application versus environment repositories — the two-repo patternWhy the cleanest GitOps split is an application repo that holds code and a separate environment repo that holds desired state, and how Argo CD and Flux wire each side to the other.→
- Monorepo with overlays — Kustomize and Helm values in one treeA single repository holding all environments, with Kustomize overlays or Helm values files driving the per-environment differences. The single-source-of-truth model and its trade-offs.→
- Multi-repo per environment — dev, staging, prod each their own repoHard isolation per environment by giving each cluster its own repository. The strongest authorization boundary, the heaviest operational cost, and the model that survives regulatory and compliance requirements.→
- Source Hydrator and the mirror — Argo CD sources that are not the application repoHow Argo CD Sources, the Source Hydrator, and the mirror pattern let a controller read desired state from a generated repo rather than from the application repo. The bridge between CI and the cluster without losing the two-repo boundary.→
- Promotion models — promote the immutable artifact vs promote the Git refThe two ways to move a release between environments: move the artifact (same digest, new environment) or move the Git ref (same source, new revision). The trade-offs in audit, rollback, and supply-chain trust.→
- The architecture decision — when each model fitsThe decision matrix that picks the right GitOps repository architecture for a given team: team size, regulatory regime, supply-chain posture, cluster topology, and the cost of the wrong choice.→
Part LXXVII
Argo CD
6 checks
- Argo CD architecture — the three components and the data flowThe Argo CD control plane decomposed into API server, repo server, and application controller. How a Git commit becomes a cluster reconcile and where each component owns a phase of that path.→
- The Application CRD — source, destination, sync, and the resource modelThe Application custom resource is Argo CDs deployment unit. This lesson decomposes its spec into source, destination, sync policy, and ignore fields, and explains how the CRD maps onto the three-component architecture from LXXVII-01.→
- Source types and Helm/Kustomize — directory, repo, helm, kustomize, pluginHow the Application source field maps to a renderer. Plain directory, Git repo, Helm chart, Kustomize overlay, and Config Management Plugin - what each renderer takes, what it returns, and which production pattern each enables.→
- Sync policies and sync windows — automated, manual, prune, and the time-bound gateHow the Application syncPolicy decides whether the controller reconciles a diff itself or waits. How sync windows gate automated syncs to specific time bands. Why prune and self-heal are separate dimensions, and when to enable each.→
- Projects and RBAC — the multi-tenant boundary and the policy CSVHow Argo CD AppProjects define a multi-tenant boundary. Source allowlists, destination allowlists, and the policy.csv that turns the project into a permissions object. Why projects are the unit of audit and the unit of blast radius.→
- ApplicationSet and cluster generation — one resource, many ApplicationsHow the ApplicationSet controller generates Argo CD Applications from a single template. List, cluster, git directory, git file, and matrix generators; the template that fills in name, source, and destination; and the operator workflow for managing many clusters.→
Part LXXVIII
Flux
6 checks
- The Flux architecture — the toolkit composition and the controller modelHow Flux decomposes into the GitOps Toolkit, the role each controller plays, and why the controller-per-CRD model is the architectural contract that every subsequent Flux lesson builds on.→
- GitRepository and source-controller — the source of desired stateHow source-controller reads Git, Helm, OCI, and S3 sources, the GitRepository CRD field by field, and how the produced artifact is consumed by downstream controllers.→
- Kustomization controller — reconciling Kustomize manifests into the clusterHow kustomize-controller consumes a source artifact, runs Kustomize against a path, applies the rendered manifests, prunes deleted resources, and reports health. The Kustomization CRD field by field.→
- HelmRelease controller — reconciling Helm charts with versioned valuesHow helm-controller consumes a chart source, renders the chart with values, performs the Helm install/upgrade/test/rollback lifecycle, and reports the release status. The HelmRelease CRD field by field.→
- Image automation and update — scanning registries, filtering tags, writing back to GitHow image-automation-controller scans registries for tags, filters them with ImagePolicy, and writes updated image references back to Git. The ImageRepository, ImagePolicy, and ImageUpdateAutomation CRDs.→
- Notification controller and webhooks — the alerting layerHow notification-controller receives events from the other Flux controllers, fans them out to receivers, and exposes a webhook receiver for inbound Git host events. The Alert, Provider, and Receiver CRDs.→
Part LXXIX
Sync Strategies
6 checks
- Manual sync — the default, the audit, and the discipline of waitingWhy manual sync is the default for a new Application; what the operator sees before invoking argocd app sync; why a human in the loop is still the right answer for high-risk workloads; the audit the diff produces.→
- Automated sync — the convenience and the controls it requiresWhat automated sync means; the controls (branch protection, drift detection, sync windows, audit) that must accompany it; how to choose automated sync per environment; the cost of an out-of-cycle apply.→
- Self-heal — when automatic correction is right and when it is wrongWhat self-heal reverts and what it does not; the cases where self-heal is the right operational answer (chart owns live state) and the cases where it is the wrong one (legitimate out-of-band edits); how to disable self-heal per-Application.→
- Prune and the blast radius — what prune deletes and the safety disciplineWhat prune actually deletes (resources owned by the controller, present in the previous apply, absent in the new one); the blast radius of an unintended prune; the per-Application discipline and the Prune=false sync option override.→
- Sync waves and phasing — ordering, dependencies, and the wave annotationWhy ordering matters in a multi-Application cluster; how Argo CD sync waves (argocd.argoproj.io/sync-wave annotation) and Flux dependsOn field express dependencies; how to design waves so a single failure does not stall the whole fleet.→
- Replace, force, and server-side apply — the options and their consequencesWhat server-side apply does and why it is the production default; what the Replace option does in Argo CD; what the force field does in Flux; when each option is right and the operator-ownership consequence of force.→
Part LXXX
GitOps Pruning
6 checks
- What prune is — Git desired state deletes cluster actualThe contract: when a resource leaves the Git manifest, the controller deletes the cluster resource. Why this is the GitOps audit story for deletions and the most dangerous button in the sync policy.→
- Prune versus orphan — the two failure modes for absent manifestsPrune deletes resources the controller tracks. Orphan leaves them in the cluster because no controller owns them. The two failure modes look similar in GitOps dashboards but require different remediation.→
- Orphaned resources and the cluster — what survives when the manifest disappearsOrphans are cluster resources no GitOps controller tracks. They survive every prune, accumulate drift, and silently break the "Git is the source of truth" promise. Detection, classification, and the production rules for managing them.→
- Prune safety mechanisms — dry-run, sandbox clusters, and the escape hatchThe three safety mechanisms that make prune survivable: dry-run diff before sync, sandbox clusters for first-prune verification, and the Prune=false sync option as a per-Application and per-resource escape hatch.→
- Incident — a prune deleted production — the recovery procedureA walkthrough of a production prune incident: the typo in the chart, the automated sync that executed it, the diagnostic steps, and the GitOps-native recovery that restores the resource as a Git commit.→
- Prune policy decision framework — when to allow prune, when to forbid itThe decision framework for choosing prune policies per Application: ephemeral workloads allow it, shared resources forbid it, persistent state requires an ownership audit first. The per-Application boundary and the standing configuration rules.→
Part LXXXI
Synced versus Healthy
6 checks
- What Synced means — desired state matches actual stateHow GitOps controllers define Synced, what reconciliation actually compares, why Synced is a structural property of the cluster versus Git, and the operator commands that surface the current Sync state.→
- What Healthy means — the workload is functioning correctlyHow GitOps controllers define Healthy, what resource-level and Lua health checks actually test, why Healthy is a workload-layer property, and the operator commands that surface the current Health state.→
- The Synced and Healthy matrix — four combinationsThe 2x2 matrix of Synced-or-OutOfSync against Healthy-or-Degraded, what each quadrant means operationally, how a controller classifies combinations, and why the matrix is the right mental model for the dashboard.→
- Progressing and Degraded states — the intermediate statesHow Progressing and Degraded differ from Healthy, why Progressing is a transient signal that resolves on its own, why Degraded is a stable signal that requires intervention, and how to tell them apart from the operator CLI.→
- The 3 AM test — what tells you the workload is broken when the dashboard is greenThe mental checklist an on-call engineer runs when paged for a GitOps-managed workload that the dashboard says is healthy, the external signals that catch what the controller cannot, and the order in which to run the checks.→
- When Synced is not Healthy — the failure modes the platform does not catchThe recurring categories of failure that produce a Synced but Degraded state, the diagnostic patterns for each, and the operator actions that close the gap between manifest correctness and workload correctness.→
Part LXXXII
GitOps Secrets
6 checks
- The GitOps secret problem — why plaintext credentials cannot live in GitThe fundamental conflict between Git-as-source-of-truth and secret material; the four properties a GitOps platform must preserve while still using Git as the conduit; why the answer is not "stop putting secrets in Git" but "put different things in Git".→
- External secret systems — Vault, cloud KMS, and the controller that bridges themThe external-system class of GitOps secret management; how HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, and GCP Secret Manager act as the trust boundary; how the External Secrets Operator reconciles cluster Secrets from external stores; what goes in Git and what stays out.→
- Encrypted Git workflows — encrypt the value, leave the keys elsewhereThe encrypted-at-rest class of GitOps secret management; how ciphertext committed to Git is reconciled by a controller holding the key elsewhere; the trust boundaries that make the model work; the differences between age, PGP, and cloud KMS as the key holder.→
- Bitnami Sealed Secrets — cluster-bound encryption and the controller as the trust boundaryThe Sealed Secrets model: a per-cluster keypair where the controller in the cluster holds the only private key; kubeseal produces a SealedSecret that only that cluster can decrypt; how the model differs from SOPS and external systems; the rotation and recovery path.→
- SOPS and Mozilla SOPS — file-level encryption for the GitOps repositorySOPS as the file-level encryption tool for YAML, JSON, ENV, and binary files in a GitOps repository; how age, PGP, and cloud KMS plug in as key holders; the Flux integration that decrypts on apply; per-value encryption and what it preserves.→
- The secret reference pattern — what goes in Git, what never shouldThe third class of GitOps secret management: a name-only repository where the cluster, CI, and runtime cooperate via a name-resolution layer; the boundaries between commit, store, and runtime; when the reference pattern is the right answer; the failure modes that motivate it.→
Part LXXXIII
GitOps RBAC
6 checks
- The GitOps RBAC model — repository, controller, and cluster as three layersGitOps permissions are not one permission system but three stacked ones: who can merge, what the controller is allowed to apply, and what the controller identity can do in the cluster. How the three compose, and why the weakest layer sets the effective privilege.→
- Repository permissions — the merge is the deployWho may merge to the GitOps repository is a production access-control decision, not a workflow preference. Branch protection, required reviews, path ownership, and the automation identities that quietly hold write access to desired state.→
- Controller cluster permissions — what the reconciler is allowed to doThe reconciling identity is the account that actually writes to the API server. Default privilege in Argo CD and Flux, scoping with ServiceAccount impersonation, the escalate verb, and how to audit what the controller can reach.→
- Argo CD RBAC in detail — projects, policy rows, and the Casbin modelHow the argocd-rbac-cm ConfigMap, AppProject roles, and the Casbin engine combine into an authorisation decision. Resources and actions, glob matching, the priority of deny, the trap in policy.default, and how to test a policy before it ships.→
- Flux multi-tenancy — lockdown flags and the per-namespace modelHow Flux separates platform admins from tenants: cross-namespace reference blocking, remote base blocking, default service accounts, and the per-namespace service account model that turns tenancy into Kubernetes RBAC.→
- Environment boundaries — dev, staging, and production as separate concernsWhy a directory name is not a boundary. The four axes an environment separation can be drawn on, what shared-cluster and separate-cluster models actually buy, and how to make a promotion the only sanctioned crossing.→
Part LXXXIV
Environment Promotion
6 checks
- Promotion by artifact — the same digest moves through environmentsPromotion by artifact means the same digest is referenced by every environment manifest. The CI pipeline builds once, resolves the digest, and every downstream environment pins that digest. The audit unit is the SHA-256.→
- Promotion by Git ref — the Git reference moves; the artefact is built per environmentPromotion by Git ref means each environment reconciles against its own commit in the environment repository. The rendered manifests differ per environment; the artifact is built per environment or referenced at the ref the env repo pins.→
- Promotion by environment repo — each environment has its own Git statePromotion by environment repo means each environment has a dedicated Git repository that records its own desired state. Promotion is the act of pushing a change from one repo to the next, often via a controlled promotion pipeline that opens pull requests across repos.→
- Promotion by ApplicationSet — programmatic generation of promotion targetsAn ApplicationSet is a controller that generates Argo CD Applications from a template and a set of generators. Promotion by ApplicationSet means the promotion targets are derived from cluster, Git, or matrix data rather than declared by hand.→
- Promotion windows and approvals — the time-bound gatesA promotion window is a time-of-day or day-of-week rule that allows or denies syncs. An approval is a human gate that must be cleared before the sync proceeds. Together they enforce the operational discipline that the GitOps controllers cannot infer.→
- The promotion decision — which model fits which organisationChoosing the promotion model is the most consequential GitOps decision a team makes. This lesson synthesises the four models into a decision framework based on supply-chain posture, organisational structure, scale, and audit requirements.→
Part LXXXV
GitOps Rollback
7 checks
- Git revert versus controller rollback — what Git undoes and what the controller undoesThe two meanings of "rollback" in a GitOps system: Git revert undoes the change in the source of truth; the controller rollback undoes the change in the cluster. Both are needed; they touch different artifacts; choosing the wrong one leaves drift.→
- Data rollback versus state rollback — the two meanings of "rollback" for an infrastructure changeA "rollback" in infrastructure engineering is two distinct operations: rolling back the desired state (the manifests, the Terraform code, the configuration) and rolling back the data the previous state wrote. The first is recoverable with git revert; the second usually is not.→
- Argo CD rollback and history — `argocd app rollback`, `argocd app history`, and the rollback UIHow Argo CD records every successful sync as a history entry; how argocd app history lists them; how argocd app rollback targets a specific history ID or the previous sync; the UI rollback button and its semantics; the boundary between a Git revert and an Argo CD rollback.→
- Flux rollback via revert — `git revert` plus reconcile, and the workflow the controller followsWhy Flux has no native rollback command; how the rollback is composed from a Git revert and a flux reconcile; the .status.history field on the Kustomization; the workflow the on-call engineer follows; the boundary between the Git revert and the controller reconciliation.→
- The rollback decision framework — when to revert Git, when to deploy a hotfix, when to forward-fixThe decision framework an on-call engineer follows at the start of an incident: revert the bad commit, ship a hotfix, or write a forward-fix. The signals that distinguish each path; the cost each path imposes; the data question each path must answer.→
- The rollback decision framework — when to revert Git, when to deploy a hotfix, when to forward-fixThe decision framework an on-call engineer follows at the start of an incident: revert the bad commit, ship a hotfix, or write a forward-fix. The signals that distinguish each path; the cost each path imposes; the data question each path must answer.→
- Post-rollback investigation — the audit trail, the fix-forward, and the lesson that closes the loopThe post-rollback investigation that turns a rollback or forward-fix into an organisational lesson; the audit trail; the fix-forward that prevents recurrence; the lesson that closes the loop between the incident, the investigation, and the next change.→
Part LXXXVI
GitOps Failure Modes
6 checks
- Repository unavailable — what happens when Git is downWhat a GitOps control loop does when the source-of-truth repository is unreachable: stalled syncs, diverging live state, and how to tell a blip from an outage.→
- Authentication failure — token expired, key rotatedWhat happens when the credential the GitOps controller uses to fetch the repository stops working: expired tokens, rotated SSH keys, and the cascade across applications that share a single secret.→
- Controller unavailable — the GitOps control plane is downWhat happens when the GitOps controller itself is not running: no reconciliation, no diff, no alerts, and a cluster that drifts silently. The control plane is the single most important component to monitor.→
- Bad manifests — the controller refuses to applyWhat happens when the manifests in Git are valid YAML but invalid Kubernetes: the controller computes a diff, refuses to apply, and marks the application Degraded. The detection, the remediation, and the CI gap that let it through.→
- Unhealthy deployment — synced but brokenWhat happens when the GitOps controller has applied the desired state but the application inside the cluster is failing: CrashLoopBackOff pods, failed health checks, and the gap between Kubernetes-level reconciliation and application-level health.→
- Secret dependency failure — External Secrets Operator cannot reach VaultWhat happens when a GitOps-deployed External Secrets resource cannot fetch from Vault: the controller reports Ready=False, the Secret is missing or stale, and the cluster has a secret-dependency failure invisible to the GitOps controller.→
Part LXXXVII
GitOps During Incidents
6 checks
- The incident versus GitOps tension — when the model fights the responderWhy incident response and the GitOps reconciliation loop are sometimes in conflict; the three incident shapes that make the model fight the responder; the rule that the controller pauses rather than fights during a declared incident.→
- Break-glass procedures — the discipline of breaking the model on purposeWhy break-glass procedures must be written before the incident; what a GitOps break-glass procedure contains; the role of declared incidents, named owners, time bounds, and post-incident reconciliation in keeping the break repairable.→
- Disabling self-heal — how, who, and for how longThe exact Argo CD and Flux commands for suspending self-heal on a single Application; the per-Application scope that distinguishes a surgical break from a cluster-wide one; the time bound that prevents the break from becoming permanent; the audit trail the suspend produces.→
- Post-incident Git reconciliation — the manual change must enter GitWhy the emergency edit must be committed to Git after the incident; the timing and ownership of the reconciliation commit; the validation steps that confirm the cluster and Git agree; the cost of leaving an emergency edit uncommitted.→
- The postmortem and the policy update — what changes after the incidentWhy the post-incident review must produce a policy update, not just a narrative; the categories of policy that change after a GitOps incident; the timeline for updating runbooks, break-glass procedures, sync windows, and review gates; the loop that closes when the policy is in production.→
- Pragmatism versus purity — when GitOps can waitWhen the GitOps model is the wrong default; the workloads and environments where traditional CI/CD is acceptable; the boundary between GitOps-applicable and GitOps-inappropriate; the team-cost calculus that determines whether GitOps is worth the discipline.→
Part LXXXVIII
Infrastructure GitOps
6 checks
- What infrastructure GitOps is — Terraform, Pulumi, and reconciliationWhy extending GitOps beyond Kubernetes requires a different controller model; how Terraform and Pulumi reconcile infrastructure declared in Git against state held outside the cluster; the boundary between Kubernetes-native GitOps and infrastructure GitOps.→
- Terraform and GitOps with Atlantis — pull-request-driven plan and applyHow Atlantis turns Terraform into a GitOps pull-request workflow; the plan-on-PR, apply-on-approval model; state locking across concurrent pull requests; the production discipline around the atlantis plan and atlantis apply commands.→
- Terraform and the Argo CD application controller — manifests as Git stateHow argoproj-labs/terraform-controller brings Terraform into a Kubernetes-native GitOps loop; the Terraform and Workspace CRDs; how manifests declared in Git reconcile cloud resources through an in-cluster operator; the boundary between in-cluster and out-of-cluster reconciliation.→
- Pulumi and Kubernetes GitOps — the Kubernetes operator patternHow the Pulumi Kubernetes Operator runs Pulumi programs as in-cluster jobs declared by a Stack custom resource; how the operator reconciles cloud resources from inside the cluster; the boundary between Pulumi Cloud and self-hosted backends.→
- Network and firewall GitOps — when the tooling supports itWhy extending GitOps to network and firewall configuration is harder than extending it to Kubernetes or cloud accounts; which vendors and tooling actually implement a reconcile loop; the boundary between policy-as-code (which GitOps well) and device-as-code (which GitOps less well).→
- The infrastructure GitOps limitations — state, drift, secretsWhere infrastructure GitOps falls short: state file corruption and loss; drift between the state backend and the cloud; secrets that the loop must handle but Git cannot store; the controller-shaped limits that distinguish infrastructure GitOps from Kubernetes GitOps.→
Part LXXXIX
Repository Security
6 checks
- MFA and account security — the first line of repository defenceWhy the engineer account is the real perimeter of an infrastructure repository; the factor hierarchy from SMS to hardware keys; recovery codes, break-glass identities, and organisation-wide MFA enforcement without locking out the automation.→
- Access control and the principle of least privilegeRepository roles on GitHub and GitLab, why teams beat individual grants, how machine identities differ from human ones, and the access-review cadence that keeps a permission list from becoming a historical record of everyone who ever touched the repo.→
- Branch protection deep dive — the full set of optionsEvery branch protection control on GitHub and GitLab, what each one prevents, which combinations are load-bearing for an infrastructure repository, and the bypass paths that make a protected branch less protected than the settings page suggests.→
- Required status checks — what runs and what must passThe difference between a check that runs and a check that blocks; context names as a fragile contract; strict mode and the up-to-date requirement; skipped checks that hang a merge forever; and why a required check on a fork pull request is not the check you think it is.→
- Secret scanning and push protection — native controls and gitleaksEnabling native secret scanning and push protection on GitHub and GitLab, wiring gitleaks where native scanning does not reach, handling bypasses as auditable events, and storing the credential in the place it belongs once it has been taken out of the diff.→
- Repository audit and monitoring — who changed what, whenThe forge audit log as the record of everything Git does not store: role grants, protection changes, token minting, and bypasses. Streaming the log off-platform, alerting on the events that matter, and reconstructing a timeline that survives a compromised administrator.→
Part XC
CI Platform Security
6 checks
- What the CI platform can touch — the blast radius of a CI compromiseThe CI platform as a convergence point for cloud credentials, cluster credentials, Terraform state, container registries, secret stores, and the production network. The blast radius is the union of every credential the platform holds.→
- Cloud access from CI — what cloud credentials doWhat an IAM role, a service account, and a subscription identity enable when the CI assumes them. The structural fix is OIDC federation; the operational discipline is verifying the assumed identity with aws sts get-caller-identity before every privileged step.→
- Kubernetes access from CI — what cluster credentials doWhat a kubeconfig and a service-account token enable from a CI runner. The structural fix is short-lived tokens bound to a specific apply or reconciliation; the discipline is verifying cluster access with kubectl auth can-i before every privileged action.→
- Terraform state access from CI — what state access enablesWhat reading and writing Terraform state from the CI enables: exfiltration of every resource ID and secret in the team's record of what exists. The structural fix is per-run scoped credentials plus state locking that the CI cannot circumvent.→
- Registry and secret store access from CI — what registry and secret-store creds enableWhat container registry push and pull credentials, and what secret store read credentials, enable from a CI runner. A push credential can replace an artifact under a digest the deploy expects; a read credential can read every secret the workflow declared.→
- Production network access from CI — the production boundaryWhy private VPC connectivity, bastion hosts, and jump boxes turn the CI into a production-network entry point. The structural fix is identity-based access mediated by short-lived credentials, not network reach; the discipline is egress allowlists plus audit logging on every production call.→
Part XCI
Least Privilege CI/CD
6 checks
- The validation versus deployment identity — two identities, two boundariesWhy a CI/CD pipeline needs two distinct cloud identities — one for validation that can only read, and one for deployment that can only act on the system it owns. The most common production incident is the single identity that can both read and write.→
- Scoped IAM roles — what each role can do; the smallest setHow to write the IAM permission policy for a least-privilege CI/CD role. Action-level scoping, resource-level scoping, condition keys, and the structural difference between terraform plan and terraform apply permissions.→
- Scoped Kubernetes RBAC — ServiceAccounts, Roles, RoleBindingsHow Kubernetes RBAC scopes a CI/CD pipeline: one ServiceAccount per job, Role (not ClusterRole) for namespace-scoped verbs, RoleBinding (not ClusterRoleBinding) for namespace-scoped grants, and kubectl auth can-i --list --as to audit effective permissions.→
- Ephemeral per-job credentials — OIDC short-lived tokensThe structural move from long-lived secrets to per-job credentials issued by an identity federation. OIDC tokens in the cloud, projected ServiceAccount tokens in Kubernetes, dynamic secrets from Vault. Each credential lives for one job; the blast radius is the job duration.→
- Least privilege in GitOps — what the controller can do; what it cannotHow a GitOps controller (Argo CD, Flux) maps to Kubernetes RBAC. The application controller, the repo server, and the application set controller each have distinct ServiceAccount scopes. Reconcile and prune permissions are the privilege ceiling.→
- The least-privilege decision — what to lock down firstHow to prioritise least-privilege work when not everything can be tightened at once. The decision is driven by blast radius, frequency of use, time-to-revoke, and the cost of a false negative in the audit. The first lock-down is the production deploy identity; the last is the read-only validator.→
Part XCII
Protected Environments
6 checks
- The protected environment pattern — the platform-side guard for production deploysHow protected environments work as a first-class platform construct in GitHub Actions and GitLab CI; the difference between an environment and a deployment target; the canonical shape of a production environment with all three rules; why the construct must live in platform configuration, not workflow code.→
- Required reviewers and the wait timer — the human gate that buys the team five minutesHow required reviewers and the wait timer compose into the human side of the production gate; how reviewers are named (users and teams); what the wait timer catches that reviewers do not; the audit trail each rule produces; why the timer set to zero is no timer at all.→
- Environment secrets and isolation — secrets scoped to a deployment targetHow environment-scoped secrets work in GitHub Actions and GitLab CI; the difference between repository secrets, environment secrets, and organisation secrets; why the production secret must not be readable from a PR build; the isolation boundary the environment secret draws; the real CLI commands for managing environment secrets.→
- Deployment branch restriction — only the right source can deploy to productionHow the deployment branch restriction rule works in GitHub Actions and GitLab CI; how to limit deploys to specific branches, tag patterns, or commit patterns; the source-of-truth gate that prevents a feature branch from writing to production; the real CLI commands for configuring the branch restriction; how the rule composes with tag-triggered workflows.→
- Bypass and bypass actors — the role-based exceptions that weaken the gateHow the bypass mechanism works in GitHub Actions and GitLab CI; who is permitted to bypass the gate (bypass actors in GitHub, allowed tiers in GitLab); why the bypass actor list must be the smallest possible set; the audit trail a bypass leaves; the production discipline of never bypassing the gate.→
- Environment policy as code — terraform-github-actions and terraform-gitlab-providerHow to manage protected environments and their rules as code through the terraform-github-actions and terraform-gitlab-provider Terraform providers; why the protected environment configuration itself must be version-controlled and reviewed; the canonical pattern of a repository-of-record for environment policy; drift detection and remediation; the production discipline of treating environment policy like infrastructure.→
Part XCIII
Credential Rotation
6 checks
- Why rotate credentials — the cost of long-lived secretsWhy every long-lived secret has a finite exposure window; the cost of holding a credential unchanged for years; the leak-surface accumulation that makes age the dominant risk variable; the discipline of treating rotation as routine hygiene rather than incident response.→
- Deploy keys and SSH key rotation — the lifecycle of a read/write credential for a Git remoteHow deploy keys work; the read-only versus read/write distinction; the rotation lifecycle of an SSH keypair (generate, install, cut over, revoke, retain); the GitHub CLI commands for adding and removing deploy keys; the operational checklist for a zero-downtime rotation.→
- Token rotation cadence — PAT, OAuth, and OIDC tokensThe three token types a CI system holds: personal access tokens, OAuth app tokens, and OIDC tokens; the lifetime and rotation cadence appropriate to each; the gh auth token refresh command; the discipline of short-lived tokens superseding long-lived ones.→
- Cloud credential rotation — IAM access keys and OIDC federationHow IAM access keys work in AWS; the rotation lifecycle of an access key (issue, distribute, cut over, retire); the aws iam create-access-key CLI command; the OIDC federation pattern that replaces access keys with short-lived STS credentials; the operational checklist for both rotation and migration.→
- Registry credential rotation — push and pull credentialsHow container registries authenticate pushes and pulls; the difference between image-pull secrets and image-push credentials; the rotation lifecycle for registry credentials; the per-namespace pull-secret pattern; the kubectl commands for managing image pull secrets.→
- GitOps controller credential rotation — the controller's repository and cluster accessHow a GitOps controller authenticates to its source repository and to the target cluster; the rotation lifecycle for the controller's two credentials; the Flux and Argo CD credential sources; the operational checklist for rotating the controller without losing reconciliation continuity.→
Part XCIV
Incident: Secret Leak
6 checks
- The incident arrives — first alert, first scope, first ten minutesHow a confirmed secret leak reaches the on-call engineer; the difference between an alert and an incident; the scope questions that determine the next hour of work.→
- Revoke or rotate first — the first action and the rationale that forces itThe first remediation action after scope: disable, rotate, or revoke. The consumer-update sequence that prevents production from breaking during the rotation. The audit fields that survive the timeline.→
- Assess exposure — who had access and what was used in the windowAfter the credential is disabled, the next step is exposure assessment: enumerating every system that held a copy of the repository in the leaked state, every consumer that authenticated with the credential, and every external surface that may have indexed the commit.→
- Inspect usage — log analysis and the timeline that survives the auditThe use-window audit: every API call made by the credential between commit time and disable time, with the resources touched and the calling IP. The timeline artefact the security team uses to distinguish exposure from compromise.→
- Remove repository exposure — history cleanup with filter-repoThe history rewrite that removes the leaked credential from the repository: git filter-repo with --replace-text, the reflog expiry and gc prune that complete the cleanup, the force-push that lands the rewrite, the coordination with forks and mirrors that must follow.→
- Prevent recurrence — the controls that should have caught itThe prevention controls that close the door the leak walked through: pre-commit hooks, server-side secret scanning, CI scan jobs, branch protection, and secret manager adoption.→
Part XCV
Incident: Compromised Runner
6 checks
- The runner incident arrives — detection, initial scope, first ten minutesHow a confirmed CI/CD runner compromise reaches the on-call engineer; the difference between a runner anomaly and a confirmed compromise; the four scope questions that determine the next hour of work.→
- Isolate the runner — remove from pool, drain new jobs, preserve evidenceThe isolation sequence for a confirmed runner compromise: drain, snapshot, stop, deregister. The evidence-preserving commands for self-hosted, Kubernetes, and systemd runners. The order in which isolation commands land.→
- Revoke credentials — what credentials did the runner have, and how to revoke themThe credential inventory for a CI/CD runner: GITHUB_TOKEN, cloud access keys, registry tokens, SSH deploy keys, Vault tokens. The disable-rotate-revoke sequence applied to each class. The audit fields that survive the timeline.→
- Determine artifact impact — what artifacts did the runner produce, and which are suspectThe artifact inventory for a CI/CD runner: container images, build artifacts, Terraform plans, SBOMs, signed attestations. The per-job forensic trace that links each artifact to the runner that produced it. The decision rule for which artifacts to invalidate.→
- Invalidate and rebuild — the rebuild from clean source, on a clean runnerThe rebuild sequence: drain the invalidation, re-trigger from clean commits, build on a clean runner, re-sign with a new key, re-deploy. The verification gates that prove the rebuilt artifact is clean. The post-rebuild audit fields.→
- Prevent recurrence — ephemeral runners, network isolation, attestation per jobThe prevention controls for a runner compromise: ephemeral runners destroyed after every job, network isolation that constrains the runner to a known egress list, per-job attestation that proves which runner produced which artifact.→
Part XCVI
Incident: Malicious Dependency
6 checks
- The supply-chain incident arrives — detection and initial triageHow a confirmed malicious-dependency alert reaches the on-call engineer; the difference between an advisory, a scanner hit, and a confirmed compromise; the four scope questions that determine the next hour of work.→
- Stop affected builds — pause CI and pin to known-goodThe pause sequence for a confirmed malicious-dependency incident: stop the affected CI workflows, block the malicious version range at the registry or proxy, pin every consumer to a known-good version. The order of the stop commands and the audit fields.→
- Identify impacted artifacts — what used the bad versionThe artifact inventory for a malicious-dependency incident: every container image, build artifact, and deployable produced from a lockfile that resolved to the malicious version. The per-build forensic trace that links each artifact to the lockfile and the commit. The decision rule for which artifacts to invalidate.→
- Rotate credentials and revoke tokens — what the malicious code touchedThe credential inventory for a malicious-dependency incident: every secret a malicious install hook could have read, every token the artifact had access to, every deploy key the artifact used. The disable-rotate-revoke sequence applied to each class. The audit fields that survive the timeline.→
- Rebuild trusted artifacts — clean source, pinned versionsThe rebuild sequence for a malicious-dependency incident: re-trigger from clean commits, regenerate the lockfile with the pinned known-good version, build on a clean runner, re-sign with a new key, re-deploy. The verification gates that prove the rebuilt artifact is clean.→
- Improve dependency controls — lockfile enforcement, scan-on-PRThe prevention controls for a malicious-dependency incident: lockfile enforcement with hash pinning, scan-on-PR with `npm audit` and `pip-audit`, registry allowlists, SBOM baseline diff. Each control catches a compromise path the previous control did not address.→
Part XCVII
CI/CD Disaster Recovery
6 checks
- The CI/CD DR question — what fails, what survives, what to rebuildHow a CI/CD disaster is defined; what survives a regional outage (repositories, IaC, local clones); what fails (control plane, runners, secret store, artifact registry); the recovery order dictated by the dependency graph.→
- Recovering the control plane — hosted service versus self-hostedHow the control plane is recovered in a CI/CD disaster: the hosted path (recover the org, repos, workflows, secrets from backups or imports) and the self-hosted path (restore config from IaC). The Terraform and Ansible commands that recreate the orchestrator.→
- Recovering runners — the rebuild plan; the registration tokensHow self-hosted runners are rebuilt during a CI/CD disaster: the runner image, the registration command, the runner scope (repo, org, enterprise). The ./config.sh --url --token registration pattern and the secrets the runner needs to fetch.→
- Recovering secrets — the secret store; the rotationHow the secret store is recovered during a CI/CD disaster: the external secret manager (Vault, AWS Secrets Manager), the control-plane secret store (GitHub Actions Secrets, GitLab CI Variables), the rotation discipline, and the gh secret set command for re-entry.→
- Recovering the artifact registry — the backup planHow the artifact registry is recovered during a CI/CD disaster: the registry backup (Harbor, ECR, GHCR), the cross-region replication, the restore sequence, and the rebuild-from-source path when no backup exists.→
- The disaster recovery drill — the quarterly rehearsalHow the CI/CD DR plan is rehearsed: the tabletop exercise, the live regional failover drill, the RTO and RPO measurements, and the post-drill improvements. The cadence, the script, and the artefacts the drill must produce.→
Part XCVIII
Git Hosting Failure
7 checks
- The Git hosting failure scenario — what happens when the platform is downHow a Git hosting outage is defined; the four scopes of failure (control plane, API, git protocol, web UI); how engineers detect a regional failure vs a partial degradation; the production signals (status page, DNS, TLS, API latency) that drive the failover decision.→
- Multi-remote failover — two remotes, automatic failoverHow a single clone can hold the canonical remote plus a backup remote; the git remote add backup pattern; git push --all backup for the failover push; the failover script that flips the canonical remote when the vendor is unreachable.→
- Mirror repositories and CDN — the read-only fallbackHow a read-only mirror built with git clone --mirror serves as the read path for CI and GitOps during a forge outage; the CDN that fronts the mirror for low-latency clones; the push-mirror sync that keeps the mirror current.→
- Local bare repository fallback — the last resortHow a local bare repository serves as the last-resort Git service when the mirror and the canonical are both unreachable; the git init --bare bootstrap; serving the bare repo over SSH or local filesystem; the limits of the local fallback.→
- Self-hosted Git as fallback — Gitea, GitLab CEHow a self-hosted Git forge (Gitea or GitLab CE) serves as the long-term fallback when the hosted forge is unreachable; the bootstrap, the migration of repositories, the push-mirror sync back to the hosted forge when it recovers.→
- Recovery time objectives — what RTO is acceptable for a Git hosting failureHow the RTO for a Git hosting failure is derived from the team workflow that depends on the forge; the four tiers of RTO (mirror-only, multi-remote failover, self-hosted fallback, local bare); the cost of each tier and the cadence of rehearsal.→
- Self-hosted forge and CI control-plane lifecycleOperate GitLab, Jenkins, Forgejo, or Gitea as production control planes: dependencies, TLS, SSO, SMTP, storage, backups, restore, upgrades, HA, observability, and disaster recovery.→
Part XCIX
Artifact Registry Failure
6 checks
- The registry failure scenario — when pulls return 503 and the deploy falls overHow an artifact registry outage is defined; the four scopes of failure (manifest endpoint, blob endpoint, auth, write API); the symptoms in the deploy log; how a registry failure is distinguished from a network failure.→
- Impact on deployment — what stops working, what keeps workingHow a registry outage cascades into the deployment pipeline: what fails (image pull, Helm chart pull, signature verification, GitOps sync); what keeps working (cached images, source repositories, drift detection); the partial-degradation model that distinguishes critical from non-critical paths.→
- Registry replication — the multi-region strategyCross-region replication for the artifact registry: pull-through caches, Harbor replication policies, ECR cross-region replication, GHCR mirrors; the replication lag and the RPO it defines; how to verify a replica is current before the failover flip.→
- Immutable tag fallback — when :latest is the only optionHow to gracefully degrade when the registry is unreachable: pinned-digest deploys hold the line; :latest becomes a temporary fallback for non-critical workloads; the rules for using mutable tags under failure conditions; the risks and the audit trail the fallback must leave.→
- Registry monitoring and alerting — the metrics that matterThe four production metrics for an artifact registry (pull latency, pull error rate, storage used, replication lag); the alert thresholds; the SLOs derived from these metrics; the dashboard that exposes them; the on-call runbook the alerts fire into.→
- The registry drill — the quarterly rehearsalHow the registry DR plan is rehearsed: the tabletop exercise for the registry runbook; the live regional failover that promotes the cross-region replica; the verification of reachability, digest presence, and signature trust; the artefacts the drill must produce and the cadence.→
Part C
Runner Capacity
6 checks
- Runner pool sizing — the workload model and the concurrency limitHow to size a runner pool from workload data: jobs per day, peak-hour arrival rate, average duration, and the concurrency limit that follows from Little's Law.→
- Queueing and wait times — what happens when jobs exceed capacityWhat happens to jobs when arrival rate exceeds runner capacity: the queue grows, wait time rises non-linearly, and SLA breaches cascade. How to recognise saturation early and what metrics to watch.→
- Ephemeral runner sizing — per-job cost and the speed-versus-money trade-offThe cost of a single ephemeral runner: compute time, control-plane overhead, image-pull latency, and the speed-versus-money trade-off between larger and smaller runner shapes.→
- Runner autoscaling with ARC — Kubernetes-based scale to zeroHow Actions Runner Controller scales runner pods in response to queue depth, what HPA metrics look like for runner pods, the kubectl commands to inspect autoscaling state, and the operational knobs that matter.→
- Cost of hosted versus self-hosted runners — the financial trade-offThe per-minute cost of GitHub-hosted runners, the cost components of self-hosted runners, where the break-even lives, and the right method for comparing the two options on a workload.→
- The capacity plan — the document that turns numbers into a budgetWhat the capacity plan contains: workload projections, SLOs, headroom rules, alerting thresholds, the financial commitment, and the review cadence that keeps the document alive.→
Part CI
Pipeline Performance
6 checks
- Measure before optimising — the metrics and the baselineA pipeline cannot be optimised until it is measured. The metrics that matter are wall-clock duration, queue time, cache-hit rate, runner cost per job, and flakiness. The baseline is the numbers recorded before any change, so the optimisation can be evaluated against the same workload.→
- Cache and sharding — the two levers for pipeline durationCaching reduces redundant work within a single job by reusing previously produced artefacts; sharding splits one job into N parallel jobs to reduce wall-clock duration. They are the two main levers for cutting pipeline wall-clock duration, and they work on different parts of the cost model.→
- Parallelism and matrix — fan-out, fan-in, and the trade-offsGitHub Actions matrix strategy fans a job out across N combinations of variables. Each combination runs on its own runner. The fan-out gives parallel execution; the fan-in gives a single pass/fail signal. The trade-offs are runner cost, cache writes, and the longest-shard-dominates wall-clock.→
- Artifact and layer caching — Docker layer cache and build cacheContainer builds have their own caching model: the Docker layer cache reuses unchanged layers across builds; the build cache (BuildKit, Buildx) reuses intermediate artefacts by content hash. Both cut wall-clock duration on the build step itself, distinct from dependency caches for npm or pip.→
- Skip when nothing changed — path filters and conditional executionA workflow that runs on every push regardless of which files changed wastes CI minutes. Path filters (paths, paths-ignore) restrict the trigger to relevant files; conditional execution (if:) restricts individual steps. Together they skip the work that does not need to happen.→
- The performance loop — measure, optimise, re-measurePipeline performance is not a one-time optimisation. It is a loop: measure the baseline, pull one lever, re-measure against the same workload, compare, decide whether to keep the change. The loop runs whenever the workload changes, the runner pool changes, or a new lever becomes available.→
Part CII
Large Repository Performance
6 checks
- The large repository problem — clone, fetch, and checkout at scaleA large repository is slow in three different ways: the three timing profiles scale with different parts of the anatomy and demand different baselines and different mitigations.→
- Binaries and large files — what does not belong in GitGit is a delta-compressing text store. Compiled binaries, image assets, model weights, and database dumps do not delta-compress, do not diff in review, and do not shrink. Identify them, route them out.→
- Git LFS fundamentals — pointers and object storageGit LFS stores large file contents in a separate object store and commits small pointer files. The pointer indirection shifts the bandwidth cost from clone to LFS pull, trades git-native history for an external store, and demands deliberate workflow integration.→
- Shallow clones and partial checkouts — fetching less historyA shallow clone truncates history at a depth; a partial clone defers blobs on demand. Both cut initial bandwidth; both impose limits on what the client can do. The trade is fetch cost versus local capability.→
- Sparse-checkout and filter — only what you needA sparse-checkout restricts the working tree to a subset of paths; a filter restricts the packfile to a subset of objects. Combining them is the standard monorepo working-tree pattern; each adds capability independently.→
- Repository design discipline — prevention versus curePerformance is cheaper when applied at repo creation than retrofitted. Discipline is preventive: gitignore at first commit, pre-commit hooks from day one, attributes in place, push protection. Cure is git filter-repo, which breaks pinning.→
Part CIII
Infrastructure Repository Anti-Patterns
6 checks
- Committed secrets in IaC — credentials that should never reach GitThe anti-pattern of hardcoded credentials, tokens, and keys inside Terraform, Ansible, or Kubernetes manifest files; why secret scanning catches what discipline should have prevented; the rotation response when a secret is already committed.→
- Terraform state in Git — the file that does not belong in the repositoryWhy terraform.tfstate, terraform.tfstate.backup, and .terraform/ directories should never be committed; what state actually contains; the remote-backend alternatives that turn state into a controlled artifact rather than a tracked file.→
- Generated files in the repository — why build output does not belong in GitThe anti-pattern of committing .terraform/, dist/, node_modules/, target/, *.tfplan, and other generated artifacts; the .gitignore discipline that keeps a working tree reviewable; the pre-commit hooks that enforce it.→
- No ownership or CODEOWNERS — the repository without a reviewer mapThe anti-pattern of a repository where every engineer can approve every change; why required-reviewer discipline lives in CODEOWNERS plus branch protection; the failure mode when ownership is implicit rather than enforced.→
- Direct production pushes — bypassing the merge gateThe anti-pattern of pushing changes to the default branch without a pull request, or skipping CI to land a hotfix; why direct pushes to the protected branch break the audit trail; the consequences when an incident must reconstruct who, what, and why.→
- Mutable dependencies and loose tags — the tag that is not a contractThe anti-pattern of floating module versions, mutable image tags, and unpinned provider sources; why a tag is a mutable pointer, not an immutable reference; the pinning disciplines that turn dependencies into content-addressed artifacts.→
Part CIV
CI/CD Anti-Patterns
6 checks
- Permanent privileged runners — long-lived self-hosted runners with standing credentialsThe anti-pattern of an always-on self-hosted runner holding sudo, a Docker socket, or long-lived cloud credentials; ephemeral runners with OIDC federation as the fix.→
- Secrets in logs — why masking is not enoughThe anti-pattern of relying on the forge log-masker to hide secrets; the three ways secrets end up in logs anyway (URL-embedded credentials, base64 blobs, file redirects); ::add-mask as defence-in-depth, not primary control.→
- Mutable dependencies — the floating-tag contractThe anti-pattern of resolving dependencies at job time by tag or branch rather than by lockfile or digest; how a yanked or replaced upstream version turns yesterday's build into today's supply-chain compromise; lockfiles and digest pinning as the fix.→
- No artifact identity — relying on tags instead of digestsThe anti-pattern of referencing artifacts by tag rather than by digest; how a re-tagged or substituted image produces a deployment that points at bytes the team never reviewed; digest pinning and signed provenance as the fix.→
- Rebuilding per environment — why promotion, not re-buildThe anti-pattern of running a separate build for staging and production; how re-building per environment produces a different artifact in production than the one the team reviewed in staging; build once, promote the artifact across environments as the fix.→
- Unsafe auto-apply — auto-merge without guardrailsThe anti-pattern of wiring a CI pipeline or GitOps controller to apply changes to production automatically on merge; the controls required to make auto-apply safe (required reviewers, plan reconciliation, drift detection, environment protection rules).→
Part CV
GitOps Anti-Patterns
6 checks
- Cluster-admin everywhere — the controller holds the keys to the clusterThe anti-pattern of binding a GitOps application controller to a cluster-wide ClusterRole with full verbs; the scoped ServiceAccount per project and the per-kind verbs that bound the blast radius when a manifest, a webhook, or an impersonation bug fires.→
- Unsafe prune — automated deletion without a diff, a sandbox, or an opt-outThe anti-pattern of enabling Argo CD automatic pruning or Flux spec.prune at the Application or Kustomization level without pairing it with dry-run diff, sandbox verification, and the Prune=false per-Application and per-resource opt-out. The safety stack that makes prune a controlled operation.→
- Plaintext secrets in Git — the secrets that the repository remembersThe anti-pattern of committing Kubernetes Secrets, .env files, or cloud credentials in plaintext to a GitOps repository; the sealed-secret, SOPS, and external-secret reference patterns that keep credentials out of the audit trail of who can clone the repo.→
- Uncontrolled drift — when the cluster forgets GitThe anti-pattern of running a GitOps controller without self-heal, without drift detection alerts, and without a procedure for resolving out-of-band changes; self-heal, IgnoreDifferences, and the reconciliation budget that turn drift from a silent failure into a managed one.→
- No environment separation — one controller, one repo, every clusterThe anti-pattern of running a single GitOps controller against every environment with a single repository and no environment boundaries; the per-environment AppProject, per-cluster controller, and promotion-by-ref model that turn environment separation from an accident into an architecture.→
- The GitOps discipline — the synthesisThe synthesis of every anti-pattern into the GitOps discipline: cluster-scoped controller, dry-run diff and sandbox-verified prune, encrypted or referenced secrets, self-heal with IgnoreDifferences and a reconciliation budget, per-environment AppProjects with per-cluster controllers and promotion-by-ref. The whole architecture as one consistent set of choices.→
Part CVI
Change Management
6 checks
- The change question — six questions a change must answer before it is appliedThe six questions a change must answer before it is applied: what, why, when, who, where, how. Why "how" is the new question that extends the five-question audit. How the questions translate into a pull request template that enforces audit-grade answers.→
- The change record — the artefact that survives the changeThe change record as the durable artefact that survives the change: the issue, the pull request, the linked discussion, and the merged commit. Why the change record is in Git, not in a wiki. How to design a change-record template that enforces the six questions and survives an audit six months later.→
- The change author and approver — the human links in the audit chainThe author and the approver as the two human links in the audit chain; the four-eyes principle; CODEOWNERS as the machine-readable mapping from path to approver; the difference between technical approval and change-board approval; how required reviewers and branch protection enforce separation of duties.→
- The change rollback plan — what we do when the change goes wrongThe rollback plan as the sixth question of the change question; three rollback patterns (revert PR, blue/green, feature flag); how to write a rollback plan in the PR that the on-call can execute at 02:00; pre-mortem as the discipline that surfaces the rollback before the change is applied.→
- The change communication — who needs to know, when, and howThe communication plan as part of the change record; the audience matrix (operators, stakeholders, customers, auditors); pre-change, during-change, and post-change communication; status page integration; why change communication is in the PR description, not in a separate announcement.→
- The change postmortem — what we learned and what we change nextThe change postmortem as the closing artefact of the change record; blameless postmortem rules; the action item as a new pull request; the postmortem repository as the durable home for organisational learning; why the postmortem is published, not archived.→
Part CVII
Production Infrastructure Delivery Architecture
6 checks
- The reference architecture — the diagram and the componentsThe production delivery architecture as a single diagram; the seven components and four planes; what the reference architecture is and is not; why the engineer's laptop is outside the trust boundary.→
- The trust boundaries — where the system trusts and does notThe seven trust boundaries in the reference architecture; the difference between a boundary and an edge; what crosses each boundary and how it is verified; common attacks at each boundary; the boundary between the laptop and the source repo as the most violated.→
- The identity flow — from engineer to commit to productionThe chain of identity from engineer to running pod; SSH key at the laptop, GitHub user at the repo, OIDC workload at CI, IRSA role at the cluster, ServiceAccount at the namespace; why identity delegation replaces shared credentials; how OIDC federation eliminates long-lived cloud secrets.→
- The artifact flow — from source to image to deploymentThe path of an artefact from source commit to running container; source to layered image, cosign signature, provenance attestation, SBOM; registry by digest; manifest by digest; SLSA Build L3 track; why the digest is the artefact identity.→
- The deployment flow — from plan to apply to runtimeThe four steps of the deployment flow: plan, approve, apply, verify; how the plan step records intent; how the approve step records the decision; how the apply step reconciles the cluster to the manifest; how the verify step confirms the runtime matches the plan; the GitOps reconcile loop as a continuous deployment flow.→
- The observability flow — what we see and what we do notThe four signals of observability: metrics, logs, traces, audit trail; what observability answers and what it does not; the gap between observability and auditability; the unification of receipt and observation into a queryable audit trail; why observability cannot replace auditability.→
Part CVIII
Infrastructure-as-Code Integration
6 checks
- The IaC tooling landscape — Terraform, Ansible, Kubernetes, Docker, and the boundariesA map of the four dominant infrastructure-as-code tool families; what each is for, what each cannot do, where their pipelines meet, and why the boundaries between them matter for a CI/CD/GitOps architecture.→
- Terraform in the pipeline — plan as artefact, apply as gateWhere Terraform fits in the CI/CD/GitOps flow; the plan file as the deliverable of CI; remote state as the source of truth; the drift problem; how the apply step differs from the plan step.→
- Ansible in the pipeline — the role, the limits, the dry-runWhere Ansible belongs in the CI/CD/GitOps flow; why idempotent execution is a different posture from plan-then-apply; how ansible-playbook --check fits into CI; the cases where Ansible is the right tool and the cases where it is the wrong one.→
- Kubernetes in the pipeline — manifest validation and the GitOps syncWhere Kubernetes fits in the CI/CD/GitOps flow; manifest validation as a CI gate; the GitOps controller as the production entry point; the difference between kubectl apply from CI and a controller-driven sync.→
- Docker and OCI in the pipeline — the container build and the registryWhere Docker and OCI artefacts fit in the CI/CD/GitOps flow; the build as a deterministic, content-addressable pipeline stage; the registry as the artefact store; the digest as the artefact identity; the relationship between build, sign, push, and pull.→
- Network IaC in the pipeline — the missing pieceWhy network configuration lags behind cloud, host, and workload configuration in CI/CD adoption; the vendor API fragmentation problem; the analysis-versus-provision gap; how Batfish, NAPALM, and Ansible network modules fit into a pipeline; what production discipline looks like when the tool is imperative and the device is shared.→
Part CIX
Terraform Delivery Pipeline
6 checks
- The Terraform delivery pipeline — an end-to-end viewThe complete shape of a production Terraform pipeline: commit, fmt, validate, lint, security, plan-as-artifact, policy gate, human approval, apply, drift detection. What each stage owns and what it deliberately does not own.→
- fmt and validate in CI — the cheapest checksPlacing terraform fmt -check -recursive -diff and terraform validate -json at the front of the Terraform pipeline. Why they run before init, why init uses -backend=false on the PR job, and why warnings are surfaced separately from errors.→
- tflint and fmt-deep in CI — the lint layerPlacing tflint --init and tflint --recursive as the third stage of the Terraform pipeline. How tflint differs from terraform validate: provider-aware rules, deprecated attribute detection, deep format checks. Why this stage runs after fmt+validate and before security scanning.→
- tfsec and checkov in CI — the security layerPlacing tfsec and checkov as the fourth stage of the Terraform pipeline. What each scanner catches, how their rule sets differ, why the security stage runs after tflint and before plan, and how to wire findings as build failures.→
- Plan as an artifact — the review surfaceThe terraform plan -out=tfplan -input=false -lock-timeout=300s command, the binary plan file as the review artefact, terraform show -json tfplan as the machine-readable form, and the SARIF/Slack/PR-comment wiring that makes the plan the surface humans actually review.→
- Apply and drift detection — the production boundaryThe apply stage consuming the saved plan artefact, the production-boundary credentials, the lock-timeout contract, and the scheduled drift-detection job that detects divergence between cloud state and configuration between deploys.→
Part CX
Ansible Delivery Pipeline
6 checks
- The Ansible delivery pipeline — an end-to-end viewThe complete shape of a production Ansible delivery pipeline: commit, yamllint, ansible-lint, ansible-playbook --syntax-check, molecule converge, molecule verify, idempotency check, approval, apply. What each stage owns and what it deliberately does not own.→
- YAML and lint in CI — yamllint and ansible-lint as the first gatesWhy the first two gates in an Ansible pipeline are yamllint and ansible-lint; what each catches; how to configure them so they fail closed; how the CI scope differs from the editor scope.→
- Syntax check and Molecule — the static and dynamic checkWhy ansible-playbook --syntax-check is the last static gate before Molecule; what Molecule converge and Molecule verify prove; how the boundary between static and dynamic checks maps to the pipeline.→
- Staged validation environments — dev, staging, prodWhy Ansible delivery requires per-environment inventories; how inventories for dev, staging, and production differ; how the same role promotes through stages without rebuilding.→
- Idempotency and change detection — the production disciplineWhy idempotency is what makes Ansible safe to re-run; how change detection extends idempotency from a property of the role to a property of the fleet; what production discipline looks like when both are in place.→
- Secrets and runtime variables — the safe-handling patternHow Ansible Vault, environment variables, and runtime variables interact; how the pipeline keeps secrets out of the repository, out of CI logs, and out of Molecule scenarios; the boundary between committed defaults and runtime-resolved secrets.→
Part CXI
Kubernetes Delivery Pipeline
6 checks
- The Kubernetes delivery pipeline — an end-to-end viewThe complete shape of a production Kubernetes delivery pipeline: source commit, image build, manifest render, OCI package, policy gate, GitOps sync, reconcile loop, runtime feedback. What each stage owns and which boundary its credentials live in.→
- Manifest render and validate — the CI side of the pipelineHow CI turns a commit and a values file into a rendered, validated manifest bundle; helm template and kustomize build as the two render verbs; kubeconform and conftest as the two validation gates; what each catches and what each deliberately does not.→
- Helm or Kustomize build — the packaging decisionWhen to pick Helm versus Kustomize; how each packages its render output; the OCI artefact as the unit the GitOps controller consumes; how values files and overlays compose per environment; and why the choice has to be made deliberately, not by inertia.→
- Container image build and push — the immutable artefactHow CI produces a signed container image with a digest; buildx multi-platform builds; layer caching; SBOM and cosign as the attestations; how the digest lands in the rendered manifest; and why image promotion is digest promotion.→
- GitOps sync and reconcile — the production entry pointHow Argo CD and Flux pull the OCI manifest bundle, render it server-side, apply it to the cluster with server-side apply, and reconcile on a loop; sync windows and self-heal; drift detection and the OutOfSync state; why the controller, not kubectl, is the production entry point.→
- The CD loop closing — observability and feedbackHow runtime signals - sync status, application health, rollout events, alert noise - close the loop on the Kubernetes delivery pipeline; SLOs for rollout health; notifications that open issues or trigger reverts; why the loop only counts when observation drives the next commit.→
Part CXII
Container Delivery Pipeline
6 checks
- The container delivery pipeline — end-to-end viewThe full pipeline from a source commit to a deployed, signed, attested image: test, build, scan, sign, push, deploy. What each stage produces, what each stage consumes, and where the chain of trust begins and ends.→
- Source to test — lint, unit, integrationThe first stage of container delivery: the commit moves from source to a green test result. Lint as a fast filter, unit tests as the cost-of-fix minimiser, integration tests as the closest thing to a real environment.→
- Build and OCI image — BuildKit, multi-stageThe build stage produces a content-addressed OCI image from a green source. BuildKit features (cache mounts, secret mounts, multi-stage), how the per-commit tag is set, and why the digest, not the tag, is the artefact.→
- SBOM and vulnerability scan — syft and trivyTwo observation stages that consume the build digest and produce attestations: syft for the SBOM inventory and trivy for the CVE report. Why both are needed, where they run in the pipeline, and what their results cannot do.→
- Sign and verify — cosign and the policyThe signing stage closes the chain: cosign signs the digest with the CI runner identity, attaches SBOM and vuln report as attestations, and produces a verification policy that the admission controller enforces before any pod can pull.→
- Registry and deploy — the artefact goes homeThe final stage of container delivery: the signed, attested image is pushed to the registry by digest, the digest lands in the rendered manifest, and the GitOps controller syncs the cluster. Promotion is digest promotion; rollback is digest rollback.→
Part CXIII
Observability Integration
6 checks
- Observability as deployment marker — the change-cause annotationWhy every deploy must be a visible mark on the observability timeline; the change-cause annotation as the bridge between the pipeline and the metrics; how a Kubernetes Deployment can carry a deployment annotation directly; the difference between a deploy marker and an incident marker.→
- Deploy events and correlation — the timeline viewHow the pipeline emits a deploy event into the observability timeline; the difference between an annotation (point in time) and an event (typed record); the join between commit, deploy, and incident; the timeline view as the on-call interface.→
- Performance change detection — did the deploy cause the regressionHow the deploy marker and the metric histogram together identify a deploy-caused regression; the before-and-after comparison window; the role of exemplars in linking a slow request to the deploy that introduced the regression; the false-positive modes that must be filtered out.→
- Error rate and deploys — the deploy-to-error graphHow the deploy marker joins the error-rate panel; the per-deploy error budget view; the structure of a deploy-to-error graph; how the error rate is decomposed by version label; the alert thresholds that must be calibrated to deploy frequency.→
- Audit logs from the pipeline — who did whatThe audit log as the record of every pipeline action; the link between the CI audit log and the deployment audit chain; the fields that must be captured for postmortem use; the retention policy that survives personnel changes; the operational discipline that prevents audit log tampering.→
- The observability of observability — meta-monitoringThe discipline of monitoring the monitoring system itself; the signals that must be observable about the observability backend; the failure modes of an observability system that is silently degraded; the meta-alerts that distinguish a noisy observability system from a silent one; the structural patterns that keep observability honest.→
Part CXIV
Deployment Markers
6 checks
- The deployment marker pattern — leaving a trailWhy every deploy must leave a deliberate marker on the workload object, the metric stream, and the log line; the three surfaces a marker covers; why a deploy without markers is operationally indistinguishable from a phantom change.→
- The change-cause annotation — the kubectl annotation and the OpenTelemetry attributeThe kubernetes.io/change-cause annotation as the canonical change-cause marker on the Deployment object; the OpenTelemetry service.version and deployment.environment attributes as the parallel change-cause surface on the telemetry stream; the dual write that ties the workload identity to the telemetry identity.→
- Deploy timestamp and version labels — the runtime identityThe app.kubernetes.io/version label as the immutable runtime identity of the workload; the deploy timestamp as the wall-clock moment the change landed; the kubectl label write that propagates the identity to pods and metric labels; why a mutable label breaks the trail.→
- The deploy event log — the platform-side auditThe deploy event log as the platform-side record of every deploy; ArgoCD and Flux application events, Kubernetes Events API, and the central audit backend; what the event log preserves that the annotation and the label do not; the retention and tamper-resistance that the event log requires.→
- Correlation with metrics and logs — the joined queryHow the deploy marker joins to the metric stream and the log line; the PromQL query that filters by version label; the LogQL query that filters by service.version attribute; the joined query that produces the postmortem view; the trace-to-deploy pivot that closes the correlation loop.→
- The 3 AM incident and the deploy marker — the real testThe full incident reconstruction using the deploy marker; how the annotation, the label, and the event log come together at 3 AM; the timeline view that compresses the investigation; the rollback decision that the marker makes reliable; the discipline that turns a marker-equipped team from minutes-of-MTTR into seconds.→
Part CXV
Production Operating Model
6 checks
- The application and platform split — who owns whatThe application/platform split as the foundational operating-model decision; what the application team owns, what the platform team owns, and the named boundary that makes a delivery system auditable.→
- The infrastructure team role — the boundariesThe infrastructure team as the owner of the underlying substrate - cloud accounts, network, storage, base images; the boundaries with the platform team; what the infrastructure team produces for the platform team to consume.→
- The platform team role — the shared capabilityThe platform team as the consumer of the infrastructure contract and the producer of a delivery capability; the internal customer; the platform as a product; what the platform team does not do.→
- The security team role — the advisory and auditThe security team as an advisor and auditor, not a gatekeeper; the security policy as code; the security review as a PR comment, not a meeting; the security audit as a queryable artefact; what the security team does and does not own.→
- The reviewer and approver role — the human gateThe reviewer and approver as the human gate in an otherwise automated pipeline; the four-eyes principle; the CODEOWNERS file as the reviewer source of truth; what the reviewer checks, what the approver checks, and why they are different roles.→
- The shared responsibility model — the synthesisThe shared responsibility model as the synthesis of infrastructure, platform, security, application, reviewer, and approver roles; the change as the unit that activates every role; how the model appears in a single pull request; what the operating model produces and what it costs when it is missing.→
Part CXVI
Governance Without Bureaucracy
6 checks
- The control versus the bureaucracy — the trade-offDistinguish a control from a bureaucracy; recognise why controls drift into ceremony when the cost of the control exceeds the loss it prevents; identify the failure mode of "control without feedback" and the trade-off a governance programme must make explicit.→
- Structural versus procedural controls — the differenceDistinguish structural controls (the system prevents the loss) from procedural controls (the human is asked to prevent the loss); recognise why structural controls scale and procedural controls drift; identify which classes of loss require which class of control.→
- Policy as code — OPA, Conftest, and KyvernoWhat policy as code is; the three engines an infrastructure team will encounter (OPA and Rego, Conftest for manifests, Kyverno for Kubernetes); how to write a policy, run it locally, and wire it into CI; the trade-off between general-purpose and Kubernetes-native engines.→
- Shifted-left controls — catching at PR time, not deploy timeWhat shifted-left means for governance controls; the cost difference between a failure caught at PR time and a failure caught at deploy time; how to wire shifted-left controls (linters, policy-as-code, actionlint, secret-scanners) into the pull-request pipeline; the rules for keeping the leftward shift from overwhelming the PR.→
- Continuous compliance — the automated auditWhat continuous compliance is; the difference between a point-in-time audit and an automated evidence pipeline; how to wire compliance checks (CIS Benchmarks, SOC 2 controls, in-toto attestations) into the same PR pipeline as governance; the audit evidence the programme produces and the cadence of evidence collection.→
- The least bureaucratic controls — the disciplineThe discipline of the least bureaucratic control: name the loss, size the mechanism, measure the friction, assign the owner, expire the rule; the failure modes of every shortcut; the structural properties of a control programme that stays a control programme.→
Part CXVII
Compliance and Audit
6 checks
- The compliance reconstruction question — what the auditor asksFrame compliance as a reconstruction question; identify the six categories of question an auditor runs against a production deployment; recognise why "we have logs" is not an audit trail until those logs can answer every question in under five minutes.→
- The reconstructability test — can you answer every question?Run the reconstructability test; pick a random recent change and answer the six auditor categories from a fresh terminal in under five minutes each; measure time-to-answer; identify the failure modes that make the test fail; set the quarterly cadence.→
- The deployment receipt — the artefact of recordDefine the deployment receipt as the artefact that joins the commit to the deployment; specify its contents (commit, artifact digest, runner identity, timestamps, signatures); emit it from CI; query it from a fresh terminal; bind the receipt to retention.→
- The change authorisation record — the approval trailDefine the change authorisation record as the audit trail of who approved the change, when, and on what grounds; distinguish required reviewers from advisory reviewers; bind the record to the PR, the branch policy, and the change board when applicable.→
- Segregation of duties in the pipeline — the structural controlDefine segregation of duties as a structural control enforced by the pipeline, not a procedural control enforced by policy; identify the four duty pairs that must be segregated (author/approver, author/deployer, approver/deployer, deployer/operator); recognise the role of OIDC and signed identities.→
- Six months later and the auditor — the practical testWalk through the auditor visit six months after the change; answer the six categories from a fresh terminal using gh run view and argocd app history; time the answers; identify the gaps the architecture review missed; close the gaps before the next audit.→
Part CXVIII
Final Reference Architecture
6 checks
- The reference architecture — the integrated view of every fleetThe full production reference architecture as one queryable picture; the five fleets, the four planes, the signed edges between them; how the fleets combine into a single delivery system; the difference between a fleet and a tool, and a plane and a stage.→
- The GitHub Actions control plane — the CI fleetThe Actions fleet in detail: workflows, reusable workflows, composite actions, OIDC tokens, environment protection rules, run concurrency, and the relationship between the Actions control plane and the runner fleet; how Actions is operated as a fleet rather than a tool.→
- The Argo CD control plane — the GitOps fleetThe Argo CD fleet in detail: Application, ApplicationSet, sync waves, cluster registry, RBAC at the controller, sync windows, diff customisation, and the relationship between the Argo CD fleet and the cluster registry; how Argo CD is operated as a fleet rather than a tool.→
- The ephemeral runner fleet — the execution surfaceThe runner fleet in detail: ephemeral runners, immutable images, autoscaling via ARC/KEDA, network egress, secret injection, runner isolation, and the relationship between the runner fleet and the Actions control plane; how a runner fleet is operated as a product.→
- The artifact registry fleet — the artifact layerThe registry fleet in detail: OCI by digest, cosign signatures, retention and replication, provenance and SBOM as OCI artefacts, immutable tags, and the relationship between the registry fleet and the manifest repository Argo CD reads from.→
- The observability and audit fleet — the operational visibilityThe observability and audit fleet in detail: metrics, logs, traces, and the audit trail derived from receipts; the difference between an observation and an attestation; the unification of receipt and observation into a queryable audit trail; how observability becomes auditable by leaving the data in its native stores and querying across them.→
Part Final
Final Assessment
6 checks
- Git internals and recovery — recapRecap of Git as a content-addressed object store: blob/tree/commit/tag objects, the commit DAG, refs and the reflog, and the recovery flows that depend on them — reflog navigation, revert, reset, and filter-repo.→
- CI/CD architecture and runner security — recapRecap of the CI/CD control plane (forge, orchestrator, workflow engine), the runner fleet (hosted, self-hosted, ephemeral, autoscaling), and the runner security model (privileged mode, Docker socket, OIDC federation, short-lived credentials).→
- Supply chain and signing — recapRecap of the supply-chain trust boundaries (source, dependency, CI, artifact, deployment), the SBOM formats (SPDX, CycloneDX), the SLSA build levels, and the Sigstore stack (cosign, Fulcio, Rekor) for keyless signing and transparency-log attestation.→
- GitOps reconciliation and drift — recapRecap of the GitOps model (declarative, versioned, pulled, continuously reconciled), the controllers (Argo CD, Flux), the reconciliation loop, and the drift problem (manual, accidental, emergency) with self-heal and prune as the controls.→
- Incident response and secrets — recapRecap of the incident response flows for the three CI/CD incident classes (secret leak, runner compromise, supply-chain compromise), the secrets-management model (external secret stores, SOPS, Sealed Secrets), and the rotation cadence for long-lived credentials.→
- Auditability and compliance — recapRecap of the audit chain (commit → pipeline → artifact → deployment), the deployment receipt, the change authorisation record, segregation of duties in the pipeline, and the reconstructability test that determines whether a six-month-old incident can be answered.→