Hybrid and middleware — polyglot repos, submodules, partial clones, and sparse-checkout

A pure monorepo and a pure multi-repo are the two endpoints of a spectrum. Most production infrastructure codebases live in the middle: a monorepo that pulls a versioned submodule, a multi-repo that ships a shared library as a partial clone, a working tree that uses sparse-checkout to keep the engineer’s day-to-day fast. The hybrid patterns are the most common and the most operationally expensive, because the team gets the complexity of both models without the simplicity of either.

The four hybrid patterns

The patterns that bridge a monorepo and a multi-repo are:

• Submodules. A parent repository embeds a pinned reference to a child repository. The parent’s clone does not pull the child’s history; the parent’s tree records the pinned commit of the child.
• Partial clones. A clone that omits blobs (--filter=blob:none) or commits (--filter=tree:0) from the initial fetch. The omitted objects are downloaded on demand.
• Sparse-checkout. A working tree that contains only a subset of the repository’s directories. The repository itself is full; the working tree is restricted.
• Versioned references. A consumer repository declares a pinned tag of a producer repository, and the build pins the digest.

flowchart LR
    subgraph M["Monorepo"]
        MS["submodule: shared/terraform-modules"]
        MP["partial clone: --filter=blob:none"]
    end
    subgraph S["Multi-repo"]
        SP["sparse-checkout: services/payments"]
        SV["versioned ref: shared-terraform v1.4.2"]
    end

Submodules in production

A submodule is a parent repository’s pointer to a pinned commit in a child repository. The pattern is correct for versioned reuse: a Terraform module consumed by many parent repositories should be a submodule, because the consumption is a contract about a specific commit, not a moving target.

PARENT_URL="git@github.com:example/infra-parent.git"
SHARED_URL="git@github.com:example/shared-modules.git"

git clone "$PARENT_URL"
cd infra-parent
git submodule add "$SHARED_URL" modules/shared
git submodule update --init --recursive

The four commands are the entire workflow: clone the parent, add the submodule, fetch the pinned commit, and verify the working tree. The pitfall is that the parent’s tree records the pinned commit, but the cloned working tree does not contain the child until git submodule update --init --recursive runs.

Partial clones and sparse-checkout

A partial clone is a bandwidth optimisation: the server omits blobs from the initial fetch, and the client downloads them on demand. A sparse-checkout is a working-tree optimisation: the client restricts the working tree to a subset of the directories.

REPO_URL="https://github.com/example/monorepo.git"

git clone --filter=blob:none "$REPO_URL"
cd monorepo
git sparse-checkout init --cone
git sparse-checkout set "services/payments"
git ls-files | wc -l

The five commands are the workflow: partial clone, sparse init, sparse set, and verify. The verification step is git ls-files | wc -l; the count should be small (the engineer’s directory), not the full repository size.

The pitfall is that the partial clone and the sparse-checkout are separate mechanisms. A engineer who runs git sparse-checkout set services/payments on a full clone has a working tree of services/payments/, but the .git directory still contains the full history. The bandwidth saving only comes from the partial clone (--filter=blob:none).

Versioned references and pinned tags

A versioned reference is the multi-repo analogue of a submodule. The consumer repository declares a pinned tag of the producer repository, and the build pins the digest:

PRODUCER_TAG="v1.4.2"

PRODUCER_URL="git@github.com:example/shared-modules.git"

git clone --depth 1 --branch "$PRODUCER_TAG" "$PRODUCER_URL"
git ls-files | wc -l

The --branch flag tells the server to fetch only the tagged commit, and the --depth 1 flag tells the server to omit the history. The consumer is now pinned to a specific commit, and the build can verify the digest.

Production discipline

1. Pin every submodule to a commit, not a branch. A branch reference is a git pull of latest by another name.
2. Use --filter=blob:none for partial clones, not just a shallow --depth. A shallow clone omits history; a partial clone defers blobs.
3. Use git sparse-checkout init --cone for sparse-checkout, not the legacy non-cone mode. The cone mode is the only one that scales.
4. Verify every hybrid setup with git ls-files | wc -l. A working tree that contains the full repository is a working tree that is paying the full cost.

Cross-course references

• Linux for Production Sysadmins - Part XXVI (RepoLayout) covers the filesystem analogue: a single root tree with bind-mounted subtrees versus a per-service tree.
• Ansible for Production Sysadmins - Part XXXVII (RepoArch) covers the role-based layout of a monorepo Ansible repository, where the roles are versioned through Ansible Galaxy or a shared collection.
• Terraform for Production Sysadmins - Part IX (State) covers why Terraform state is the strongest argument for pinning every module reference to a commit, not a branch.

Quiz