The large repository problem — clone, fetch, and checkout at scale

A repository that feels slow is slow in three ways. The three costs - initial clone, incremental fetch, working-tree checkout - scale with three parts of the anatomy. Treating “the clone is slow” as one problem usually moves the wrong dial.

The three profiles

Initial clone is paid once when an engineer or CI runner fetches the repository for the first time. Cost: the served packfile.

Incremental fetch is paid on every git fetch. Cost: divergence from the remote ref.

Working-tree checkout is paid when git checkout materialises the tree to disk. Cost: bytes to write, with a multiplier on the largest files.

flowchart LR
    A["clone URL"] --> B["server packfile"]
    B --> C["client receive + verify"]
    C --> D["materialise tree"]
    A -.->|"daily"| E["git fetch"]
    E --> F["incremental pack"]

A monorepo that ships hourly cares about fetch. New engineer onboarding cares about clone.

What dominates the packfile

Packfile compression is delta-encoded against similar neighbours. Text compresses well: a 100 KB YAML and its revision often delta to a few KB. A compiled binary does not - byte-level similarity is near zero, and the packfile stores the full binary in every revision.

git rev-list --objects --all | \
    git cat-file --batch-check='%(objecttype) %(objectsize) %(rest)' | \
    awk '/^blob/' | sort -k2 -nr | head -20

git-sizer summarises top contributors.

Production discipline

1. Record all three profiles before optimising.
2. Identify largest blobs with git rev-list --objects.
3. Measure server and client cost separately.
4. Compare against the same workload.

Cross-course references

• Linux for Production Sysadmins Part XXVI (RepoLayout): filesystem analogue.
• Terraform for Production Sysadmins Part IX (State): bandwidth on plan.

Quiz