Binaries and large files — what does not belong in Git

Git is a delta-compressing content store. Text files revise in small deltas; the store earns its keep. The moment a 50 MB compiled binary lands in the tree, the store stops being delta: the binary and its predecessor share no bytes, the delta is the full file, and each revision costs the full 50 MB.

The four categories

• Compiled output. .o, .jar, .exe, .bin. Duplication of state.
• Image and media. .png, .mp4, .pdf. Pipeline output or tolerate LFS.
• Datasets and weights. .parquet, .pt, .onnx. Model registry.
• Backups, archives. .sql.gz, .tar. Live state; bandwidth, not safety.

git rev-list --objects --all | \
    git cat-file --batch-check='%(objecttype) %(objectsize) %(rest)' | \
    awk '/^blob/' | sort -k2 -nr | head -20

Ranks blobs by size. git-sizer reports the same grouped by extension.

Why binary content is irreversible

A 200 MB .parquet committed in 2022 cannot be removed without rewriting every commit that referenced it. Tags, signed signatures, and CI runs point at different hashes; audit trail breaks for every consumer. A .parquet that never lands costs nothing; one that lands later costs a forced rewrite that breaks pinned systems.

Pre-commit enforcement and routing

A .gitignore is a declaration; a pre-commit hook is the enforcement. check-added-large-files refuses any staged file above the threshold (512 KB sensible; 100 KB if LFS).

repos:
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.6.0
    hooks:
      - id: check-added-large-files
        args: ['--maxkb=512']

Replacement stores: compiled -> artifact; media -> object store; weights -> MLflow; backups -> managed backup. Repository stores a version reference, never the bytes.

Production discipline

1. Set a pre-commit size threshold (512 KB default, 100 KB if LFS).
2. Apply .gitignore before the first commit.
3. Route assets to their correct store.
4. Treat any large file as deliberate.

Cross-course references

• Linux for Production Sysadmins Part XXVI: build output in /var/cache.
• Terraform for Production Sysadmins Part IX: state in a backend.

Quiz