Keeping Large Fixture Files Out of the Repo

A repository that has been accumulating fixtures for a few years usually contains more fixture bytes than source bytes, and every one of them is in every clone, forever — git stores history, so deleting a twelve-megabyte JSON file does not make the repository smaller. The symptoms are familiar: slow clones, slow CI checkouts, unreviewable pull requests where a regenerated fixture produces a forty-thousand-line diff. This guide covers moving large fixtures out to content-addressed storage, fetching them reliably in CI, and — the more important half — reducing how many large fixtures you need at all. It sits under test data management.

Root Cause Analysis

Large fixtures accumulate because they are the path of least resistance at the moment they are created. A developer needs a realistic API response, saves the one they have, and commits it. Nothing at that moment signals a cost, and the cost arrives later, spread across everyone who clones the repository.

Git makes this worse than it first appears. Because every version of every file is retained, a fixture regenerated monthly for two years contributes twenty-four copies to the history. Deleting the file leaves all of them, so the repository never shrinks without a history rewrite that everybody has to cooperate with.

The second problem is review. A pull request that regenerates a fixture shows thousands of changed lines, none of which anyone reads. Real changes hide in that noise: a field that quietly disappeared, a value that changed meaning. Once fixtures are unreviewable, they stop being verified at all and start drifting from what the service actually returns.

The cost is paid by every clone forever, which is why the decision matters at commit time.

Reproducible Setup

Find out what you actually have before choosing a mechanism; the answer is often concentrated in a handful of files.

# largest files currently tracked
git ls-files -z | xargs -0 du -h 2>/dev/null | sort -rh | head -10
# 14M  test/fixtures/catalogue-snapshot.json
# 9.2M test/fixtures/orders-2025.json

# largest objects in history, which is what a clone downloads
git rev-list --objects --all \
  | git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' \
  | awk '$1=="blob" {print $3, $4}' | sort -rn | head -10

Implementation

Step 1 — Ask whether the fixture needs to be large at all. Most large fixtures are large by accident: a full catalogue where three products would do, a year of orders where a week demonstrates the same behaviour. Shrinking is free and removes the problem rather than relocating it.

// scripts/fixtures/trim.ts — keep the variety, drop the volume
import { readFileSync, writeFileSync } from 'node:fs';

const all = JSON.parse(readFileSync('test/fixtures/catalogue-snapshot.json', 'utf8')) as Product[];

// one representative per distinct shape, rather than all 40,000 rows
const seen = new Set<string>();
const trimmed = all.filter((p) => {
  const shape = `${p.type}|${p.variants.length > 0}|${p.discount != null}|${p.tags.length > 3}`;
  if (seen.has(shape)) return false;
  seen.add(shape);
  return true;
});

writeFileSync('test/fixtures/catalogue.json', JSON.stringify(trimmed, null, 2));
console.log(`${all.length} → ${trimmed.length} products`);
// 41208 → 47 products

Step 2 — Store what remains by content hash in object storage. The repository keeps a small manifest; the bytes live elsewhere and are immutable, so a given hash always means the same content.

// test/fixtures/manifest.json — this is what gets committed
{
  "orders-2025.json": {
    "sha256": "9f2c1ab4e7d3…",
    "bytes": 9646080,
    "url": "s3://acme-test-fixtures/9f2c1ab4e7d3.json"
  }
}

// scripts/fixtures/fetch.ts
import { createHash } from 'node:crypto';
import { readFileSync, writeFileSync, existsSync, mkdirSync } from 'node:fs';
import manifest from '../../test/fixtures/manifest.json';

const CACHE = '.cache/fixtures';
mkdirSync(CACHE, { recursive: true });

for (const [name, entry] of Object.entries(manifest)) {
  const cached = `${CACHE}/${entry.sha256}`;
  if (!existsSync(cached)) {
    const res = await fetch(entry.url.replace('s3://', 'https://acme-test-fixtures.s3.amazonaws.com/'));
    writeFileSync(cached, Buffer.from(await res.arrayBuffer()));
  }
  const actual = createHash('sha256').update(readFileSync(cached)).digest('hex');
  if (actual !== entry.sha256) throw new Error(`Fixture ${name} failed its integrity check`);
  writeFileSync(`test/fixtures/${name}`, readFileSync(cached));
}

Step 3 — Cache the downloads in CI, keyed on the manifest. The fetch should happen once per manifest change, not once per run.

# .github/workflows/test.yml
      - uses: actions/cache@v4
        with:
          path: .cache/fixtures
          key: fixtures-${{ hashFiles('test/fixtures/manifest.json') }}
      - run: npx tsx scripts/fixtures/fetch.ts
      - run: npx vitest run

Step 4 — Make the fetch automatic for developers too. A manual step will be forgotten, and the resulting failure — a missing file — is confusing rather than instructive.

// package.json
{
  "scripts": {
    "postinstall": "tsx scripts/fixtures/fetch.ts",
    "test": "tsx scripts/fixtures/fetch.ts && vitest run"
  }
}

Most fixtures should move left rather than right — shrinking beats relocating.

Step 5 — Guard the boundary with a check rather than a convention. A size limit in CI stops the next large fixture from being committed by someone who did not know the policy.

#!/usr/bin/env bash
# scripts/fixtures/check-size.sh
MAX=262144   # 256 KB
fail=0
while IFS= read -r f; do
  size=$(wc -c < "$f")
  if [ "$size" -gt "$MAX" ]; then
    echo "::error file=$f::Fixture is $((size/1024)) KB; the limit is $((MAX/1024)) KB — trim it or move it to storage"
    fail=1
  fi
done < <(git diff --name-only --diff-filter=AM origin/main...HEAD -- 'test/fixtures/*')
exit $fail

Step 6 — Decide what happens when storage is unavailable. A test suite that cannot run without a network call to object storage is fragile in a specific, annoying way. Cache aggressively, and fail with a message that names the manifest entry and the cache path rather than a bare fetch error.

Verification

Verify the manifest and the storage agree, since a fixture uploaded but not recorded — or recorded but not uploaded — fails only when someone clones fresh.

npx tsx scripts/fixtures/fetch.ts
# fetching orders-2025.json (9.2 MB)… ok, sha256 verified
# catalogue.json cached, sha256 verified

Then verify a cold clone works, which is the scenario the whole scheme exists to serve and the one nobody tests.

git clone --depth 1 "$REPO_URL" /tmp/cold && cd /tmp/cold
npm ci && npm test
# the postinstall hook should fetch everything with no manual step

Finally, verify the repository actually got smaller — or, more precisely, stopped growing. A count of bytes added to history per month tells you whether the policy is working better than a one-off measurement does.

git log --since="6 months ago" --numstat --format= -- 'test/fixtures/*' \
  | awk '{added+=$1} END {print "lines of fixture added in 6 months:", added}'
# lines of fixture added in 6 months: 412

Clone size is the visible cost; unreviewable fixtures are the expensive one.

Troubleshooting

Symptom: the repository is still huge after moving fixtures out. Diagnosis: history still contains every past version. Fix: a history rewrite is the only remedy, and it is disruptive — coordinate it, do it once, and make sure the size check is in place first so the problem does not recur.

Symptom: CI fails intermittently fetching fixtures. Diagnosis: the cache key does not cover the manifest, so every run downloads afresh and occasionally hits a transient error. Fix: key the cache on the manifest hash as in Step 3, and add a short retry around the fetch — with the cache working, the fetch happens rarely enough that a retry is cheap.

Symptom: developers hit a missing-file error. Diagnosis: the automatic fetch is only wired into postinstall, which does not run when switching branches. Fix: also run it as part of the test script, and make the error message name the command to run rather than reporting a missing path.

Symptom: a fixture is out of date and nobody noticed. Diagnosis: nothing verifies that the fixture still matches what the real service returns. Fix: add a scheduled check that fetches a live response and compares its shape — not its values — against the fixture, which is a contract check in miniature and closely related to contract testing.

FAQ

Why not use git LFS?

It works and is a reasonable choice when the files must stay conceptually in the repository. The reasons to prefer an explicit manifest are that LFS requires every clone to have the extension configured, it complicates shallow clones in CI, and it makes the fixture’s size invisible again — which removes the pressure to ask whether it should be large at all.

How small should a fixture be?

Small enough to read. If a reviewer cannot scan it and understand what case it represents, it is too big regardless of byte count. In practice this means most fixtures are a few dozen lines, built by a factory rather than stored at all — see factory functions vs fixtures in Vitest.

What about binary fixtures like images or PDFs?

The same rules apply, with more force, since binaries neither diff nor compress well in git. Content-addressed storage is the right home, and the integrity check matters more because a corrupted binary produces a baffling failure rather than a parse error.

Does this apply to snapshot files?

Snapshots are fixtures with a particular update workflow, and an oversized snapshot has the same reviewability problem. If a snapshot is too large to read, that is usually a signal to assert on specific properties instead, rather than a signal to move it to storage.

Back to Test Data Management
Building a typed test data builder API — generating small fixtures instead of storing large ones.
Anonymising production data for test fixtures — producing the large artifact this guide stores.
Loading fixtures lazily to cut test startup time — the runtime cost of the fixtures you keep.