Keeping Mutation Testing Fast in CI

Mutation testing is abandoned for one reason far more often than any other: the job takes too long, so it gets moved to a weekly schedule, then to a manual trigger, then to nobody. The techniques that keep it fast are all about running fewer mutants rather than running them faster — an incremental report so unchanged files are skipped, a glob scoped to code where the result matters, and concurrency matched to the machine rather than to optimism. This guide is for engineers who have a working Stryker setup and now need it to live in CI without becoming the slowest thing in the pipeline. It assumes the configuration from running Stryker mutation testing with Vitest and targets a run that finishes in a couple of minutes on an ordinary hosted runner.

Root Cause Analysis

The total cost of a run is the number of mutants multiplied by the tests executed per mutant, plus a fixed startup cost per concurrent runner. Every optimisation attacks one of those three terms, and it is worth knowing which one you are short on before changing anything, because the wrong lever makes things worse.

Mutant count is usually the dominant term and the easiest to reduce, because most repositories include far more in the glob than anyone will act on. Tests per mutant is controlled by coverage analysis: with per-test data, a mutant runs only its covering tests, which on a well-structured suite is a handful. Without it, every mutant runs everything.

Startup cost is the term people forget. Each concurrent runner gets a sandbox copy of the project and starts its own Vitest process, so raising concurrency past the available cores adds copies and context switching without adding throughput. On a two-core hosted runner, concurrency: 8 is reliably slower than concurrency: 2, and it also inflates timeouts, which then get misreported as killed mutants.

Reproducible Setup

Establish a baseline you can compare against, and record it so later changes are measurable rather than felt.

time npx stryker run 2>&1 | tail -20
# Mutation score: 76.40%
# real  8m41.103s     ← the number to beat

// stryker.config.mjs — the starting point
export default {
  testRunner: 'vitest',
  coverageAnalysis: 'perTest',
  mutate: ['src/**/*.ts', '!src/**/*.test.ts'],
  concurrency: 4,
  reporters: ['clear-text', 'json'],
  jsonReporter: { fileName: 'reports/mutation/mutation.json' },
};

# how many mutants is the glob actually producing?
npx stryker run --dryRunOnly 2>&1 | grep -i "mutant"
# Instrumented 8,412 mutant(s)   ← far more than anyone will act on

Implementation

Step 1 — Cut the glob to code where a survivor would change your behaviour. This is not a compromise on rigour; it is a statement about where the technique pays. Presentational components, generated clients and configuration objects produce mutants you will never act on.

mutate: [
  'src/domain/**/*.ts',
  'src/lib/pricing/**/*.ts',
  'src/lib/permissions/**/*.ts',
  '!src/**/*.test.ts',
  '!src/**/*.stories.ts',
  '!src/**/generated/**',
],
// Instrumented 1,140 mutant(s)   ← 7× less work, ~95% of the signal

Step 2 — Turn on incremental mode and persist the report. With a stored report, Stryker re-tests only mutants in files that changed, reusing previous results for everything else. The stored file must survive between runs, so it is cached, not rebuilt.

export default {
  // …
  incremental: true,
  incrementalFile: 'reports/mutation/stryker-incremental.json',
};

# .github/workflows/mutation.yml
      - uses: actions/cache@v4
        with:
          path: reports/mutation/stryker-incremental.json
          key: stryker-${{ github.sha }}
          restore-keys: stryker-

Step 3 — Scope pull request runs to changed files. For the merge gate, mutate only what the branch touched. The nightly run keeps the full picture; the pull request run answers “did this change arrive with weak tests”.

#!/usr/bin/env bash
# scripts/mutate-changed.sh
set -euo pipefail
base="${1:-origin/main}"
files=$(git diff --name-only "$base"...HEAD -- 'src/domain/*.ts' | grep -v '\.test\.ts$' || true)

if [ -z "$files" ]; then
  echo "no mutated source changed; skipping"
  exit 0
fi

npx stryker run --mutate "$(echo "$files" | paste -sd, -)"

Step 4 — Match concurrency to the runner, and say so explicitly. Read the core count rather than hard-coding a number that is right on a laptop and wrong in CI.

import os from 'node:os';

export default {
  // …
  concurrency: Math.max(1, Math.min(os.cpus().length - 1, 8)),
  timeoutMS: 15_000,
  timeoutFactor: 2.5,
};

The timeoutFactor setting deserves a word. Stryker measures how long the covering tests take during the baseline run and allows each mutant that time multiplied by the factor, plus timeoutMS. Expressing the allowance as a multiple rather than a fixed number is what makes the configuration portable: the same file works on a fast laptop and a contended hosted runner, because the allowance scales with what the machine actually demonstrated it can do.

Step 5 — Fail the job on a threshold, not on a wobble. An incremental, changed-files run has a small mutant count, so a single equivalent mutant can swing the percentage. Gate on the survivor count in changed files rather than on the score alone.

// scripts/gate.ts — fail only on new survivors in changed code
import { readFileSync } from 'node:fs';

const report = JSON.parse(readFileSync('reports/mutation/mutation.json', 'utf8'));
const survived = Object.values(report.files)
  .flatMap((f: any) => f.mutants)
  .filter((m: any) => m.status === 'Survived');

if (survived.length > 0) {
  console.error(`${survived.length} surviving mutant(s) in changed files`);
  for (const m of survived.slice(0, 10)) console.error(`  ${m.mutatorName} @ ${m.location.start.line}`);
  process.exit(1);
}

Verification

Measure each change rather than stacking them hopefully. Run the baseline, apply one lever, and record the difference; the numbers are usually dramatic enough that it is obvious which levers matter for your project.

time npx stryker run   # after narrowing the glob
# real  1m12.004s      ← from 8m41s, same score on the code that matters

Then verify incremental mode is actually being used, since a missing or stale cache silently falls back to a full run without any warning.

npx stryker run 2>&1 | grep -i incremental
# Incremental: 1,081 of 1,140 mutant(s) restored from the previous report

Finally, verify the pull request lane behaves correctly for a branch that changes nothing mutated — it should skip cleanly rather than running the whole glob, which is the failure mode that quietly puts eight minutes back into every build.

One last measurement is worth taking before declaring the job tuned: the wall clock of the whole CI pipeline, not just the mutation step. A mutation job that runs in parallel with a slower test job costs nothing at all in pipeline time, while the same job appended after everything else costs its full duration. Placing it alongside the existing test matrix, rather than downstream of it, is often a bigger saving than any Stryker setting.

Troubleshooting

Symptom: the incremental run is as slow as a full one. Diagnosis: the stored report is not being restored, usually because the cache key includes the commit SHA with no restore-keys fallback, so every run starts cold. Fix: use a prefix restore key as in Step 2, and confirm with the log line from the verification section.

Symptom: many mutants report as timeouts in CI but not locally. Diagnosis: concurrency exceeds the runner’s cores, so each mutant’s tests run on a contended machine and exceed timeoutMS. Fix: derive concurrency from os.cpus(), and raise timeoutFactor rather than timeoutMS so the allowance scales with the measured baseline.

Symptom: the changed-files run passes but the nightly run finds survivors in the same files. Diagnosis: the changed-files run mutated only the edited file while the survivor lives in a caller the change affected. Fix: expand the branch scope to the directory rather than the file, or accept it — this is exactly the gap the nightly lane exists to close.

FAQ

Related

• Back to Mutation Testing & Assertion Quality
• Running Stryker mutation testing with Vitest — the configuration this guide tunes.
• Choosing mutation score targets per module — what the gate should actually require.
• Caching dependencies and test artifacts in CI — the cache mechanics behind incremental runs.