Match runner size, workers and shards to your suite: measure the scaling curve, find where parallelism stops paying, and pick the cheapest setup.

Right-Sizing CI Runners for Test Throughput

Most teams pick a runner size once, never revisit it, and then argue about test speed for a year. The right size is not a matter of taste: a suite has a scaling curve, and measuring it takes twenty minutes and tells you exactly where extra cores stop buying time and start buying contention. This guide walks through that measurement for a Vitest and Playwright suite, shows how worker count interacts with core count and with sharding, and ends with a decision rule for picking the cheapest configuration that meets a feedback-time budget. It is for engineers who own a pipeline and can change its runner labels, and it sits under continuous integration test orchestration.

Root Cause Analysis

Test suites do not scale linearly with cores, and the reason is that only part of the work is parallel. Setup — installing dependencies, starting a database, compiling TypeScript — happens once regardless of how many workers run afterwards. The parallel part shrinks as workers increase; the serial part does not. That is why doubling cores rarely halves wall clock, and why the fourth doubling often achieves nothing at all.

The second effect works against you at the top end. Each worker needs memory and file-descriptor headroom, and browser-based tests need a great deal of both. Past the point where the machine can hold every worker comfortably, additional workers cause swapping, longer garbage collection pauses and timing variability — which shows up first as flakiness rather than as slowness, and is therefore usually misdiagnosed.

The third is that sharding and workers solve the same problem at different granularity, and they compound. Four runners with four workers each is sixteen concurrent test processes, and a suite with a shared external dependency may not tolerate that even though each individual machine looks comfortable.

The scaling curve of a typical test suite Wall clock falls steeply from one to four workers, flattens between four and eight, and rises again beyond eight as contention outweighs parallelism, with the serial setup time forming a floor throughout. serial floor — setup, install, compile 1 2 4 6 8 16 workers knee: stop here contention
Every suite has a knee; paying for cores beyond it buys variance rather than speed.

Reproducible Setup

Make worker count controllable from the environment so the sweep is a loop rather than a series of commits.

// vitest.config.ts
import { defineConfig } from 'vitest/config';

const workers = process.env.TEST_WORKERS ? Number(process.env.TEST_WORKERS) : undefined;

export default defineConfig({
  test: {
    pool: 'threads',
    poolOptions: { threads: { maxThreads: workers, minThreads: workers } },
    reporters: ['default', 'json'],
    outputFile: { json: './reports/results.json' },
  },
});
# a clean baseline: no cache, no other jobs, same commit each time
git stash list && node -e "console.log('cores:', require('os').cpus().length)"

Implementation

Step 1 — Sweep the worker count on one machine. Three repetitions per setting is enough to see past noise, and the whole sweep takes under half an hour for most suites.

#!/usr/bin/env bash
# scripts/sweep-workers.sh
set -euo pipefail
for w in 1 2 4 6 8 12 16; do
  total=0
  for run in 1 2 3; do
    start=$(date +%s%3N)
    TEST_WORKERS=$w npx vitest run --silent >/dev/null
    end=$(date +%s%3N)
    total=$(( total + end - start ))
  done
  echo "$w workers: $(( total / 3 )) ms"
done

Step 2 — Separate the serial floor from the parallel work. Time the setup steps on their own; that number is the asymptote no amount of parallelism will beat, and it is often the thing actually worth optimising.

time npm ci                      # 42s   ← serial, cacheable
time npx tsc --noEmit            # 18s   ← serial, cacheable
time TEST_WORKERS=1 npx vitest run  # 340s ← the parallelisable part
# floor ≈ 60s; even infinite workers cannot go below it

Step 3 — Find the knee and stop there. The knee is the last setting where adding workers still removes a meaningful share of the remaining time. In the sweep above it is typically four to six on a hosted runner.

// scripts/knee.ts
const samples = [
  { workers: 1, ms: 340_000 }, { workers: 2, ms: 196_000 }, { workers: 4, ms: 128_000 },
  { workers: 6, ms: 112_000 }, { workers: 8, ms: 108_000 }, { workers: 16, ms: 131_000 },
];

let best = samples[0];
for (const s of samples.slice(1)) {
  const gainPct = ((best.ms - s.ms) / best.ms) * 100;
  if (gainPct < 8) { console.log(`knee at ${best.workers} workers`); break; }
  best = s;
}

Step 4 — Then choose between a bigger machine and more machines. Once workers are at the knee, further speed comes from sharding across runners — and that trade has a different cost profile, because each shard repeats the serial floor.

# .github/workflows/test.yml
jobs:
  unit:
    strategy:
      matrix: { shard: [1, 2, 3, 4] }
    runs-on: ubuntu-latest-4-cores
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 20, cache: npm }
      - run: npm ci
      - run: npx vitest run --shard=${{ matrix.shard }}/4
        env: { TEST_WORKERS: 4 }
A bigger machine versus more machines One larger runner pays the serial setup once but hits the knee sooner, while four smaller runners pay setup four times yet cut the parallel work further, so the better choice depends on how large the serial floor is. One 8-core runner setup 60s tests 108s at the knee total 168s Four 4-core runners setup 60s shard 32s total 92s per shard setup 60s shard 32s ×4 machine-minutes a large serial floor makes sharding expensive — shrink it before buying machines
Sharding multiplies the serial floor, so the floor is usually the first thing worth attacking.

Step 5 — Cap workers explicitly rather than relying on defaults. Both runners default to something derived from the host, which differs between a laptop and a container that reports the host’s cores while being limited to fewer.

// vitest.config.ts — a number that is right in both places
import os from 'node:os';

const detected = os.availableParallelism?.() ?? os.cpus().length;
const workers = Number(process.env.TEST_WORKERS ?? Math.min(detected, 4));

Step 6 — Write the decision down where the pipeline is defined. A runner label and a worker cap are choices with reasons behind them, and the reasons are invisible six months later. A comment naming the measured knee and the date it was measured saves the next person the entire sweep, and tells them when the measurement is stale enough to repeat.

# Measured 2026-09-18: knee at 6 workers on a 4-core hosted runner; serial floor
# is ~60s (npm ci + tsc). Re-sweep after any large change to the browser tier.
runs-on: ubuntu-latest-4-cores
env: { TEST_WORKERS: 6 }

Verification

Verify that the chosen setting is genuinely at the knee by re-running the sweep after any significant change to the suite. Suites move: adding browser tests shifts the knee downward because each worker costs far more memory.

./scripts/sweep-workers.sh
# 1 workers: 341200 ms
# 2 workers: 197400 ms
# 4 workers: 129100 ms   ← 35% better than 2
# 6 workers: 112800 ms   ← 13% better than 4
# 8 workers: 108900 ms   ← 3% better than 6 → knee at 6

Then verify that the setting does not increase flakiness, which is the failure mode of oversubscription. Run the suite ten times at the chosen setting and count non-deterministic results.

for i in $(seq 1 10); do npx vitest run --silent >/dev/null || echo "run $i failed"; done
# (no output — ten clean runs)

Finally, verify the containerised view of the machine matches reality, because a runner inside a container frequently reports the host’s core count while being limited to a fraction of it — which makes every default wrong at once.

node -e "console.log('reported:', require('os').cpus().length)"
cat /sys/fs/cgroup/cpu.max
# 400000 100000     ← 4 cores, whatever os.cpus() claims
Symptoms of the wrong runner size Too few workers shows as a steady, unvarying long run; too many shows as timing flakiness, timeouts and memory errors while the average barely improves. Too few workers long but very consistent runs low memory use throughout cheap to fix: raise the cap Too many workers timeouts that vary run to run heap errors under load often misread as flaky tests
Oversubscription rarely looks like slowness — it looks like flakiness, which is why it survives so long.

Troubleshooting

Symptom: more workers made no difference at all. Diagnosis: the suite is dominated by its serial floor, or by a single very long test file that no amount of parallelism can split. Fix: measure the floor as in Step 2, and split the long file — parallelism operates at file granularity in both runners, so one four-minute file sets a hard lower bound.

Symptom: heap out-of-memory errors appear only in CI. Diagnosis: the container reports host cores, so the default worker count is several times what the memory limit supports. Fix: read the cgroup limit rather than os.cpus(), or set the cap explicitly from the workflow where the runner size is known.

Symptom: the suite is faster with fewer workers. Diagnosis: genuine contention, most often on a shared resource such as a single test database rather than on CPU. Fix: give each worker its own namespace as in isolating end-to-end tests with per-worker data, then re-sweep — the curve usually changes shape entirely.

FAQ

Is a bigger runner or more runners cheaper?

It depends almost entirely on the serial floor, because sharding pays that floor once per shard. With a sixty-second install and a hundred-second test phase, four shards spend four minutes of machine time to save about seventy seconds of wall clock. If feedback time is what you are buying, that can be worth it; if cost is, shrink the floor with better caching first.

Should unit and end-to-end suites use the same runner size?

Rarely. Unit tests are CPU-bound and scale well to the core count; browser tests are memory-bound and usually need one to two workers per two cores. Sizing them together means one of the two is wrong, and the usual outcome is an oversubscribed browser job blamed on flaky tests.

How often should the sweep be repeated?

After any significant change in the suite’s composition — a new browser tier, a large batch of tests, a runner image upgrade — and otherwise about twice a year. It is cheap, and the knee genuinely moves; a configuration chosen two years ago is unlikely to still be the right one.

Does this apply to self-hosted runners?

Yes, with one addition: self-hosted machines are often shared between jobs, so the effective core count during your run is not the machine’s core count. Measure under realistic load rather than on an idle machine, or the knee you find will be optimistic in exactly the way that produces flakiness later.