Continuous Integration Test Orchestration

Orchestration is what turns a pile of tests into a pipeline. A suite that runs comfortably on a laptop can still take twenty minutes on a single CI runner, block every open pull request behind it, and burn a fixed budget of compute on work that never changed. This topic area is about the layer that sits between your test files and your CI provider: how work is partitioned across machines, how state is carried between runs, when a pipeline should stop early, and how to avoid running tests that the current change cannot possibly affect. It is a direct extension of a deliberate test pyramid strategy — the pyramid tells you what to test at each layer, and orchestration decides how those layers are scheduled so feedback arrives in minutes rather than coffee breaks. The worked examples use Vitest as the primary runner on GitHub Actions, with YAML for the pipeline and TypeScript for the glue scripts, but the partitioning and caching principles transfer to any runner and any provider.

The core insight is that CI wall-clock time is not the same as test compute time. A ten-minute suite spread across five runners returns in two minutes plus overhead; the same suite with a warm dependency cache skips ninety seconds of install; the same suite scoped to the files a branch actually touched might run a tenth of the cases. These are independent, composable levers, and this topic area gives each one a dedicated guide. Read the four guides together and you have a pipeline that scales sub-linearly with your codebase instead of degrading with every test you add.

Architectural Scope & Boundaries

This topic area covers pipeline-level concerns: how a suite is split, scheduled, and gated once the tests themselves already exist and pass locally. It deliberately stops short of writing the tests. Deciding which behaviours deserve a unit, an integration, or an end-to-end test is the job of the parent overview and its taxonomy; orchestration assumes those decisions are made and asks only how to execute the resulting suite efficiently and reliably in an automated environment.

The boundary with flakiness is worth drawing precisely. Orchestration can expose flakiness — sharding changes execution order, parallelism changes timing, and bail changes which failures you see first — but it does not cure it. When a test fails only under parallel load, the fix belongs to the discipline of quarantining flaky tests in CI, not to a pipeline tweak. Likewise, orchestration touches coverage only at the reporting seam: when you shard a suite, each shard produces a partial coverage report that must be merged before it can be checked against a threshold, which connects directly to enforcing coverage thresholds in a monorepo. Everything inside those two seams — partition, cache, gate, select — is in scope here.

The final boundary is the provider. GitHub Actions supplies the concrete YAML in these guides because it is the most common target, but the concepts map cleanly onto GitLab CI, CircleCI, and Buildkite. A matrix job is a matrix job; a cache key is a cache key; --shard and --bail are runner flags, not provider features. Where a provider genuinely differs — artifact retention windows, cache eviction policy, concurrency limits — the guides flag it, but you should read the mechanics as portable and the YAML as one dialect of a shared idea.

One concept is worth naming explicitly before the guides, because it underlies all four levers: the distinction between a gating run and a reporting run. A gating run answers a single boolean question — may this commit merge? — and every optimisation that preserves that boolean is fair game, including skipping tests a change cannot reach and stopping at the first failure. A reporting run answers richer questions — what is our total coverage, which tests are flaky, how long does the full suite take? — and those answers require running everything and measuring it. The same physical suite serves both roles under different settings, and almost every mistake in CI orchestration traces back to applying a gating optimisation to a run that was supposed to be a reporting run, or trusting a gating run to answer a question only a reporting run can. Keeping the two roles distinct in your pipeline is the single mental model that makes the rest of this topic area safe to apply.

Prerequisites

• A Vitest suite that passes locally with vitest run and a committed vitest.config.ts.
• A CI provider with matrix jobs and a cache action available — GitHub Actions is assumed.
• Node.js pinned by a .nvmrc or engines field so the cache key is stable across runs.
• A lockfile (package-lock.json, pnpm-lock.yaml, or yarn.lock) committed to the repository.
• Coverage reporting configured if you gate on it, using a per-shard-friendly reporter such as json.
• Baseline timing data: run the suite once in CI and record the wall-clock time you are trying to beat.

Step-by-Step Implementation

Step 1 — Establish a single, reproducible install. Before optimising anything, make the install deterministic. Use npm ci against a committed lockfile so every runner resolves the identical dependency tree, and pin Node with a setup action. This is the foundation the cache key will later hash against.

# .github/workflows/test.yml
name: test
on: pull_request
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version-file: '.nvmrc'
          cache: 'npm'
      - run: npm ci
      - run: npx vitest run

Step 2 — Shard the suite across a matrix. Split the run into independent slices so wall-clock time falls toward compute-time divided by shard count. Vitest’s --shard=i/n flag partitions test files deterministically, and a matrix supplies the index.

jobs:
  test:
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2, 3, 4]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version-file: '.nvmrc', cache: 'npm' }
      - run: npm ci
      - run: npx vitest run --shard=${{ matrix.shard }}/4 --reporter=json --outputFile=results-${{ matrix.shard }}.json

The full mechanics, including how to size n and merge reports, live in sharding Vitest across GitHub Actions runners.

Step 3 — Cache what does not change. Dependencies and expensive build artifacts are stable across most runs; hashing them into a cache key and restoring them removes install and rebuild time from the critical path.

      - uses: actions/cache@v4
        with:
          path: |
            node_modules
            .vitest-cache
          key: deps-${{ runner.os }}-${{ hashFiles('**/package-lock.json') }}
          restore-keys: deps-${{ runner.os }}-

The trade-offs — key granularity, cache poisoning, and artifact hand-off between jobs — are covered in caching dependencies and test artifacts in CI.

Step 4 — Fail fast on the merge path. When a pull request has already broken, running the remaining nine minutes of tests wastes compute and delays the red signal. Bail stops the runner at the first failure, and fail-fast on the matrix cancels sibling shards.

// vitest.config.ts — bail after the first failing test in gating runs
import { defineConfig } from 'vitest/config';

export default defineConfig({
  test: {
    bail: process.env.CI ? 1 : 0,
  },
});

The nuance — when bail helps and when it hides information — is the subject of failing fast with bail in large test suites.

Step 5 — Run only what the change affects. The cheapest test is the one you correctly skip. Vitest can restrict a run to tests reachable from the files a branch changed, turning suite time from a function of repository size into a function of diff size.

# scope the run to files changed since the base branch
npx vitest run --changed origin/main

The dependency-graph reasoning and the safety rails that stop you skipping something you needed are detailed in running only tests affected by a change.

Configuration Reference

The knobs below are the ones you will actually turn. Each has a safe default for a gating pull-request build and a different setting for a nightly full run.

The single rule that ties the table together: a pull-request build optimises for the fastest possible red or green, while a nightly build optimises for completeness. Bail and affected-selection make PR builds fast but partial; turning both off on main and nightly runs restores the full-suite guarantee that a threshold check or a release gate depends on.

A subtlety in the table deserves emphasis: sharding and matrix cancellation pull in opposite directions on a coverage-collecting build. Sharding wants every shard to finish so its partial coverage can be merged, which means fail-fast must stay false even on a gating PR build — otherwise one failing shard cancels its siblings and the merge job has nothing to combine. That is why bail lives at the runner level in this configuration rather than at the matrix level: runner-level bail shortens each shard’s own run without cancelling the others, so you keep both the compute saving and the complete set of per-shard reports. The moment you stop collecting per-shard artifacts, matrix fail-fast: true becomes available again as an even more aggressive saving. Reading the table as a set of independent switches misses these couplings; reading it as two coherent profiles — fast-partial and complete — makes the right combination obvious.

Verification & Assertions

Every orchestration change must be verified against a simple contract: it may make the pipeline faster, but it must never change which commits are considered passing. The way to prove this is to run the optimised pipeline and the naive pipeline against the same commit and assert identical pass/fail verdicts.

// scripts/verify-orchestration.ts — compare optimised vs full run
import { execSync } from 'node:child_process';

function verdict(cmd: string): boolean {
  try { execSync(cmd, { stdio: 'ignore' }); return true; }
  catch { return false; }
}

const full = verdict('npx vitest run');
const optimised = verdict('npx vitest run --changed origin/main --shard=1/1');

if (full !== optimised) {
  throw new Error(`Orchestration changed the verdict: full=${full} optimised=${optimised}`);
}
console.log('Verdicts match — orchestration is verdict-preserving.');

For sharding specifically, assert that the union of shard file lists equals the full file list and that the intersection is empty — no test should run twice and none should be dropped. For caching, assert a cache miss on a lockfile change and a cache hit on an unrelated code change; a cache that always hits is stale, and one that always misses is misconfigured. These assertions belong in the pipeline itself, not in a runbook, so a regression in orchestration fails loudly.

Edge Cases & Failure Modes

The most common failure is uneven shards. Vitest distributes files, not cases, so a single file with a thousand slow integration tests will make its shard the long pole regardless of n. The fix is to split fat test files or to weight the distribution, not to raise the shard count. A related trap is order-dependent tests that pass serially and fail once sharding changes their neighbours — that is not an orchestration bug but a latent test bug the orchestration surfaced, and it should be triaged through your flakiness workflow.

Caching has its own sharp edge: a cache key that is too coarse serves stale node_modules after a dependency bump, producing failures that reproduce only in CI. Always hash the lockfile, never a hand-maintained version string. Bail’s failure mode is informational rather than mechanical — a bailed run tells you a test failed but not how many, which is exactly wrong for a flaky suite where you want the full failure count. Reserve bail for suites you trust to be deterministic. Finally, affected-test selection is only as safe as its dependency graph: a test that reaches production code through a dynamic import or a config file the graph does not trace can be wrongly skipped, which is why every affected-selection setup needs a periodic full run as a backstop.

These failure modes share a family resemblance worth internalising: each is a case where an optimisation quietly changes the meaning of a run rather than just its speed. An uneven shard still produces the right verdict but a misleading timing profile. A stale cache produces a verdict against the wrong code. A bailed run produces a truthful but incomplete failure set. A wrong skip produces a green verdict that a full run would have turned red. In every case the pipeline still emits a pass-or-fail signal, so nothing looks broken until the day the hidden gap costs you a regression. The defence is the same across all four: hold a periodic, unoptimised full run as ground truth, and continuously assert that the fast pipeline’s verdict agrees with it. Optimisation you cannot audit against a baseline is not optimisation — it is a slow accumulation of blind spots, and the discipline of this topic area is refusing to accept a speed-up you cannot prove is verdict-preserving.

Performance & CI Impact

The levers compound multiplicatively, which is why measuring them together matters. Suppose a suite takes twelve minutes on one runner. Affected-test selection on a typical feature branch might cut the executed cases by 70%, dropping the base to under four minutes. Sharding that remainder across four runners returns it in roughly one minute of wall clock plus overhead. A warm dependency cache removes another ninety seconds of install that would otherwise sit on the critical path of every shard. The end state is a sub-two-minute pull-request signal from a suite that a naive pipeline runs in twelve — a six-fold improvement with no tests deleted and no coverage lost.

There is an ordering to these levers that maximises their combined effect. Affected-selection should run first, because pruning the suite shrinks the input every later stage operates on — there is no point sharding or caching work you were never going to run. Caching sits underneath everything as a constant-factor reduction on the overhead each remaining stage pays. Sharding then parallelises the pruned, warm-cached remainder, and bail trims the failure path of each shard last. Applied in that order the savings compound; applied out of order they interfere, most visibly when teams shard first and then discover affected-selection would have made most of their shards trivially empty. Think of orchestration as a pipeline of filters, each narrowing what reaches the next, with the full nightly run standing outside the pipeline as the untouched reference.

The cost side is real and worth naming. Every shard pays the fixed overhead of checkout, Node setup, and cache restore, so past a point adding shards buys diminishing wall-clock returns while linearly increasing billed runner-minutes. Caching adds storage cost and a small restore latency. Affected-selection adds graph-computation time and the operational risk of a wrong skip. The discipline is to profile before and after each change against a recorded baseline, keep the nightly full run as the completeness guarantee, and treat the pull-request pipeline as an aggressively optimised approximation whose correctness is continuously re-checked against that full run. Orchestration done well is invisible: developers simply notice that CI is fast and trustworthy, which is the entire point.

In This Topic Area

• Sharding Vitest across GitHub Actions runners — partition a suite across a matrix and merge the per-shard reports back into one verdict.
• Caching dependencies and test artifacts in CI — design cache keys that hit when they should and miss when they must, and hand artifacts between jobs.
• Failing fast with bail in large test suites — stop the pipeline at the first failure on the merge path without losing signal you need.
• Running only tests affected by a change — scope a run to the tests a diff can reach, with a backstop that stops wrong skips.

Related

• Back to Modern JavaScript Test Strategy & Pyramid Design
• Vitest vs Jest for CI speed — the runner choice that sets the baseline every lever here multiplies against.
• Quarantining flaky tests in CI — contain the flakiness that parallel orchestration tends to expose.
• Enforcing coverage thresholds in a monorepo — merge sharded coverage reports before gating on them.