Flaky Test Mitigation

A flaky test is one that passes and fails against identical code, and left unmanaged it corrodes the entire signal your suite produces. Once engineers learn that a red build might just be noise, they start re-running pipelines reflexively, merging through failures, and eventually ignoring the test results altogether — the exact opposite of what a test pyramid strategy is supposed to deliver. This guide treats nondeterminism as an engineering defect with a defined lifecycle: detect it, contain it without hiding real regressions, fix the root cause, and only then return the test to the trusted lane. The techniques here apply across the JavaScript stack, but the worked examples use Vitest as the primary runner and Playwright for browser-level checks, with concrete knobs for retries, quarantine annotations, and seeded fixtures that you can adopt incrementally.

Architectural Scope & Boundaries

Mitigation work sits at the seam between test authoring and CI orchestration, and it touches all three tiers of the pyramid differently. Unit tests are rarely flaky for environmental reasons; when they are, the cause is almost always shared mutable state or unseeded randomness, which is fixable at the source. Integration tests flake on timing, ordering, and leaked module state. End-to-end and browser tests flake on real network latency, animation timing, and resource contention under parallel load. The strategies in this section are deliberately tiered to match those causes.

Matching the fix to the tier matters because the same symptom — an intermittent red — has a different root cause and a different economical remedy at each level. A unit flake is almost never worth quarantining, because seeding or resetting shared state fixes it outright in minutes; paying the overhead of a separate lane for it would be wasteful. An end-to-end flake, by contrast, often reflects genuine timing or contention that takes real investigation, and containment buys the team the time to do that investigation without holding the merge queue hostage. Reading a flake by its tier tells you immediately whether to reach for a source fix, a retry budget, or the quarantine lane, and prevents the common mistake of applying a heavyweight CI mechanism to a problem a one-line seed would have solved.

This material covers four things and explicitly excludes a fifth. It covers: classifying a failure as a genuine regression versus nondeterminism; bounding retries so they buy stability without masking bugs; isolating unstable specs into a separate lane with measurable exit criteria; and removing the most common source of nondeterminism — unseeded data and uncontrolled time. It does not cover writing the underlying assertions or component harnesses; for that, see Playwright component testing and the broader Component & Integration Testing work. Mitigation assumes the test is correct in intent and only its determinism is in question.

The boundary that matters most is between containment and concealment. A retry that silently turns a real intermittent bug green is concealment; a retry that surfaces the flake in a report while keeping the merge queue moving is containment. Every knob in this section is chosen to stay on the containment side of that line. The practical test for any mitigation you introduce is a single question: after this change, is a genuine one-in-fifty product defect more visible or less visible? A quarantine lane that dashboards a growing flake population makes the defect more visible; a bumped retry count that closes the investigation makes it less. The two can look identical on a build-status badge — both show green — which is precisely why the discipline has to be judged by the data it preserves rather than by the colour of the pipeline.

A second boundary worth naming is between individual flakiness and systemic flakiness. A single racy assertion is an authoring bug you fix in place. But when a whole suite’s flake rate rises after a parallelism change or an infrastructure migration, no amount of per-test seeding will help, because the cause is contention or resource exhaustion that lives outside any one spec. Mitigation therefore operates at two scopes at once: it gives authors the tools to kill individual flakes at the source, and it gives platform owners a lane and a trend line to manage the systemic residue that individual fixes cannot reach.

Prerequisites

• Vitest 1+ or 2+ installed, with a working vitest.config.ts (see the Vitest configuration setup baseline).
• Playwright 1.4x installed with @playwright/test and a playwright.config.ts.
• CI that can run named jobs conditionally and upload artifacts (GitHub Actions assumed in examples).
• A way to persist run history — even a JSON file committed to a metrics branch works to start.
• @faker-js/faker (v8+) if your fixtures generate synthetic data.
• Agreement from the team on a flake budget: the maximum acceptable retry rate before a test is quarantined.
• Familiarity with the cost-benefit analysis of test layers so quarantine decisions weigh the value of each test honestly.

Step-by-Step Implementation

The lifecycle below moves a suspect test from detection through to either a clean fix or a justified removal. Each step has a focused, runnable configuration.

Step 1 — Make flakes observable before you react to them. You cannot manage what you cannot count. Configure Playwright to retain a trace on the first retry so every flake produces a forensic artifact rather than a vanished failure.

// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  retries: process.env.CI ? 2 : 0,
  reporter: [['list'], ['json', { outputFile: 'results.json' }]],
  use: {
    trace: 'on-first-retry',
    screenshot: 'only-on-failure',
    video: 'retain-on-failure',
  },
});

Step 2 — Bound retries with a deliberate budget, not an open door. Two retries is a common ceiling: it absorbs genuine one-in-a-thousand environmental blips while keeping a persistently failing test visibly red. Set retries to 0 locally so authors feel their own flakes immediately.

// vitest.config.ts
import { defineConfig } from 'vitest/config';

export default defineConfig({
  test: {
    // Vitest retries individual tests; keep it low and CI-only.
    retry: process.env.CI ? 2 : 0,
    reporters: ['default', 'json'],
    outputFile: { json: './vitest-results.json' },
  },
});

Step 3 — Tag the unstable specs. Annotate suspect tests so tooling can route them. Playwright supports tags directly in the title; Vitest uses a custom annotation convention you can filter on.

// example.spec.ts (Playwright)
import { test, expect } from '@playwright/test';

test('checkout completes @flaky', async ({ page }) => {
  await page.goto('/checkout');
  await expect(page.getByRole('status')).toHaveText('Order placed');
});

// example.test.ts (Vitest) — name-based tagging filtered in CI
import { test, expect } from 'vitest';

test('[quarantine] settles async price calc', async () => {
  expect(await computePrice()).toBe(4200);
});

Step 4 — Route tagged tests into a non-blocking lane. The main job excludes the quarantine tag and stays a hard merge gate; a second, non-blocking job runs only the tagged tests and reports trends without breaking the build.

# .github/workflows/test.yml (excerpt)
jobs:
  trusted:
    runs-on: ubuntu-latest
    steps:
      - run: npx playwright test --grep-invert @flaky
  quarantine:
    runs-on: ubuntu-latest
    continue-on-error: true   # informational, never blocks merge
    steps:
      - run: npx playwright test --grep @flaky

Step 5 — Attack the root cause with determinism. Most flakes that survive into quarantine are data- or time-driven. Freeze the clock and seed every random source so a fixture that fails on Tuesday at 23:59 UTC also fails on your laptop at noon.

// vitest.setup.ts
import { beforeEach, afterEach, vi } from 'vitest';
import { faker } from '@faker-js/faker';

beforeEach(() => {
  vi.useFakeTimers();
  vi.setSystemTime(new Date('2026-06-21T12:00:00Z'));
  faker.seed(20260621); // identical synthetic data every run
});

afterEach(() => {
  vi.useRealTimers();
});

Step 6 — Promote a test back to the trusted lane on evidence, not hope. Once a fix lands, the test must demonstrate stability — for example, 50 consecutive green runs in the quarantine lane — before its tag is removed. This exit criterion is what separates mitigation from sweeping the problem under the rug.

Configuration Reference Table

Verification & Assertions

Confirm the machinery works before trusting it. After enabling retries with tracing, force a known intermittent failure and check that the report distinguishes a flaky outcome from a clean pass. Playwright’s summary will read something like 1 flaky rather than 1 passed, and a trace.zip will appear under test-results/. That distinction is the whole point: a green build with zero flaky entries is trustworthy, while a green build with a rising flaky count is a warning you can act on.

For seeded determinism, assert reproducibility directly. Generate a fixture twice within the same seeded context and assert deep equality; then run the file in isolation versus inside the full suite and confirm identical output. A divergence proves state is leaking across files — the signature failure mode that quarantining alone would only hide.

import { test, expect, beforeEach } from 'vitest';
import { faker } from '@faker-js/faker';

beforeEach(() => faker.seed(42));

test('seeded fixture is reproducible', () => {
  const a = faker.person.fullName();
  faker.seed(42);
  const b = faker.person.fullName();
  expect(a).toBe(b);
});

The quarantine lane is verified by inspecting CI: the trusted job must turn red on a real regression while the quarantine job stays informational. Open a PR that breaks a quarantined test and confirm the merge button remains enabled; break a trusted test and confirm it blocks. This pair of deliberately broken pull requests is the contract that makes the whole system safe, and it is worth re-running whenever branch-protection rules change, because a single mislisted required check can silently promote the quarantine job back into a merge gate and undo the decoupling the lane was built to provide.

Verification should also cover the trend, not just a single run. A green build with zero flaky entries proves nothing about direction; only a series of builds does. Run the trend script across several consecutive CI runs and confirm the recorded flake count is flat or falling — a rising line, even under an all-green pipeline, is the earliest signal that a systemic flake source has appeared and that source-level fixes are not keeping pace with new instability. Treat that number as a first-class health metric alongside pass rate and suite duration, and review it on the same cadence you review coverage, so mitigation stays a measured programme rather than a reactive scramble each time a build goes red.

Edge Cases & Failure Modes

Retries that mask a real intermittent bug. If a feature genuinely fails one request in fifty, retries will paper over it and ship the defect. Guard against this by treating a rising flaky rate as a regression signal in its own right — track the count, alert on growth, and never let “it passed on retry two” close an investigation. The companion guide on retrying flaky Playwright tests without masking bugs covers the trace-driven triage that keeps retries honest.

The single distinction that separates a healthy mitigation programme from a dangerous one is whether a retry-passed test is collapsed into a green tally or kept as a distinct, tracked outcome. The runner already draws this line for you — Vitest and Playwright both report a flaky status that is neither passed nor failed — but a pipeline that discards that status the moment the build goes green throws away the only early warning it had. The diagram below contrasts the two paths a retry-rescued failure can take.

Quarantine becoming a graveyard. Tests dumped into the quarantine lane with no exit criteria accumulate forever, and coverage silently rots. Every quarantined test needs an owner and a deadline; if neither materializes, deleting the test is more honest than pretending it guards anything.

Order-dependent failures that seeding cannot fix. Seeding randomness and freezing time will not save a test that depends on another test having run first. Detect these by shuffling execution order (--sequence.shuffle in Vitest) and isolating the failures; the fix is proper teardown, not retries.

Shared singletons across parallel workers. Module-level caches, a single MSW server, or a shared database connection will produce contention flakes that scale with fullyParallel. Scope state to the worker or reset it per-file, mirroring the reset discipline used in external service simulation. The tell-tale signature is a test that passes in isolation and at low concurrency but fails intermittently as worker count rises: the failure rate tracks parallelism, not code, which points squarely at shared state rather than a logic bug. Reproduce it deliberately by pinning the worker count high locally before you attempt a fix, so you can confirm the fix under the same contention that produced the flake.

Environment-coupled flakes that only appear in CI. Some tests are stable on a developer laptop and flake only on the CI runner because the runner is slower, more contended, or configured with a different timezone or locale. These resist local reproduction entirely, which makes the retained trace and the frozen-clock discipline non-negotiable — without a forensic artifact from the actual failing environment you are debugging blind. Pin process.env.TZ and locale in the runner config, capture a trace on first retry, and treat any “works on my machine” flake as evidence that an ambient input is leaking in rather than as proof the test is fine.

Performance & CI Impact

Retries trade wall-clock time for stability, and the trade is asymmetric: a two-retry ceiling adds latency only to tests that actually fail, so a healthy suite pays almost nothing while a sick one pays loudly — which is the correct incentive. Tracing on-first-retry keeps artifact storage proportional to flake volume rather than total test count, avoiding the gigabytes that trace: 'on' would generate.

The quarantine lane’s biggest performance win is psychological and structural: by removing unstable tests from the merge gate, you stop the cascade of full-pipeline re-runs that flakes provoke, which is often the single largest source of wasted CI minutes. Deterministic seeding has near-zero runtime cost and frequently reduces it by eliminating the retry rounds those flakes would have triggered. When you measure the impact, fold it into the same ledger you use for balancing speed and coverage in monorepo testing so flake-mitigation spend is weighed against the feedback-loop time it buys back.

The hidden cost most teams miss is not machine time but human time. A flaky suite taxes every engineer who has to decide whether a red build is real, re-run it, wait, and re-decide — a cost that compounds across the whole team on every pull request and never appears on a CI invoice. Quantify it by tracking two numbers over time: the median wall-clock time from a red build to a confident triage decision, which should fall once flakes are contained and the trend line is visible, and the ratio of retried-then-passed runs to clean first-pass runs, which is a direct proxy for how much noise the team is absorbing. When those two numbers move in the right direction, the mitigation programme is paying for itself even before you count the CI minutes saved. Budget the effort accordingly: the first hour spent seeding a data-driven flake at the unit tier typically returns far more than the same hour spent tuning retry counts at the browser tier, because it removes the flake permanently instead of merely containing it, and permanent removal is the only change that also shrinks the human decision tax.

In-Depth Guides

• Retrying flaky Playwright tests without masking bugs — set a retry budget, capture trace-on-retry, and triage whether an intermittent failure is a real defect or genuine noise.
• Quarantining flaky tests in CI — tag unstable specs, run them in a non-blocking lane, dashboard the trend, and define the exit criteria that bring them back.
• Deterministic seeding for test data in Vitest — pin faker and PRNG seeds, freeze time, and build fixtures that produce byte-identical data on every run.

Related

• Back to Modern JavaScript Test Strategy & Pyramid Design
• Cost-Benefit Analysis of Test Layers — weigh the value of each flaky test before quarantining or deleting it.
• Time & Date Control Strategies — the clock-freezing foundation that removes a whole class of timing flakes.
• Playwright Component Testing — where browser-level determinism techniques are applied at the component tier.