Detecting Flaky Tests by Repeating Runs in CI

Flaky tests are usually discovered by accident, at the worst possible moment: a red build on an unrelated pull request, at the end of a Friday, blocking a release. Detection turns that around. If you run the suite repeatedly on an unchanged commit, any test that does not always produce the same result is nondeterministic by definition, and you learn which ones on your own schedule rather than on the merge queue’s. This guide covers a repeat job, deriving per-test failure rates from run history, ranking flakes by the cost they impose, and feeding the result into the containment workflow. It sits under flaky test mitigation.

Root Cause Analysis

Nondeterminism hides because a single run cannot distinguish it from correctness. A test that fails one time in forty passes thirty-nine times, which looks exactly like a good test — until it is one of two hundred such tests, at which point most builds contain at least one failure and the suite stops being believed.

The arithmetic is worth internalising, because it explains why teams are so often surprised. With two hundred tests each failing one run in four hundred, the probability that a given build is red for no reason is about forty per cent. Each individual test looks fine; the aggregate is unusable. This is also why fixing the two worst offenders often transforms a pipeline: the distribution is usually heavily skewed.

The second reason flakes hide is that the evidence is thrown away. A retry turns a failure into a pass and the report shows green, so unless the retry is recorded explicitly there is nothing to count later. Detection therefore depends as much on keeping the data as on generating it.

Reproducible Setup

Two mechanisms feed detection: a deliberate repeat job, and retry data from ordinary runs.

// playwright.config.ts — record retries so ordinary runs contribute evidence
export default defineConfig({
  retries: process.env.CI ? 1 : 0,
  reporter: [['list'], ['json', { outputFile: 'reports/playwright-results.json' }]],
});

// vitest.config.ts
export default defineConfig({
  test: {
    retry: process.env.CI ? 1 : 0,
    reporters: ['default', 'json'],
    outputFile: { json: './reports/vitest-results.json' },
  },
});

Implementation

Step 1 — Run a scheduled repeat job on an unchanged commit. Both runners can repeat the suite in one invocation, which is far cheaper than scheduling twenty separate builds.

# .github/workflows/flake-hunt.yml
on:
  schedule: [{ cron: '0 2 * * *' }]
  workflow_dispatch:

jobs:
  repeat:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 20, cache: npm }
      - run: npm ci
      - run: npx vitest run --retry=0 --reporter=json --outputFile=reports/repeat-$RUN.json
        env: { RUN: nightly }
      - run: npx playwright test --repeat-each=10 --retries=0 --reporter=json
        continue-on-error: true

Note --retries=0 in the hunt job. Retries exist to keep the merge queue moving; here they would destroy the evidence you are trying to collect.

Step 2 — Run the repeats under realistic load. A flake caused by timing will not reproduce on an idle machine. Match the worker count and machine size of the real pipeline, or increase them to make timing flakes surface faster.

npx playwright test --repeat-each=10 --workers=8 --retries=0
# deliberately more workers than the real pipeline, to surface contention

Step 3 — Count outcomes per test, not per run. The unit of interest is a test’s failure rate across all attempts.

// scripts/flake/tally.ts
import { readFileSync, readdirSync } from 'node:fs';

type Tally = { runs: number; failures: number };
const byTest = new Map<string, Tally>();

for (const file of readdirSync('reports').filter((f) => f.startsWith('repeat-'))) {
  const report = JSON.parse(readFileSync(`reports/${file}`, 'utf8'));
  for (const suite of report.testResults) {
    for (const t of suite.assertionResults) {
      const key = `${suite.name.replace(process.cwd() + '/', '')} › ${t.fullName}`;
      const tally = byTest.get(key) ?? { runs: 0, failures: 0 };
      tally.runs++;
      if (t.status === 'failed') tally.failures++;
      byTest.set(key, tally);
    }
  }
}

const flaky = [...byTest.entries()]
  .filter(([, t]) => t.failures > 0 && t.failures < t.runs)   // neither always red nor always green
  .sort((a, b) => b[1].failures / b[1].runs - a[1].failures / a[1].runs);

for (const [name, t] of flaky) {
  console.log(`${((t.failures / t.runs) * 100).toFixed(1).padStart(6)}%  ${t.failures}/${t.runs}  ${name}`);
}

The filter is the important line: a test that fails every time is broken, not flaky, and mixing the two wastes the investigation.

Step 4 — Mine the retry data from ordinary runs too. The nightly hunt finds flakes under its own conditions; the retry record finds the ones that occur under real pipeline conditions, which is a different and equally useful population.

// scripts/flake/from-retries.ts — a Playwright result that passed after failing
const report = JSON.parse(readFileSync('reports/playwright-results.json', 'utf8'));

const walk = (suite: any, path: string[] = []): any[] =>
  [
    ...(suite.specs ?? []).flatMap((s: any) =>
      s.tests.filter((t: any) => t.results.length > 1 && t.status === 'expected')
        .map(() => [...path, s.title].join(' › ')),
    ),
    ...(suite.suites ?? []).flatMap((s: any) => walk(s, [...path, s.title])),
  ];

console.log(report.suites.flatMap((s: any) => walk(s)));

Step 5 — Rank by cost, not by rate. A test failing five per cent of the time in a job that runs two hundred times a week costs far more than one failing thirty per cent in a nightly job. Multiply the rate by the number of runs and by the time a failure wastes.

const RUNS_PER_WEEK = 190;
const MINUTES_LOST_PER_FAILURE = 9;   // re-run plus investigation

for (const [name, t] of flaky) {
  const rate = t.failures / t.runs;
  const weeklyMinutes = rate * RUNS_PER_WEEK * MINUTES_LOST_PER_FAILURE;
  console.log(`${weeklyMinutes.toFixed(0).padStart(5)} min/week  ${name}`);
}

Step 6 — Feed the ranking into containment. Detection without a next step just produces a list. The top entries go into the quarantine workflow with owners and deadlines, as described in quarantining flaky tests in CI.

One practical note on retention: keep the per-run tallies rather than only the latest ranking. A test that was fixed and has since started failing again is a different and more urgent story than one that has always been mildly unreliable, and you can only tell the two apart with history. A small append-only file per night is enough, and it makes the question “did this get worse after the framework upgrade” answerable in seconds.

Verification

Verify the hunt can actually find a flake by planting one. A test with a deliberate one-in-five random failure should appear in the ranking with roughly the expected rate.

import { test, expect } from 'vitest';

test('canary: deliberately flaky, remove before merging', () => {
  expect(Math.random()).toBeGreaterThan(0.2);   // fails ~20% of runs
});

npx vitest run --repeat-each=50 --retry=0 --reporter=json --outputFile=reports/repeat-canary.json
npx tsx scripts/flake/tally.ts | head -3
#  22.0%  11/50  src/canary.test.ts › canary: deliberately flaky

Then verify the tally distinguishes flaky from broken. A test that fails every time must not appear in the flake list, or genuinely broken tests will be quarantined instead of fixed.

Finally, verify the repeat job is running and producing data — a scheduled job that silently stopped weeks ago is the most common failure of this whole scheme, and a freshness check on the report directory catches it immediately.

Troubleshooting

Symptom: the hunt finds nothing but the pipeline is still flaky. Diagnosis: the hunt runs on an idle machine with default parallelism, so timing and contention flakes never surface. Fix: raise the worker count beyond the real pipeline’s, and run the repeat job on the same runner size as production CI rather than a larger one.

Symptom: the same test appears and disappears from the ranking. Diagnosis: the sample is too small — at ten repeats, a one-in-forty flake appears about a quarter of the time. Fix: accumulate across nights rather than treating each night as a fresh sample, which also makes the rate estimate far more stable.

Symptom: everything looks flaky after an infrastructure change. Diagnosis: a shared dependency became slower or less reliable, so the failures are environmental rather than per-test. Fix: check whether the failures cluster in time rather than by test; a spike across unrelated tests is an environment story, and quarantining individual tests would be the wrong response entirely.

FAQ

Related

• Back to Flaky Test Mitigation
• Quarantining flaky tests in CI — what to do with the ranking.
• Eliminating test order dependence — the most common cause the hunt surfaces.
• Tracking test duration trends over time — the same accumulation discipline for timing.