Keep a per-run duration history, normalise for machine noise, rank the slowest files, and alert on sustained drift rather than one slow run.

Tracking Test Duration Trends Over Time

Test suites do not become slow; they become slow one forty-millisecond test at a time, over months, with no single commit anyone would object to. By the time somebody says “the tests take forever”, the increase has compounded past the point where any one change explains it, and the conversation has nowhere to start. A duration history fixes that by making the slope visible while it is still shallow: you can see the week the suite crossed five minutes, name the change that did it, and decide deliberately. This guide covers recording the data, separating genuine slowdown from machine noise, ranking the files responsible, and alerting on sustained drift. It builds on test observability and reporting.

Root Cause Analysis

Wall clock is a noisy measurement of a real thing. The same suite on the same commit varies by tens of percent between runs on shared CI hardware, because the machine’s other work, the network, and cold caches all contribute. That noise is why a single measurement is useless and a trend is not: over twenty runs the noise averages out and the underlying slope remains.

The second complication is that a suite’s duration changes for several unrelated reasons, and treating them alike produces false alarms. Adding tests makes the suite legitimately longer; making existing tests slower does not. Changing the runner’s parallelism changes wall clock without changing any test. Moving from a two-core to a four-core runner halves the number while the suite is identical. A trend that does not separate these is a trend nobody trusts for long.

The third is granularity. A total for the whole suite tells you that something got slower; a per-file breakdown tells you what. Since both come from the same report, there is no reason to record only the total — and the ranked list of the ten slowest files is, in practice, the artifact people actually act on.

Individual runs are noisy; the rolling median shows the real slope Scattered per-run measurements vary widely from run to run, while a rolling median through them rises steadily, revealing a genuine slowdown that no single measurement would establish. median runs any one run could be blamed on the machine; the slope cannot
Track the rolling median, not the last run — the last run is mostly a measurement of the machine.

Reproducible Setup

Emit JSON from the runner, which carries per-file timings as well as the total, and prepare a place to append records.

// vitest.config.ts
export default defineConfig({
  test: {
    reporters: process.env.CI ? ['default', 'json'] : ['default'],
    outputFile: { json: './reports/vitest-results.json' },
  },
});
git checkout --orphan metrics && git rm -rf . && echo "[]" > history.json
git add history.json && git commit -m "metrics: init" && git push -u origin metrics
git checkout -

An orphan branch keeps the metrics history out of the main branch’s log entirely, which matters once it is being appended to on every run.

Implementation

Step 1 — Record a normalised record per run, not just a duration. Store the denominators you will need later: test count, worker count, and the runner’s core count. Without them, you cannot tell a slower suite from a bigger one or a smaller machine.

// scripts/record-duration.ts
import { readFileSync, writeFileSync, existsSync } from 'node:fs';
import os from 'node:os';

type Run = {
  at: string; commit: string; branch: string;
  totalMs: number; tests: number; files: number;
  workers: number; cores: number;
  msPerTest: number;
};

const r = JSON.parse(readFileSync('reports/vitest-results.json', 'utf8'));
const files = r.testResults as any[];
const tests = files.flatMap((f) => f.assertionResults).length;
const totalMs = files.reduce((n, f) => n + (f.endTime - f.startTime), 0);

const run: Run = {
  at: new Date().toISOString(),
  commit: (process.env.GITHUB_SHA ?? 'local').slice(0, 7),
  branch: process.env.GITHUB_REF_NAME ?? 'local',
  totalMs,
  tests,
  files: files.length,
  workers: Number(process.env.VITEST_MAX_THREADS ?? os.cpus().length),
  cores: os.cpus().length,
  msPerTest: Math.round(totalMs / Math.max(tests, 1)),
};

const history: Run[] = existsSync('history.json') ? JSON.parse(readFileSync('history.json', 'utf8')) : [];
history.push(run);
writeFileSync('history.json', JSON.stringify(history, null, 2));

Step 2 — Record the per-file breakdown too, but roll it up. Keeping every file’s timing for every run is wasteful; keeping the slowest twenty per run is enough to build a ranked list and costs almost nothing.

// scripts/record-files.ts
const slowest = files
  .map((f: any) => ({ name: f.name.replace(process.cwd(), ''), ms: f.endTime - f.startTime }))
  .sort((a, b) => b.ms - a.ms)
  .slice(0, 20);

writeFileSync(
  `by-file/${run.commit}.json`,
  JSON.stringify({ at: run.at, commit: run.commit, slowest }, null, 2),
);

Step 3 — Compare medians, not runs. A rolling median over the last twenty runs on the default branch is stable enough to act on and simple enough that nobody argues about the statistics.

// scripts/trend.ts
import { readFileSync } from 'node:fs';

const history = JSON.parse(readFileSync('history.json', 'utf8')) as any[];
const main = history.filter((r) => r.branch === 'main');

const median = (xs: number[]) => {
  const s = [...xs].sort((a, b) => a - b);
  return s[Math.floor(s.length / 2)] ?? 0;
};

const recent = median(main.slice(-20).map((r) => r.totalMs));
const baseline = median(main.slice(-60, -40).map((r) => r.totalMs));
const changePct = baseline ? ((recent - baseline) / baseline) * 100 : 0;

console.log(`median now ${(recent / 1000).toFixed(1)}s, was ${(baseline / 1000).toFixed(1)}s (${changePct.toFixed(1)}%)`);
console.log(`per test: ${median(main.slice(-20).map((r) => r.msPerTest))}ms`);

Step 4 — Alert on sustained drift, with a threshold that respects noise. A single slow run is noise; a median twenty per cent above the baseline for two weeks is a fact. Alert on the second and ignore the first.

// scripts/alert.ts
const DRIFT_PCT = 20;

if (changePct > DRIFT_PCT) {
  const worst = JSON.parse(readFileSync(`by-file/${main.at(-1)!.commit}.json`, 'utf8'));
  console.error(`Suite duration up ${changePct.toFixed(1)}% vs baseline.`);
  console.error('Slowest files:');
  for (const f of worst.slowest.slice(0, 5)) console.error(`  ${(f.ms / 1000).toFixed(1)}s  ${f.name}`);
  process.exit(1);
}
Distinguishing the four reasons wall clock changed More tests raises total time but leaves time per test flat; slower tests raises both; fewer workers raises total while per-test cost is unchanged; a smaller machine raises everything, which the recorded core count reveals. Cause total ms per test tells you by more tests added up flat test count existing tests slower up up the real alarm fewer workers up flat worker count smaller runner up up core count
Recording the denominators is what lets one number answer four different questions.

Step 5 — Publish the ranked list where it will be read. The list of the five slowest files, with their share of total time, is the artifact that turns a trend into an action.

node scripts/trend.ts >> "$GITHUB_STEP_SUMMARY"
# median now 268.4s, was 214.9s (24.9%)
# per test: 412ms
#   38.2s  /src/features/checkout/checkout.test.tsx
#   29.7s  /src/features/search/search.test.tsx

Verification

Verify the history is actually growing, because a silently broken collector produces a flat line that looks like stability.

node -e "
  const h = require('./history.json');
  const last = new Date(h.at(-1).at);
  const ageH = (Date.now() - last) / 36e5;
  console.log('records:', h.length, 'newest:', last.toISOString(), 'age:', ageH.toFixed(1) + 'h');
  if (ageH > 48) { console.error('metrics are stale'); process.exit(1); }
"

Then verify the normalisation works by changing a denominator on purpose. Halve the worker count for one run and confirm the total rises while time per test stays roughly flat — if both move, the record is not capturing what you think.

VITEST_MAX_THREADS=2 npx vitest run && node scripts/record-duration.ts
node -e "const h=require('./history.json'); console.log(h.at(-1))"
# { totalMs: 512000, tests: 640, workers: 2, cores: 8, msPerTest: 800 }

Finally, verify the alert fires. Add a deliberately slow test, let a few runs accumulate, and confirm the drift check goes red and names the file. An alert that has never fired is an alert you cannot rely on.

From a drift alert to a decision A sustained rise leads to the ranked file list, then to a choice: make the test faster, move it to a cheaper tier, or accept the cost deliberately and raise the baseline. drift alert median up 25% ranked files two files, 60% of it make it faster — fix the real cost move it down a tier accept it and raise the baseline
Accepting the cost is a legitimate outcome — provided it is a decision rather than a drift nobody noticed.

Troubleshooting

Symptom: the trend jumps when nothing changed. Diagnosis: the runner type changed — a hosted runner swapped size, or the job moved between pools. Fix: this is exactly why the core count is recorded; segment the trend by core count, or normalise to time per test per worker so the comparison survives a hardware change.

Symptom: feature-branch runs pollute the history. Diagnosis: the collector records every run rather than the default branch only. Fix: filter by branch when computing the trend, as in Step 3, while still recording everything — branch data is useful for the ranked list even when it is not comparable over time.

Symptom: the slowest-file list is dominated by one setup-heavy file every time. Diagnosis: a file that boots a database or a browser once per run legitimately carries that cost. Fix: record setup time separately where the runner reports it, so the list ranks by the test work rather than by a fixed overhead that no optimisation of the tests will touch.

FAQ

How many runs before a trend means anything?

Twenty on the default branch is a reasonable minimum for a stable median, which on an active repository is a couple of days. Below that the noise dominates and you will chase phantoms. If your repository is quiet, widen the window in time rather than reducing the count — a median over the last thirty days is fine.

Should duration failures block merges?

Blocking on absolute duration is brittle, because a legitimate new test suite can exceed the threshold. Blocking on per-test time, or on a percentage rise against the baseline, aligns much better with what you actually care about. Even then, a warning that names the files is usually enough — the point is to start the conversation, not to stop the work.

Where should this data live long term?

A JSON file on a metrics branch is sufficient for years, and its diffability is genuinely useful when investigating a step change. Roll up records older than thirty days into daily aggregates to keep the file small. Move to a database only when you have questions the file cannot answer, which for most teams never happens.

Does this replace profiling a slow test?

No — it identifies which test to profile. Once the ranked list names a file, the actual work is measuring where its time goes, which is a different exercise covered by the runner’s own tooling. The trend tells you where to look and when to look; it does not tell you what to change, and treating it as though it does leads to guessy optimisation.