Running a Test Health Review Cadence

Suites degrade slowly and invisibly, and nobody is ever assigned to notice. A recurring review is how a team notices: thirty minutes on a fixed agenda, driven by numbers that were collected automatically, ending with decisions that have owners. It is a small commitment and the only reliable mechanism most teams find for keeping a suite healthy over years rather than months. This guide covers the agenda, the inputs that make it cheap to prepare, the rule that keeps it from becoming a status meeting, and how to tell whether it is working. It sits under test ownership models.

Root Cause Analysis

Test health has no natural owner and no deadline, so it loses to everything that has either. A flaky test is somebody’s problem only at the moment it blocks them, and their incentive at that moment is to get unblocked rather than to fix it. Multiply that by a year and the suite’s condition is the accumulated result of a thousand decisions nobody made deliberately.

The second reason is that the evidence is scattered. The information needed to see the decline — flake rate, duration trend, skipped count, quarantine population — exists in different places and nobody has a reason to assemble it. A review with no prepared inputs becomes an exchange of impressions, which produces no decisions and is quickly cancelled.

The third is that test-health meetings tend to become status reports. Someone reads the numbers aloud, everyone agrees they are concerning, and nothing is assigned. A review earns its half hour only if every item on the agenda leaves with an outcome and a name attached to it.

Reproducible Setup

The agenda should be generated, not written. Everything on it comes from data the pipeline already produces.

// scripts/review/agenda.ts
import { readFileSync, writeFileSync } from 'node:fs';

const history = JSON.parse(readFileSync('metrics/history.json', 'utf8'));
const quarantine = JSON.parse(readFileSync('metrics/quarantine.json', 'utf8'));
const skips = JSON.parse(readFileSync('metrics/skipped.json', 'utf8'));

const lines = [
  `# Test health review — ${new Date().toISOString().slice(0, 7)}`,
  '',
  '## 1. Scorecard (5 min)',
  ...scorecardRows(history).map((r) => `- ${r.label}: **${r.value}** (${r.delta}) — ${r.owner}`),
  '',
  '## 2. Quarantine (10 min)',
  ...quarantine.map((q: any) => `- \`${q.test}\` — owner ${q.owner}, deadline ${q.deadline}${q.overdue ? ' **OVERDUE**' : ''}`),
  '',
  '## 3. New skips since last review (5 min)',
  ...skips.filter((s: any) => s.addedSinceLastReview).map((s: any) => `- \`${s.test}\` — added by ${s.author}`),
  '',
  '## 4. Slowest five files (5 min)',
  ...slowestFiles(history).map((f: any) => `- ${(f.ms / 1000).toFixed(1)}s — \`${f.name}\``),
  '',
  '## 5. Decisions (5 min)',
  '| Item | Decision | Owner | By |',
  '| --- | --- | --- | --- |',
];

writeFileSync('metrics/REVIEW.md', lines.join('\n'));

# .github/workflows/review-agenda.yml
on:
  schedule: [{ cron: '0 7 1 * *' }]     # the morning of the review
jobs:
  agenda:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with: { ref: metrics }
      - run: npx tsx scripts/review/agenda.ts
      - run: cat metrics/REVIEW.md >> "$GITHUB_STEP_SUMMARY"

Implementation

Step 1 — Attach it to a forum that already exists. A new recurring meeting is the first casualty of a busy month. Thirty minutes appended to an existing engineering review survives; a standalone invitation does not.

Step 2 — Keep the agenda fixed and short. Five items, thirty minutes, the same order every time. Predictability is what lets people prepare in two minutes rather than skipping it.

1. Scorecard — five numbers and their direction (5 min)
2. Quarantine — every entry, by name, with its deadline (10 min)
3. New skips since last time (5 min)
4. The five slowest files (5 min)
5. Decisions — what changes, who does it, by when (5 min)

Step 3 — Give the quarantine section the most time, because it is where decisions are hardest. Each entry gets one of three outcomes and nothing else: promote, extend once with a reason, or retire.

// scripts/review/quarantine-decisions.ts
type Outcome = 'promote' | 'extend' | 'retire';

export function applyDecision(test: string, outcome: Outcome, owner: string) {
  // promote: remove the tag, the test rejoins the blocking lane
  // extend:  new deadline, recorded with the reason; only once
  // retire:  delete the test and close the ticket
}

Step 4 — Require an outcome per item, recorded in the same file. The decisions table is the meeting’s only artifact, and it is what next month’s review opens with.

## 5. Decisions

| Item | Decision | Owner | By |
| --- | --- | --- | --- |
| `checkout › promo code` flake | retire — replaced by a unit test | @sam | 2026-10-02 |
| e2e suite 4m12s → 5m01s | split `checkout.spec.ts` into two files | @dana | 2026-10-09 |
| 5 new skips in `billing` | investigate, report next review | @priya | 2026-10-16 |

Step 5 — Open the next review by checking the previous decisions. This single habit does more for follow-through than any tracking tool, because an unfinished item is read aloud in front of the people who agreed it.

Step 6 — Let the review change the policy. A rule that generated three exceptions this month is a rule to revise, and this is the forum with both the evidence and the authority, as described in writing a testing policy a team will follow.

A last structural point: keep the review’s artifact in the repository rather than in a document tool. The decisions table belongs on the metrics branch beside the numbers that produced it, which means it is versioned, greppable, and available to the script that opens the next review. A meeting whose only record lives in someone’s notes application loses its history the first time that person changes teams, and the history is most of what makes the review worth holding.

Verification

Verify the review is producing change rather than conversation. Count decisions made and decisions completed over a quarter; a completion rate below half means the items are too large or the owners were volunteered rather than volunteering.

git log --since="3 months ago" -p -- metrics/REVIEW.md \
  | grep -cE "^\+\| .* \| .* \| @" 
# 14 decisions recorded
grep -c "done" metrics/decisions-log.md
# 11 completed — a healthy rate

Then verify the inputs are fresh, because a review driven by stale numbers quietly teaches people to distrust the whole exercise.

node -e "
  const h = require('./metrics/history.json');
  const ageDays = (Date.now() - Date.parse(h.at(-1).at)) / 864e5;
  console.log('newest metric is', ageDays.toFixed(1), 'days old');
  if (ageDays > 3) { console.error('stale — fix the collector before the review'); process.exit(1); }
"

Finally, verify the meeting is still thirty minutes. Growth in duration is the earliest sign that it is turning into a status report, and the remedy is to move discussion out and keep only the decision in.

Troubleshooting

Symptom: the same items appear every month. Diagnosis: decisions are too large to complete between reviews, or nobody genuinely owns them. Fix: break each into a change one person can make in an afternoon, and have owners volunteer rather than be assigned — an unwilling owner is an item that will reappear.

Symptom: attendance falls away. Diagnosis: the meeting is producing no visible change, so the time is not earning its place. Fix: pick the single most irritating thing — usually the worst flake — and fix it as a result of the review, visibly. One completed decision restores attendance faster than any amount of process.

Symptom: the review turns into a debate about metrics. Diagnosis: the numbers arrive without their assumptions, so they can be argued with indefinitely. Fix: publish the inputs alongside the values, as in building a test health scorecard, and timebox the discussion — if a number is disputed, the decision is to verify it, not to relitigate it live.

Symptom: quarantined tests are extended indefinitely. Diagnosis: extension is the comfortable choice and nothing limits it. Fix: allow one extension per test, ever, and make retirement the automatic outcome of a second miss. The rule is uncomfortable exactly once, after which quarantine starts working as intended.

FAQ

Related

• Back to Test Ownership Models
• Building a test health scorecard — the first agenda item.
• Quarantining flaky tests in CI — the workflow this review keeps honest.
• Writing a testing policy a team will follow — what the review revises.