Pick five numbers that describe suite health, compute them from data you already have, publish them regularly, and give each an owner.

Building a Test Health Scorecard

A scorecard is not a dashboard. A dashboard is a place you go when you already suspect a problem; a scorecard arrives on a schedule, contains five numbers, and is read by people who were not looking for it. That difference is why scorecards change behaviour and dashboards usually do not. This guide covers choosing the five numbers for a JavaScript test suite, computing them from data the runners already emit, publishing them where the team already looks, and — the part that decides whether any of it matters — assigning each row an owner. It assumes you have the collection plumbing from test observability and reporting and a history of runs to draw on.

Root Cause Analysis

Test metrics fail to change anything for three reasons, and all three are about presentation rather than data. The first is volume: a page with thirty charts has no message, so readers take away nothing and stop opening it. Five numbers with a direction each can be absorbed in fifteen seconds, which is the actual attention budget available.

The second is the absence of a comparison. A flake rate of 1.2% means nothing on its own — the reader has no idea whether that is good. The same number shown as “1.2%, down from 1.9% last month, target under 1%” is immediately interpretable, and interpretability is what makes a number discussable.

The third and most decisive is ownership. A metric that belongs to everyone belongs to nobody: it gets noted in a meeting, nobody’s week changes, and it appears again unchanged next month. Attaching a name to each row converts an observation into an expectation, and it is the single change that most reliably turns a scorecard from decoration into pressure.

The anatomy of a row that changes behaviour Each row carries the metric name, the current value, the direction since last period, an explicit target and an owner — remove any one and the row becomes an observation rather than an expectation. metric value direction target owner what is measured where we are better or worse is it good? who acts Drop the target and nobody can interpret it; drop the owner and nobody acts on it.
Five columns per row, and the last two are the ones teams most often omit.

Reproducible Setup

Everything here reads the run history and the latest reports. No new instrumentation is needed if the collector is already running.

// scripts/scorecard/types.ts
export type Row = {
  key: string;
  label: string;
  value: string;
  delta: string;
  target: string;
  owner: string;
  healthy: boolean;
};

export type Run = {
  at: string; branch: string;
  totalMs: number; tests: number; failed: number; flaky: number; skipped: number;
  retries: number;
};
ls metrics/
# history.json        one record per run
# by-file/            slowest twenty files per run
# coverage/           per-package coverage summaries

Implementation

Step 1 — Choose five numbers, and refuse the sixth. These five answer the questions teams actually ask. Adding more dilutes the message; if something new matters more, replace a row rather than appending one.

// scripts/scorecard/metrics.ts
export const METRICS = [
  { key: 'feedback',  label: 'Feedback time (pull request suite)', target: '< 5m',   owner: 'platform' },
  { key: 'flake',     label: 'Flake rate',                          target: '< 1%',   owner: 'web-core' },
  { key: 'slowShare', label: 'Share of time in the slowest 10 files',target: '< 40%',  owner: 'web-core' },
  { key: 'skipped',   label: 'Skipped tests',                       target: '0 new',  owner: 'each team' },
  { key: 'coverage',  label: 'Coverage, domain packages',           target: '>= 85%', owner: 'domain' },
];

Step 2 — Compute each from the history, comparing periods rather than runs. Use a median over the current window against the same window a period earlier, so a single unusual run cannot move a row.

// scripts/scorecard/compute.ts
import { readFileSync } from 'node:fs';
import type { Run } from './types';

const history = (JSON.parse(readFileSync('metrics/history.json', 'utf8')) as Run[])
  .filter((r) => r.branch === 'main');

const median = (xs: number[]) => {
  const s = [...xs].sort((a, b) => a - b);
  return s.length ? s[Math.floor(s.length / 2)] : 0;
};

const windowOf = (daysAgoFrom: number, daysAgoTo: number) => {
  const now = Date.now();
  return history.filter((r) => {
    const age = (now - Date.parse(r.at)) / 864e5;
    return age >= daysAgoTo && age < daysAgoFrom;
  });
};

export const current = windowOf(30, 0);
export const previous = windowOf(60, 30);

export const feedbackSeconds = (rs: Run[]) => median(rs.map((r) => r.totalMs)) / 1000;
export const flakeRate = (rs: Run[]) => {
  const flaky = rs.reduce((n, r) => n + r.flaky, 0);
  const total = rs.reduce((n, r) => n + r.tests, 0);
  return total ? (flaky / total) * 100 : 0;
};

Step 3 — Render a direction for every row. The delta is what makes the number interpretable, and it should say better or worse rather than only up or down, since for some rows down is good.

// scripts/scorecard/render.ts
export function delta(now: number, before: number, lowerIsBetter: boolean) {
  if (!before) return 'new';
  const diff = now - before;
  const pct = (diff / before) * 100;
  const better = lowerIsBetter ? diff < 0 : diff > 0;
  const arrow = diff === 0 ? 'flat' : better ? 'improving' : 'worsening';
  return `${arrow} ${Math.abs(pct).toFixed(1)}%`;
}

Step 4 — Publish it where the team already is. A markdown table posted to the team’s channel and committed to the repository beats a bespoke page nobody bookmarks.

// scripts/scorecard/publish.ts
import { writeFileSync } from 'node:fs';
import type { Row } from './types';

export function toMarkdown(rows: Row[], period: string) {
  return [
    `# Test health — ${period}`,
    '',
    '| Metric | Now | Change | Target | Owner |',
    '| --- | --- | --- | --- | --- |',
    ...rows.map((r) => `| ${r.label} | ${r.healthy ? r.value : `**${r.value}**`} | ${r.delta} | ${r.target} | ${r.owner} |`),
    '',
    `_Generated from ${period}; source: metrics/history.json_`,
  ].join('\n');
}

writeFileSync('metrics/SCORECARD.md', toMarkdown(rows, 'September 2026'));
Cadence and audience for each reporting surface Per-run annotations serve the author immediately, the weekly trend serves whoever owns the suite, and the monthly scorecard serves the whole team in a forum that already exists. per run annotations in the diff audience: the author acted on in minutes weekly duration and flake trend audience: suite owners acted on in days monthly the five-row scorecard audience: the whole team acted on in quarters
Three surfaces, three cadences — the scorecard is the slowest and the only one with named owners.

Step 5 — Schedule it and let it post itself. A scorecard that depends on somebody remembering to generate it will be generated twice.

# .github/workflows/scorecard.yml
on:
  schedule: [{ cron: '0 8 1 * *' }]
  workflow_dispatch:
jobs:
  scorecard:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with: { ref: metrics }
      - run: npx tsx scripts/scorecard/publish.ts
      - run: cat metrics/SCORECARD.md >> "$GITHUB_STEP_SUMMARY"

Verification

Verify the computation against a period you remember. Pick a month when something notable happened — a big refactor, a runner change — and confirm the scorecard reflects it.

npx tsx scripts/scorecard/publish.ts --period 2026-07
# | Feedback time | 6m 02s | worsening 31.2% | < 5m | platform |
# ← matches the month the end-to-end suite doubled; the data is sound

Then verify the windows do not overlap and that both contain enough runs to be meaningful. A period with four runs produces a median that is essentially one measurement.

node -e "
  const { current, previous } = require('./scripts/scorecard/compute');
  console.log('current', current.length, 'previous', previous.length);
  if (current.length < 20 || previous.length < 20) console.warn('windows are thin; widen them');
"

Finally, verify the thing that actually matters: that the scorecard is read. The honest check is whether the last three editions produced a change in anyone’s work — a test deleted, a slow file split, a flake fixed. If not, the problem is the forum or the owners, not the numbers, and adding a sixth metric will not help.

Why scorecards stop working Too many metrics dilute the message, missing targets make values uninterpretable, absent owners mean nobody acts, and a dedicated meeting gets cancelled — each failure has a specific remedy. Failure Remedy thirty charts, no message five rows, replace rather than add values with no target state the target on every row no owner per row name a team, not the team its own meeting attach it to a forum that exists
Every one of these failures is about presentation and ownership — none is about the data.

Troubleshooting

Symptom: a row moves sharply with no known cause. Diagnosis: the window caught an infrastructure change — a runner size, a parallelism setting, a sharding change. Fix: record those as annotations on the history so the scorecard can print them beside the row; an unexplained jump destroys trust far faster than a bad number.

Symptom: the skipped count keeps rising and nobody notices. Diagnosis: the row reports a total rather than what changed, so five new skips hide inside a total of seventeen. Fix: report new skips this period and list them by name — a number nobody can act on is not a metric, it is a statistic.

Symptom: coverage on the scorecard disagrees with the coverage gate. Diagnosis: the scorecard averages across packages while the gate enforces per package. Fix: report the domain packages specifically, as in Step 1, and say so in the label; a single repository-wide coverage figure in a monorepo is a number without a meaning.

FAQ

Why five metrics rather than ten?

Because the constraint is attention, not data. Five rows are read in full; ten are skimmed; twenty are ignored. If a sixth number genuinely matters more than one of the five, swap it in — the discipline of keeping the count fixed is what forces the conversation about which measures actually drive decisions.

Should the scorecard include coverage at all?

Include it, but scoped and with modest expectations. Coverage is easy to collect and weakly informative, so it earns a row only when limited to code where it means something — domain packages rather than the whole repository. Pair it with the stronger signal from mutation testing and assertion quality if you run it, since that measures detection rather than execution.

What cadence works best?

Monthly for the scorecard, weekly for the trend, per run for annotations. Monthly is slow enough that the numbers move meaningfully between editions and fast enough that a regression is caught within a quarter. Weekly scorecards produce noise and reader fatigue; quarterly ones arrive after the damage.

How do I stop it becoming a stick to beat teams with?

By framing rows as shared constraints rather than performance grades, and by making sure the owner named on each row is a team that can actually change it. A flake rate owned by the team that writes the flakiest tests is fair; a feedback-time row owned by a team with no control over CI capacity is not, and it is the fastest way to make people resent the whole exercise.