Test Observability & Reporting

Most teams know exactly two things about their test suite: whether the last run was green, and a vague sense that it has got slower. Everything else — which tests fail most often, which ones cost the most minutes, whether flakiness is rising or falling, how long a developer waits for feedback — exists only as folklore. Test observability is the practice of emitting that data on every run and putting it somewhere people look, so that decisions about the suite are made from evidence rather than from whoever complained most recently. This topic sits under test pyramid strategy and covers the machine-readable reporters, the small amount of storage a trend needs, the annotations that put failures where they are seen, and the handful of numbers worth putting on a scorecard.

Architectural Scope & Boundaries

Test observability is about the suite, not about the application. The signals it produces — duration per file, failure frequency per test, retry counts, coverage movement — describe how well your testing is working, and they belong to a different audience and a different cadence from production monitoring. Confusing the two is a common mistake, and it usually ends with test metrics buried in an operations dashboard that nobody on the delivery team opens.

The technique has three distinct layers, and it is worth being clear which one you are building at any moment. Emission is the runner’s job: produce machine-readable output rather than only human-readable text. Collection is a small amount of plumbing: take that output, attach the commit and run identifiers, and append it somewhere durable. Presentation is where almost all the value is realised and almost none of the effort usually goes — the failure annotation on the pull request, the chart of wall clock over eight weeks, the five-line scorecard.

What this topic does not cover is the decision-making itself. Knowing that the end-to-end suite has grown from four to eleven minutes tells you the fact but not the remedy; the remedy comes from cost-benefit analysis of test layers and from the containment practices in flaky test mitigation. Observability’s job is to make the conversation start from numbers.

There is also a boundary of proportion. A team of six does not need a data warehouse; a JSON file committed to a metrics branch, or a small table in whatever database you already run, is enough to answer every question in this topic. The instinct to reach for a full analytics stack is where most test-observability projects die, because the plumbing becomes the project and the scorecard never ships.

One more distinction saves a lot of wasted effort. There is a difference between data about a single run and data about the suite over time, and they serve different people. A single run’s report answers “why did my change fail”, is consumed immediately, and can be thrown away in a week. The trend answers “is our testing getting better or worse”, is consumed monthly, and must be durable. Building one and expecting it to do the other job is the most common reason teams end up with an expensive artifact store and still cannot say whether the suite got slower this quarter.

Prerequisites

• Vitest 1.x or 2.x and Playwright 1.4x, both of which ship JUnit and JSON reporters
• A CI system that can upload artifacts and write to a job summary
• Somewhere durable to append results — a metrics branch, an object store bucket, or a small table
• Commit SHA and run identifier available as environment variables
• Agreement on which handful of numbers matter, before any dashboard is built
• A regular forum where the scorecard is actually read, such as an existing team review

Step-by-Step Implementation

Step 1 — Emit machine-readable output alongside the human one. Both runners take a list of reporters, so this costs nothing in developer experience: the console output stays the same and the files appear beside it.

// vitest.config.ts
import { defineConfig } from 'vitest/config';

export default defineConfig({
  test: {
    reporters: process.env.CI
      ? ['default', 'junit', 'json']
      : ['default'],
    outputFile: {
      junit: './reports/vitest-junit.xml',
      json: './reports/vitest-results.json',
    },
  },
});

// playwright.config.ts
export default defineConfig({
  reporter: [
    ['list'],
    ['junit', { outputFile: 'reports/playwright-junit.xml' }],
    ['json', { outputFile: 'reports/playwright-results.json' }],
    ['html', { open: 'never' }],
  ],
});

Gating the extra reporters on CI matters more than it appears. A local run that writes report files on every save fills the working tree with churn, and developers add the paths to their ignore file, which then hides the reports in CI too if the pattern is broad. Emitting them only where they are consumed keeps the local experience unchanged.

Step 2 — Normalise both runners into one record shape. Different tools, one vocabulary: this is what lets a single chart cover the whole pipeline instead of one chart per runner.

// scripts/collect.ts
import { readFileSync, writeFileSync, existsSync } from 'node:fs';

type TestRecord = {
  runId: string; commit: string; branch: string; at: string;
  suite: 'unit' | 'e2e';
  durationMs: number; passed: number; failed: number; flaky: number; skipped: number;
};

function fromVitest(path: string): TestRecord {
  const r = JSON.parse(readFileSync(path, 'utf8'));
  const tests = r.testResults.flatMap((f: any) => f.assertionResults);
  return {
    runId: process.env.GITHUB_RUN_ID ?? 'local',
    commit: process.env.GITHUB_SHA ?? 'local',
    branch: process.env.GITHUB_REF_NAME ?? 'local',
    at: new Date().toISOString(),
    suite: 'unit',
    durationMs: r.testResults.reduce((n: number, f: any) => n + (f.endTime - f.startTime), 0),
    passed: tests.filter((t: any) => t.status === 'passed').length,
    failed: tests.filter((t: any) => t.status === 'failed').length,
    flaky: tests.filter((t: any) => t.status === 'flaky').length,
    skipped: tests.filter((t: any) => t.status === 'pending').length,
  };
}

const history: TestRecord[] = existsSync('metrics/history.json')
  ? JSON.parse(readFileSync('metrics/history.json', 'utf8'))
  : [];
history.push(fromVitest('reports/vitest-results.json'));
writeFileSync('metrics/history.json', JSON.stringify(history, null, 2));

The normalisation deliberately throws away detail. Individual test names, stack traces and timings per assertion belong in the run’s own report, which the annotation step consumes and which expires. The history keeps only what a trend needs, which is why a JSON file stays small enough to commit for years rather than turning into a storage problem within months.

Step 3 — Put failures where the person who caused them is looking. A failure in a log file three clicks deep is a failure nobody reads. A job summary or an inline annotation is read by default.

// scripts/summary.ts
import { readFileSync, appendFileSync } from 'node:fs';

const r = JSON.parse(readFileSync('reports/vitest-results.json', 'utf8'));
const failures = r.testResults
  .flatMap((f: any) => f.assertionResults.map((a: any) => ({ file: f.name, ...a })))
  .filter((a: any) => a.status === 'failed');

const lines = [
  `## Tests: ${failures.length ? `${failures.length} failed` : 'all passed'}`,
  ...failures.slice(0, 15).map((f: any) => `- \`${f.fullName}\` — ${f.failureMessages[0]?.split('\n')[0]}`),
];
appendFileSync(process.env.GITHUB_STEP_SUMMARY!, lines.join('\n') + '\n');

Step 4 — Store the history somewhere boring. A JSON file on a metrics branch is durable, diffable, free, and entirely sufficient for years of daily runs. Reach for a database when you have a reason, not in advance.

# .github/workflows/metrics.yml
      - run: npx tsx scripts/collect.ts
      - run: |
          git config user.name "ci"
          git config user.email "ci@example.com"
          git add metrics/history.json
          git commit -m "metrics: ${GITHUB_SHA::7}" || exit 0
          git push origin HEAD:metrics

Step 5 — Render the trend as a picture, not a table. The purpose is to make a direction obvious at a glance, so a small committed chart beats a query anyone has to run. Any charting library works; what matters is that the output is committed next to the history and embedded wherever the team already reads.

// scripts/chart.ts — plot wall clock over the last sixty runs
import { readFileSync, writeFileSync } from 'node:fs';
import { chartToFile } from './lib/sparkline';

const history = JSON.parse(readFileSync('metrics/history.json', 'utf8')) as any[];
const daily = history.slice(-60).map((h) => ({ at: h.at, seconds: h.durationMs / 1000 }));

chartToFile(daily, {
  out: 'metrics/duration.png',
  title: 'Pull request suite wall clock, last 60 runs',
  yLabel: 'seconds',
  annotate: { threshold: 300, label: 'target: under five minutes' },
});

Mark the target on the chart rather than leaving it implicit. A line at five minutes turns “the suite is getting slower” into “we crossed our own threshold three weeks ago”, which is a far more actionable sentence in a planning discussion.

Step 6 — Keep the freshest report accessible without a download. Uploading an artifact is enough for archives, but a report that requires unzipping is a report nobody opens during triage. Publish the latest failure report to a stable location so a link in the failure notification lands directly on the trace.

Configuration Reference Table

Two of these deserve a comment. Playwright’s blob reporter exists specifically because sharded runs cannot be merged from JSON alone — it preserves attachments and retry structure, and merge-reports reconstructs a single report from the blobs. And GITHUB_STEP_SUMMARY is simply a file path; anything appended to it as markdown appears on the job page, which makes it the cheapest presentation surface available on most pipelines.

Verification & Assertions

Verify emission first, because everything downstream is worthless if the files are empty. After a run, the report files should exist, parse, and contain the number of tests you expect.

npx vitest run && node -e "
  const r = require('./reports/vitest-results.json');
  const n = r.testResults.flatMap(f => f.assertionResults).length;
  console.log('collected', n, 'test results');
  if (n === 0) process.exit(1);
"

Then verify collection under the conditions that break it, which is almost always sharding: four shards produce four report files, and a collector that reads one of them reports a quarter of the truth. Merge before collecting, and assert the merged total.

npx playwright test --shard=1/4 --reporter=blob
# …repeat for shards 2-4, then:
npx playwright merge-reports --reporter=json ./blob-report > reports/playwright-results.json

Finally, verify presentation by breaking something on purpose. Push a branch with a failing test and confirm the failure appears in the job summary with a readable message rather than only in the raw log. A reporting pipeline that has never been seen to surface a failure has not been tested, and this is a two-minute check.

A fourth verification is worth automating once the pipeline is established: assert that the history file actually grew. A collector that silently writes nothing — because a report path changed, or a merge step was skipped — produces a chart that simply stops moving, and a flat line is easy to mistake for stability. A three-line check that the newest record carries today’s date turns that silent failure into a visible one.

Edge Cases & Failure Modes

Sharded runs reported as separate suites. Each shard writes its own file, so naive collection records four short runs rather than one complete one, and the duration trend becomes meaningless. Diagnose by comparing the recorded test count to the real one; fix by merging reports before the collector runs.

History that grows without bound. A JSON file appended on every run becomes megabytes within a year, and the commit that writes it starts to dominate the job. Diagnose by file size; fix by rolling up — keep per-run records for thirty days and daily aggregates beyond that.

Metrics that nobody owns. A dashboard with no audience decays: a reporter breaks, the collector silently writes nothing, and the chart flatlines for weeks before anyone notices. Diagnose by checking whether the last data point is recent; fix by asserting freshness in CI, so a stale metrics file fails a job rather than quietly persisting.

Attributing flakiness to the wrong test. A test that fails because a previous test left state behind is recorded as the flaky one. Diagnose by checking whether failures cluster after a particular predecessor; fix the ordering dependency itself, using the approach in eliminating test order dependence, rather than trusting the ranking blindly.

Branch noise swamping the signal. Collecting from every branch mixes half-finished work into the trend, so the numbers jump for reasons that have nothing to do with the suite. Diagnose by grouping records by branch; fix by charting the default branch only and keeping other branches for per-run reporting.

Duration measured with the machine, not the suite. Wall clock on a shared runner reflects contention as much as test cost, so a spike may mean a busy machine rather than a slower suite. Diagnose by comparing against a CPU-time or test-count denominator; fix by tracking both, and by treating a single spike as noise and a sustained shift as signal.

Performance & CI Impact

The reporting itself is nearly free. A JUnit and JSON reporter alongside the console reporter costs milliseconds and a few hundred kilobytes; the collector is a script that runs in under a second. The one place cost appears is artifact upload on very large suites, where an HTML report with traces and screenshots can reach hundreds of megabytes — worth uploading on failure only, rather than on every run.

The bigger effect is indirect and positive. Once wall clock is a visible trend, slow tests get attention before they become intolerable, which is exactly the dynamic that keeps a suite inside the pull request pipeline instead of exiled to nightly. The same is true of flakiness: a rate that is visible falls, and one that is invisible rises, because nobody can prioritise what they cannot see.

Keep the collection step out of the critical path. It should run after the test job, ideally in a separate job that does not gate the merge, so a transient failure in the metrics plumbing never blocks a release. Metrics that can break the build are metrics that get deleted the first time they do.

One last point about audience, because it determines whether any of this survives. The scorecard is for the team that owns the suite, and it works when it is reviewed in a forum that already exists — a fortnightly engineering review, a sprint retrospective, whatever the team actually attends. A new meeting created specifically to review test metrics is the surest way to ensure the metrics are abandoned, because the meeting is the first thing cancelled in a busy month. Attach the numbers to an existing habit and they will still be there a year later.

It also pays to be explicit about who acts on each number. Feedback time is usually a platform or infrastructure concern; flake rate belongs to whoever owns the flakiest area; the skipped count belongs to whoever skipped the tests, which is why listing the skip annotations alongside the count is more useful than the count alone. A metric without an owner is a metric that gets discussed and never moved, and the scorecard is far more effective when each row has a name beside it.

In-Depth Guides

• Publishing JUnit reports to CI dashboards — emit, merge and upload results your CI can render natively.
• Tracking test duration trends over time — a durable history and a chart that makes drift obvious.
• Annotating pull requests with test failures — put the failure on the line that caused it.
• Building a test health scorecard — the five numbers and the cadence that make them matter.

Related

• Back to Modern JavaScript Test Strategy & Pyramid Design
• Flaky Test Mitigation — what to do once the flake rate is visible.
• Cost-Benefit Analysis of Test Layers — the decisions this data feeds.
• Setting up test pyramid metrics for enterprise teams — the same instinct at organisation scale.