Annotating Pull Requests With Test Failures

The distance between a failing test and the person who caused it is usually four clicks and a scroll through two thousand lines of log. Annotations close that distance: the failure appears on the changed line in the diff, with the assertion message, before anyone has opened a log at all. The mechanics are simple — parse the runner’s structured output, emit a workflow command per failure, write a summary — but the details decide whether the result is genuinely useful or one more piece of noise people learn to scroll past. This guide covers building annotations from Vitest and Playwright results, keeping the volume sane, and handling the cases where the failure’s location is not in the diff at all. It follows test observability and reporting.

Root Cause Analysis

Logs are optimised for a machine writing them, not for a human reading them later. A failed run’s log contains the setup output, the passing tests, the failure, and whatever ran afterwards, in an order determined by parallelism rather than by relevance. Finding the actual assertion means knowing what to search for, which the person who just pushed a one-line change generally does not.

The information needed is already structured. Both runners record, for each failure, the file, the line, the test name and the assertion message. The only thing missing is a step that reads it and speaks the CI system’s annotation protocol — a handful of lines that turn a buried stack trace into a red marker in the diff view.

What stops this from working is volume. A pipeline that emits eighty annotations for one broken helper teaches people to ignore annotations entirely, and once ignored they are hard to reintroduce. Useful annotation is therefore as much about suppression as about emission: cap the count, group by cause, and put the long tail in the summary rather than in the diff.

Reproducible Setup

Emit JSON from both runners so the annotation script has structured failures to read.

// vitest.config.ts
export default defineConfig({
  test: {
    reporters: process.env.CI ? ['default', 'json'] : ['default'],
    outputFile: { json: './reports/vitest-results.json' },
  },
});

// playwright.config.ts
export default defineConfig({
  reporter: process.env.CI
    ? [['list'], ['json', { outputFile: 'reports/playwright-results.json' }]]
    : [['list']],
});

# .github/workflows/test.yml — the annotation step must run on failure
      - run: npx vitest run
      - name: Annotate failures
        if: always()
        run: npx tsx scripts/annotate.ts

Implementation

Step 1 — Extract failures with a file and a line. The runner’s JSON carries the location in the stack trace; a small parser pulls the first frame that belongs to the repository rather than to a dependency.

// scripts/annotate.ts
import { readFileSync, existsSync } from 'node:fs';
import path from 'node:path';

type Failure = { file: string; line: number; col: number; title: string; message: string };

const FRAME = /\(?(\/[^\s():]+\.[jt]sx?):(\d+):(\d+)\)?/;

function locate(stack: string, fallbackFile: string): { file: string; line: number; col: number } {
  for (const raw of stack.split('\n')) {
    const m = raw.match(FRAME);
    if (m && !m[1].includes('node_modules')) {
      return { file: path.relative(process.cwd(), m[1]), line: Number(m[2]), col: Number(m[3]) };
    }
  }
  return { file: path.relative(process.cwd(), fallbackFile), line: 1, col: 1 };
}

function fromVitest(file: string): Failure[] {
  if (!existsSync(file)) return [];
  const report = JSON.parse(readFileSync(file, 'utf8'));
  return report.testResults.flatMap((suite: any) =>
    suite.assertionResults
      .filter((a: any) => a.status === 'failed')
      .map((a: any) => {
        const message: string = a.failureMessages?.[0] ?? 'Test failed';
        return { ...locate(message, suite.name), title: a.fullName, message };
      }),
  );
}

Step 2 — Emit one annotation per failure, in the CI system’s protocol. On GitHub this is a workflow command written to standard output; the newline escaping matters, because a raw newline terminates the command.

// scripts/annotate.ts (continued)
const escape = (s: string) =>
  s.replace(/%/g, '%25').replace(/\r/g, '%0D').replace(/\n/g, '%0A').replace(/:/g, '%3A');

function emit(f: Failure) {
  const summary = f.message.split('\n').slice(0, 6).join('\n');
  process.stdout.write(
    `::error file=${f.file},line=${f.line},col=${f.col},title=${escape(f.title)}::${escape(summary)}\n`,
  );
}

Step 3 — Cap the volume and group the rest. Twelve annotations are read; eighty are not. Emit the first dozen individually and summarise the remainder as a single entry.

// scripts/annotate.ts (continued)
const MAX_ANNOTATIONS = 12;

const failures = [...fromVitest('reports/vitest-results.json'), ...fromPlaywright('reports/playwright-results.json')];

for (const f of failures.slice(0, MAX_ANNOTATIONS)) emit(f);

if (failures.length > MAX_ANNOTATIONS) {
  const rest = failures.length - MAX_ANNOTATIONS;
  process.stdout.write(`::error title=More failures::${rest} further failure(s) — see the job summary\n`);
}

Step 4 — Write the full list to the job summary. The summary is unbounded and always visible, so it carries the long tail that the diff annotations deliberately omit.

// scripts/annotate.ts (continued)
import { appendFileSync } from 'node:fs';

const summaryPath = process.env.GITHUB_STEP_SUMMARY;
if (summaryPath) {
  const lines = [
    `## Test failures: ${failures.length}`,
    '',
    '| Test | Location | Message |',
    '| --- | --- | --- |',
    ...failures.map(
      (f) => `| ${f.title} | \`${f.file}:${f.line}\` | ${f.message.split('\n')[0].replace(/\|/g, '\\|')} |`,
    ),
  ];
  appendFileSync(summaryPath, lines.join('\n') + '\n');
}

Step 5 — Prefer the changed files when choosing what to annotate. A failure in a file the pull request touched is almost certainly caused by it; one elsewhere may be a knock-on effect. Sorting by that relevance puts the most likely cause at the top of a capped list.

// scripts/annotate.ts (continued)
import { execSync } from 'node:child_process';

const changed = new Set(
  execSync(`git diff --name-only origin/${process.env.GITHUB_BASE_REF ?? 'main'}...HEAD`, { encoding: 'utf8' })
    .split('\n')
    .filter(Boolean),
);

failures.sort((a, b) => Number(changed.has(b.file)) - Number(changed.has(a.file)));

A note on the Playwright side, which the code above assumes exists: its JSON report nests suites inside suites, so the extraction is a recursive walk rather than a flat map, and each failed result carries its own errors array with a location already resolved. That location is more reliable than parsing a stack trace, so prefer it where the runner gives it to you and keep the trace parser for runners that do not.

Verification

Verify the annotation renders by breaking a test on a branch and looking at the pull request rather than at the log. The marker should sit on the failing line with the assertion message visible.

npx vitest run --reporter=json --outputFile=reports/vitest-results.json || true
npx tsx scripts/annotate.ts
# ::error file=src/cart.test.ts,line=42,col=9,title=cart totals%3A sums line items::expected 42.5 …

Then verify the escaping, since an unescaped newline or colon silently truncates the annotation and produces a marker with no message — which looks like a tooling bug and is really a formatting one.

npx tsx scripts/annotate.ts | head -3 | grep -c '%0A'
# 1    ← multi-line messages survived as escaped newlines

Finally, verify the cap. Break a shared helper so dozens of tests fail, and confirm you get twelve annotations plus one overflow marker rather than a wall of red that makes the diff unreadable.

Troubleshooting

Symptom: annotations appear with an empty message. Diagnosis: the message contained a raw newline or colon, which terminates the workflow command early. Fix: escape as in Step 2 — this is the single most common cause, and it looks like a platform problem rather than an encoding one.

Symptom: the annotation lands on the wrong line. Diagnosis: the first stack frame belongs to the test framework or a helper, not to the test. Fix: skip frames inside node_modules and inside your own shared test utilities, so the first repository frame is the assertion the author wrote.

Symptom: no annotations at all on a failing run. Diagnosis: the step is skipped because the test step failed first. Fix: if: always() on the annotation step, and make sure the runner is configured to write the JSON report even when the run is red — a reporter that only writes on success is worse than none.

Symptom: annotations from a previous run persist. Diagnosis: the check is being re-created rather than updated, or a re-run added a second set. Fix: emit annotations from a single step in a single job, and let the CI system’s own check replacement handle the lifecycle rather than posting comments manually.

FAQ

Related

• Back to Test Observability & Reporting
• Publishing JUnit reports to CI dashboards — the durable record behind the annotations.
• Building a test health scorecard — aggregating what the annotations show one run at a time.
• Quarantining flaky tests in CI — keeping known-unstable failures out of the blocking path.