Managing Visual Baselines Across Browsers

A screenshot baseline that passes on Chromium fails on WebKit, a baseline captured on macOS fails on the Linux CI runner, and now the visual suite is red for reasons that have nothing to do with any component. This guide is for engineers running Playwright 1.4x across multiple browser projects who need a baseline strategy that survives a real cross-browser, cross-OS matrix. We will explain why the same component paints differently across engines and operating systems, lay out a per-project snapshot folder structure, generate baselines in CI so they match the environment that verifies them, and keep the matrix from exploding into an unmaintainable pile of near-identical PNGs. It is the companion to snapshot testing with Playwright screenshots, which covers single-baseline capture; here the problem is many baselines and keeping them coherent, within the broader visual regression testing discipline.

Root Cause Analysis

A screenshot is the output of a browser’s rasterizer, and rasterizers are not interchangeable. Chromium, Firefox’s Gecko, and WebKit each implement their own text-shaping, antialiasing, and subpixel-hinting algorithms, so the same glyph at the same size lands on a different set of pixels in each engine. Layout math that involves fractional pixels — a flex item at 33.33% width, a border that falls on a half-pixel boundary — rounds differently across engines too. The consequence is unavoidable: a single baseline image cannot be correct for three engines simultaneously, because there is no single correct rendering. Each engine has its own ground truth.

Operating system is the second axis, and it is often underestimated. Font rendering is partly an OS service: the same Chromium build produces different antialiasing on macOS, Windows, and Linux because each OS supplies a different font rasterization stack and, frequently, different system fonts as fallbacks. This is why a baseline captured on an engineer’s macOS laptop reliably fails against the Linux container that runs CI, even for the same browser project. The baseline and the verifier disagree not about the component but about how the machine draws letters. The root cause of nearly every cross-browser baseline headache is a mismatch between where a baseline was generated and where it is checked — solve that, and most of the matrix problems dissolve.

One component, one source of truth, but a distinct baseline per engine and OS.

Reproducible Setup

Define one Playwright project per engine and a snapshot path template that keys baselines by project name, so each engine writes into its own folder automatically.

npx playwright install --with-deps chromium firefox webkit

// playwright.config.ts
import { defineConfig, devices } from '@playwright/test';

export default defineConfig({
  snapshotPathTemplate:
    '__screenshots__/{platform}/{projectName}/{testFilePath}/{arg}{ext}',
  projects: [
    { name: 'chromium', use: devices['Desktop Chrome'] },
    { name: 'firefox', use: devices['Desktop Firefox'] },
    { name: 'webkit', use: devices['Desktop Safari'] },
  ],
});

The {platform} and {projectName} tokens produce paths like __screenshots__/linux/chromium/…, so the same assertion resolves to a different baseline file per engine and OS with no per-test branching.

Implementation

Step 1 — Key every baseline by project and platform. With the template above, a single toHaveScreenshot('card.png') call transparently reads and writes __screenshots__/linux/webkit/card.spec.ts/card.png under the WebKit project on Linux. Author the test once; the path template fans it out.

import { test, expect } from '@playwright/test';

test('promo card', async ({ page }) => {
  await page.goto('/components/promo');
  await page.evaluate(() => document.fonts.ready);
  await expect(page.getByTestId('promo-card')).toHaveScreenshot('promo-card.png');
});

Step 2 — Generate baselines in CI, never locally. Because OS font rendering differs, the machine that produces a baseline must be the machine that verifies it. Add a CI job that runs the update flag and uploads the regenerated baselines as an artifact for review, rather than committing screenshots captured on a developer laptop.

# .github/workflows/update-baselines.yml
name: update-baselines
on: workflow_dispatch
jobs:
  update:
    runs-on: ubuntu-latest   # the SAME image that verifies in the merge gate
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: npx playwright install --with-deps
      - run: npx playwright test --update-snapshots
      - uses: actions/upload-artifact@v4
        with:
          name: updated-baselines
          path: __screenshots__/**

Step 3 — Pin the OS image. Whatever OS your merge-gate job uses, pin it to a specific runner image tag so an upstream image bump does not silently re-rasterize every baseline. Treat the runner image as part of the baseline contract.

jobs:
  visual:
    runs-on: ubuntu-24.04   # pinned — not ubuntu-latest

Step 4 — Keep the matrix small on purpose. A full engine-by-OS matrix multiplies baseline count and review load. Do not run every OS for every engine. Pick the smallest matrix that reflects real user distribution — commonly one Linux image for all three engines in CI, adding a second OS only for the handful of components where OS-specific rendering genuinely matters. Reducing degrees of freedom here is the same discipline as reducing flaky screenshots with deterministic rendering: fewer variables, fewer baselines to maintain.

// Run all engines on one CI OS; reserve extra OSes for a tagged subset.
const engines = ['chromium', 'firefox', 'webkit'];

Step 5 — Handle intentional cross-browser differences. Some components legitimately look slightly different per engine — a native form control, a scrollbar. Accept that by letting each engine own its baseline (which the path template already does) rather than trying to force one image to satisfy all three with a loose threshold. A loose threshold to paper over engine differences also hides real regressions.

Generator and verifier share one pinned image, so font rendering never disagrees.

Verification

Confirm the matrix is coherent by running the full visual suite across all engine projects and checking that every assertion resolves to an existing baseline — a missing baseline surfaces as a first-run “written” result, which in CI must be treated as a failure, not a silent create. Then verify that the CI-generated baselines actually match the CI verifier by running the merge-gate job twice with no code change; both runs must be green. A second run that fails proves the generator and verifier environments still disagree.

# List which baselines exist per project without capturing anything new
npx playwright test --list

# Fail loudly if any baseline is missing rather than auto-creating it in CI
CI=1 npx playwright test   # first-run "written" is reported as a failure under CI

The two-clean-runs check is the real proof. If a component’s baseline was generated on the pinned CI image and the verifier uses that same image, the diff must be empty. Any non-empty diff on an unchanged component means an environment variable — OS image tag, font package version, browser build — drifted between generation and verification. When that happens, resist the urge to regenerate baselines until you have identified which variable moved; regenerating without diagnosis simply re-pins the baseline to a moving target and the flake returns on the next drift.

A non-empty diff on unchanged code is an environment signal, not a component change.

Troubleshooting

Symptom: every WebKit baseline fails while Chromium and Firefox pass. Diagnosis: WebKit baselines were never generated, so the assertions are comparing against Chromium images or against nothing. Fix: run --update-snapshots with all three projects enabled on the CI image so each engine’s folder is populated, then commit or upload the WebKit set specifically.

Symptom: baselines pass in CI but a developer’s local run shows large full-image diffs. Diagnosis: the developer’s OS rasterizes fonts differently from the pinned CI image, so locally the baseline is simply wrong for their machine. Fix: do not verify visual tests locally against CI-generated baselines; run the visual project only in CI, or give developers a container matching the CI image. This mirrors the environment-parity rule in snapshot testing with Playwright screenshots.

Symptom: an OS image upgrade turned the whole visual suite red overnight. Diagnosis: runs-on: ubuntu-latest rolled to a new image with an updated font stack, re-rasterizing every baseline. Fix: pin runs-on to a specific image tag, and when you do intend to upgrade, regenerate all baselines in the same pull request that bumps the image so the mass update is deliberate and reviewable.

FAQ

Do I really need separate baselines per browser, or can one threshold cover all engines?

You need separate baselines. The rendering differences between Chromium, Firefox, and WebKit are large enough — whole-glyph shape and antialiasing, not just a stray pixel — that any threshold loose enough to let one baseline pass all three engines would also be loose enough to miss genuine regressions. Keeping one baseline per engine, which the snapshot path template does automatically, lets each engine stay under a strict tolerance.

Why generate baselines in CI instead of committing them from my machine?

Because operating systems rasterize fonts differently, and the baseline must match the machine that verifies it. A baseline captured on macOS will not match a Linux CI runner even for identical component code, producing permanent false failures. Generating baselines on the exact CI image that runs the merge gate guarantees the generator and verifier agree, which eliminates the single most common source of cross-environment flake.

How do I stop the baseline matrix from exploding?

Run all engines on a single CI operating system rather than a full engine-by-OS grid, and add a second OS only for the specific components where OS-level rendering differences actually matter to users. Most teams get sufficient coverage from three engines on one Linux image. Every extra OS multiplies both the storage footprint and the number of images a reviewer must approve on each intentional change.

What should happen when a first run has no baseline in CI?

It must fail the build, not silently create the baseline. A missing baseline in CI usually means a test was added without its reviewed image, and auto-creating one bakes an unreviewed — possibly buggy — rendering into the contract. Run CI so that a “written” first-run result is treated as a failure, and populate baselines only through the deliberate update job whose artifacts a human approves.

Back to Visual Regression Testing — the topic overview covering the full capture-and-compare loop.
Snapshot testing with Playwright screenshots — single-baseline capture, masking, and threshold tuning.
Reducing flaky screenshots with deterministic rendering — removing the rendering variables that also inflate the matrix.
Playwright component testing — the multi-project browser setup these baselines extend.