Put a number on what tests cost — compute minutes, developer waiting and maintenance hours per tier — so coverage debates rest on data.

Measuring the Running Cost of a Test Suite

Arguments about whether a test suite is worth its cost go in circles because nobody has the cost. Everyone has an impression — “the end-to-end tests are slow”, “we spend too long on fixtures” — and impressions do not settle anything. This guide produces three numbers per tier: compute minutes, developer waiting time, and maintenance hours. Together they are enough to answer the questions that actually come up, such as whether to add a browser to the matrix, whether a flaky suite is worth repairing, and whether a tier has grown past its value. It is for tech leads who need to make a case with evidence, and it sits under cost-benefit analysis of test layers.

Root Cause Analysis

Test cost is invisible because it is paid in three separate currencies that never appear on the same page. Compute minutes appear on an infrastructure bill that the delivery team rarely sees. Waiting time is paid by every developer in small increments that nobody logs. Maintenance is paid in pull requests that look like ordinary work. Each is individually easy to dismiss, and together they are usually the largest recurring cost in the repository.

The second problem is that the three are not interchangeable and should not be summed into one figure. Compute is cheap and elastic — doubling it is a purchase decision. Waiting time is expensive and inelastic, because a developer blocked for twelve minutes does not produce eleven minutes of something else. Maintenance is the most expensive per hour and the most variable, because a flaky suite consumes attention unpredictably.

The third is attribution. A total for the whole suite cannot settle a question about one tier, and the questions people actually ask are always about a tier or a file: is this end-to-end suite worth it, should we keep this visual regression check. Measure per tier from the start, because retro-fitting attribution to aggregate data is not possible.

Three currencies of test cost Compute minutes are cheap and elastic, developer waiting time is expensive and inelastic, and maintenance hours are the most expensive and the most variable — each is measured differently and none substitutes for another. compute minutes from the CI bill cheap, elastic buy more if it helps easy to measure, easy to fix waiting time runs × people × duration expensive, inelastic cannot be bought back the number that changes minds maintenance hours from the commit history costly, variable worst when flaky hardest to see, often the largest
Keep the three separate: summing them into one figure hides exactly the distinctions that decide what to do.

Reproducible Setup

All three numbers come from data you already have: CI run records, the git history, and a little arithmetic.

# runs per week on the default branch and on pull requests
gh run list --workflow=pr.yml --limit 200 --json createdAt,conclusion,databaseId > runs.json
node -e "
  const runs = require('./runs.json');
  const week = runs.filter(r => Date.now() - Date.parse(r.createdAt) < 7*864e5);
  console.log('runs in the last 7 days:', week.length);
"
# per-job duration, which is where the per-tier split comes from
gh run view 1234567890 --json jobs --jq '.jobs[] | {name: .name, minutes: (((.completedAt|fromdateiso8601) - (.startedAt|fromdateiso8601))/60)}'
# {"name":"unit tests","minutes":2.4}
# {"name":"end-to-end","minutes":11.8}

Implementation

Step 1 — Compute minutes per tier per week. Multiply job duration by runs and by the runner’s cost multiplier, which differs between sizes and between hosted and self-hosted.

// scripts/cost/compute.ts
type Job = { tier: 'unit' | 'component' | 'e2e'; minutes: number; multiplier: number };

const RUNS_PER_WEEK = 310;

const jobs: Job[] = [
  { tier: 'unit',      minutes: 2.4,  multiplier: 1 },
  { tier: 'component', minutes: 4.1,  multiplier: 1 },
  { tier: 'e2e',       minutes: 11.8, multiplier: 4 },   // 4 shards
];

const RATE_PER_MINUTE = 0.008;

for (const j of jobs) {
  const weekly = j.minutes * j.multiplier * RUNS_PER_WEEK * RATE_PER_MINUTE;
  console.log(`${j.tier}: ${(weekly).toFixed(0)} per week, ${(weekly * 52).toFixed(0)} per year`);
}
// unit: 6 per week, 310 per year
// component: 10 per week, 529 per year
// e2e: 117 per week, 6089 per year

Step 2 — Convert wall clock into waiting time. Only the critical path counts: jobs that run in parallel cost their maximum, not their sum, and only the portion a developer actually waits for is waiting time.

// scripts/cost/waiting.ts
const CRITICAL_PATH_MINUTES = 12.4;     // longest job, not the total
const PR_RUNS_PER_WEEK = 190;           // runs a person waits on
const ATTENTION_FRACTION = 0.5;         // half the wait is genuinely lost
const HOURLY = 75;

const hoursPerWeek = (CRITICAL_PATH_MINUTES * PR_RUNS_PER_WEEK * ATTENTION_FRACTION) / 60;
console.log(`waiting: ${hoursPerWeek.toFixed(1)} h/week ≈ ${(hoursPerWeek * HOURLY * 52).toFixed(0)}/yr`);
// waiting: 19.6 h/week ≈ 76,570/yr

That ratio — six thousand a year in compute against seventy-six thousand in waiting — is the point of the whole exercise, and it reverses the instinct most teams start with.

Step 3 — Measure maintenance from the history. Changes to test files that are not accompanied by source changes are, to a good approximation, test maintenance.

#!/usr/bin/env bash
# scripts/cost/maintenance.sh — commits touching only tests, last 90 days
git log --since="90 days ago" --format='%H' | while read -r sha; do
  files=$(git show --name-only --format= "$sha")
  if [ -n "$files" ] && ! echo "$files" | grep -qv -E '(\.test\.[jt]sx?|\.spec\.[jt]s|e2e/|__tests__/)'; then
    echo "$sha $(git show -s --format='%s' "$sha")"
  fi
done | tee test-only-commits.txt | wc -l

Step 4 — Attribute maintenance to a tier. Group those commits by the directory they touch, which maps directly onto tiers in most repositories.

while read -r sha _; do
  git show --name-only --format= "$sha"
done < test-only-commits.txt | sed -E 's|^(e2e)/.*|e2e|; s|.*\.test\.tsx$|component|; s|.*\.test\.ts$|unit|' | sort | uniq -c | sort -rn
#  84 e2e
#  31 component
#  12 unit
Where the cost of a typical suite actually falls Compute is a small fraction of total cost while waiting time and maintenance dominate, and within each currency the end-to-end tier accounts for most of the spend despite holding the fewest tests. Annual cost by currency compute 6,900 waiting 76,600 maintenance 54,200 Share attributable to the end-to-end tier two thirds, from 4% of the tests
Compute is the cheapest currency and the one teams optimise first; the other two are where the money is.

Step 5 — Publish the three numbers per tier, with their assumptions. The assumptions are what make the figures arguable in a productive way; hiding them makes the numbers look authoritative and therefore suspicious.

Verification

Sanity-check the compute figure against the actual bill, since a multiplier error is easy to make and produces a number that is wrong by a factor rather than a percentage.

gh api /repos/:owner/:repo/actions/usage --jq '.billable'
# {"UBUNTU":{"total_ms":41880000,"jobs":1240}}
node -e "console.log('billed minutes last cycle:', 41880000/60000)"
# billed minutes last cycle: 698

Then sanity-check the waiting figure against lived experience by asking two or three engineers how long they wait for a pull request check. If your computed number is wildly different from what people report, the attention fraction or the critical-path assumption is wrong, and it is better to correct it now than to have the figure dismissed in the meeting.

Finally, verify the maintenance attribution by reading ten of the commits it counted. Some will be genuine maintenance, some will be new tests for new features, which is not maintenance at all. A quick manual sample gives you a correction factor and, more usefully, a feel for what the maintenance actually consists of.

Questions the three numbers can settle Adding a browser to the matrix is a compute question, moving a tier out of the pull request path is a waiting-time question, and repairing or deleting a flaky area is a maintenance question. add a browser? a compute question usually affordable unless it is on the gate move a tier off the gate? a waiting-time question compare minutes saved against escaped defects repair or delete? a maintenance question hours spent per year against value protected
Each question has a currency; answering it in the wrong one is how these discussions stall.

Troubleshooting

Symptom: the numbers are dismissed as guesswork. Diagnosis: the assumptions were not stated, so every figure looks like it could be anything. Fix: publish the inputs beside the outputs — runs per week, attention fraction, hourly rate — and invite people to change them. A figure whose assumptions are visible survives scrutiny; one that arrives as a total does not.

Symptom: maintenance looks implausibly low. Diagnosis: the heuristic counts only commits touching nothing but tests, and much test maintenance rides along with a source change. Fix: treat it as a lower bound and say so, or refine it by counting test-file line changes in mixed commits, which is noisier but closer.

Symptom: the compute figure is dominated by one runaway job. Diagnosis: something is retrying, or a matrix expanded without anyone noticing. Fix: that is a finding, not a measurement problem — a job consuming most of the budget is usually the first thing to fix, and it may be a misconfiguration rather than a real cost.

FAQ

What hourly rate should I use for waiting time?

Use a fully-loaded cost your finance team would recognise, and say which one you used. The exact figure matters less than consistency, because the decisions this informs are comparisons — this tier against that one — rather than absolute budgeting. Avoid the temptation to inflate it to make a point; a figure that looks tuned loses the argument.

Is waiting time really lost, given people multitask?

Not entirely, which is why the attention fraction exists. Context switching has a real cost, so a twelve-minute wait is neither twelve minutes lost nor zero. Half is a defensible default; some teams use a third. State which you chose, and if the conclusion flips between a third and a half, the decision was too close to call on cost alone anyway.

How often should this be recomputed?

Once or twice a year, and before any significant decision about the suite’s shape. The inputs move slowly, and recomputing monthly produces noise that invites arguments about the method rather than the conclusion. The duration trend from tracking test duration trends over time is the continuous signal; this is the periodic deep look.

Does a cheaper suite always win?

No, and framing it that way misreads the exercise. Cost is one side of a comparison whose other side is the defects a tier prevents — which is what estimating the cost of an escaped defect puts a number on. A tier that costs a great deal and prevents more is worth keeping; the point is to know which one you have.