Put a defensible number on production defects — detection cost, incident time, user impact — and use it to size a test tier, not to assign blame.

Estimating the Cost of an Escaped Defect

A test tier is worth keeping when the defects it prevents cost more than the tier does. The second half of that comparison is usually measured; the first half is usually asserted. This guide builds the missing number from data most teams already have — incident records, support tickets, deployment history — into a figure per escaped defect that is defensible enough to use in a planning conversation. It deliberately avoids precision it cannot support: the aim is an order of magnitude with visible assumptions, not a figure to four significant digits. It is for tech leads sizing a test investment, and it sits under cost-benefit analysis of test layers.

Root Cause Analysis

The cost of a defect rises sharply with the stage at which it is found, and the reason is not mystical — it is that later stages involve more people and more context switching. A defect caught by a unit test costs the author thirty seconds while the code is still in their head. The same defect found in code review costs two people a round trip. Found in production, it costs an alert, an investigation by someone who did not write the code, a fix, a review, a deployment, and whatever the users experienced in the meantime.

The second factor is impact, which varies by orders of magnitude and is what makes averages misleading. A misaligned button and a mispriced order are both defects; only one of them has a financial consequence per minute. Any single average figure will be dominated by whichever type happens to be more common, so the useful estimate is banded by severity rather than averaged.

The third is that the measurement is easily misused. A cost per escaped defect is a tool for sizing investment, and it becomes actively harmful the moment it is used to evaluate individuals — at which point defects stop being reported accurately and the data that made the estimate possible disappears.

Cost of finding the same defect at each stage Cost rises from under a minute when a unit test catches it, through code review and the end-to-end suite, to hours or days once it reaches production and involves alerting, investigation and a hotfix deployment. Cost to resolve, by stage of detection unit test — under a minute component test — a few minutes end-to-end suite — half an hour code review — a round trip production — alert, investigation, hotfix, user impact
The shape of this curve, not its exact values, is what justifies pushing detection earlier.

Reproducible Setup

Pull the raw material: incidents, hotfix deployments, and support tickets tagged as defects.

# hotfix deployments are a decent proxy for escaped defects that mattered
git log --since="12 months ago" --grep="^hotfix" --format='%h %ad %s' --date=short | tee hotfixes.txt | wc -l
# 23
# incident records, if you keep them — duration is the expensive field
gh issue list --label incident --state all --limit 200 \
  --json number,title,createdAt,closedAt,labels > incidents.json
// scripts/escaped/load.ts — one record per escaped defect, banded by severity
export type Escaped = {
  id: string;
  severity: 'critical' | 'major' | 'minor';
  detectionToFixHours: number;
  peopleInvolved: number;
  userMinutesAffected: number;   // users × minutes of degraded service
};

Implementation

Step 1 — Band by severity and refuse to average across bands. Three bands are enough; more produces arguments about classification without improving the estimate.

// scripts/escaped/bands.ts
export const BANDS = {
  critical: { label: 'data loss, money, or an outage', target: 'prevent at any reasonable cost' },
  major:    { label: 'a broken journey with a workaround', target: 'prevent if cheap' },
  minor:    { label: 'cosmetic or rarely reached', target: 'accept' },
};

Step 2 — Compute the engineering cost of each escape. Hours multiplied by people multiplied by a loaded rate, plus the fixed overhead of an unscheduled release.

// scripts/escaped/engineering.ts
const HOURLY = 75;
const RELEASE_OVERHEAD_HOURS = 1.5;    // unscheduled deploy, comms, verification

export function engineeringCost(e: Escaped) {
  return (e.detectionToFixHours * e.peopleInvolved + RELEASE_OVERHEAD_HOURS) * HOURLY;
}

Step 3 — Add the user-facing cost, conservatively and visibly. This is the number most open to challenge, so keep the model simple and the assumption explicit rather than building something elaborate that nobody trusts.

// scripts/escaped/impact.ts
const VALUE_PER_USER_HOUR = 0.4;      // state your basis; ours: revenue per active user hour

export function userCost(e: Escaped) {
  return (e.userMinutesAffected / 60) * VALUE_PER_USER_HOUR;
}

Step 4 — Produce a figure per band and a rate per year. The rate is what makes the comparison against a tier’s annual cost possible.

// scripts/escaped/report.ts
import { engineeringCost, userCost } from './engineering';

const byBand = groupBy(escaped, (e) => e.severity);

for (const [band, items] of Object.entries(byBand)) {
  const total = items.reduce((n, e) => n + engineeringCost(e) + userCost(e), 0);
  const mean = total / items.length;
  console.log(`${band}: ${items.length}/yr, mean ${mean.toFixed(0)}, annual ${total.toFixed(0)}`);
}
// critical: 3/yr, mean 14200, annual 42600
// major: 14/yr, mean 2100, annual 29400
// minor: 61/yr, mean 240, annual 14640
Comparing a tier's cost against the escapes it prevents An end-to-end tier costing about sixty thousand a year is worth keeping if it prevents most critical and major escapes, and is hard to justify if the escapes it catches are mostly minor. Annual figures to compare end-to-end tier cost 60,000 critical escapes prevented 42,600 major escapes prevented 29,400 minor escapes prevented 14,640 the tier pays for itself on the top two bands alone — if it is what catches them
The comparison is only valid for escapes the tier would actually have caught, which is the hard part.

Step 5 — Attribute each escape to the tier that should have caught it. This is the step that turns a cost figure into a decision, and it is a five-minute judgement per incident rather than an analysis.

// scripts/escaped/attribution.ts
export type Attribution = 'unit' | 'component' | 'e2e' | 'not-testable';

// Asked at the incident review: which tier is the cheapest that could
// plausibly have caught this, given how it actually failed?
export const attribute = (e: Escaped, answer: Attribution) => ({ ...e, couldHaveBeenCaughtBy: answer });

Step 6 — Record the attribution at the time, not later. Reconstructing six months of incidents from their write-ups is slow and produces answers coloured by what the team has since decided to believe. A single required field on the incident template — which tier could have caught this — costs the reviewer fifteen seconds and makes the annual estimate a query rather than a project.

Verification

Verify the figure by checking it against something independent. If your estimate says escaped defects cost the team ninety thousand a year, that should be visible as roughly one engineer-month of unplanned work — a claim people can confirm or deny from experience.

node scripts/escaped/report.ts
# total annual cost of escapes: 86,640
node -e "console.log('engineer-months equivalent:', (86640/(75*160)).toFixed(1))"
# engineer-months equivalent: 7.2

Then verify the attribution by sampling. Take five incidents and ask whether a test at the named tier would genuinely have caught them; a surprising number of production defects are not preventable by any test — a third-party outage, an unanticipated input, a capacity limit — and counting those inflates the case for testing in a way that will not survive scrutiny.

Finally, verify that the estimate changes a decision. If the numbers come out and nothing is done differently, the exercise was expensive trivia. The useful outcomes are concrete: move a tier onto the merge gate, delete one that catches only minor escapes, or invest in the tier that would have caught the critical ones.

Attributing escapes honestly, including the ones no test would catch Of twenty escapes, some were preventable by a unit test, some needed an integration or end-to-end test, and a substantial share were not preventable by any test at all — counting that last group inflates the case for testing. Twenty escapes, attributed 6 — unit logic errors in pure functions 4 — component state and rendering interactions 3 — end-to-end wiring across the whole system 7 — not testable third-party outages, capacity, real-world input
Counting the untestable third as preventable is the most common way these estimates lose credibility.

Troubleshooting

Symptom: the estimate is dismissed as invented. Diagnosis: the user-impact model is doing too much work and cannot be defended. Fix: publish the engineering cost separately, which is straightforwardly countable, and present user impact as a range with its assumption stated. A defensible smaller number beats an impressive one nobody believes.

Symptom: nobody can agree which tier should have caught an incident. Diagnosis: the question is being asked abstractly rather than about the actual failure. Fix: ask it at the incident review while the mechanism is fresh, and phrase it concretely — “what is the cheapest test that would have failed on this change?” — which usually produces immediate agreement.

Symptom: the numbers are used to blame a team. Diagnosis: the framing shifted from investment sizing to performance measurement. Fix: stop publishing per-team attribution and report only in aggregate. The estimate depends on honest incident reporting, and the fastest way to destroy that is to make reporting costly for the reporter.

FAQ

Is it worth estimating if the data is poor?

Yes, provided you are honest about the precision. Even a rough figure changes conversations from “testing feels expensive” to “this tier costs sixty thousand and prevents about seventy” — which is a question people can reason about. Poor data argues for wide bands and stated assumptions, not for skipping the exercise.

Should near-misses count?

They are useful as a leading indicator but should not be priced into the escape figure, because their cost was not actually incurred. Track them separately: a rising number of defects caught in staging suggests earlier tiers are weakening, which is worth knowing before the escapes follow.

How does this relate to coverage targets?

Only loosely, and it is a mistake to derive one from the other. Coverage measures execution, not prevention, so a tier can have high coverage and prevent few escapes. The tier’s contribution is better judged by attribution — which escapes it caught before release — and by detection quality, for which mutation testing and assertion quality is the sharper tool.

What if we have almost no escaped defects?

Then the honest conclusion may be that you are over-invested in testing, and the next question is which tier can be reduced without the rate rising. That is an uncomfortable finding, and it is exactly the kind of thing the estimate exists to surface; a suite sized for risks that are not materialising costs the same as one sized correctly.