Mocking Feature Flag Providers Deterministically
Feature flags turn one codebase into many: every flag doubles the number of paths through the code it guards, and the provider decides at runtime which path runs. In tests, that decision must be made by the test — explicitly, per case — or the suite’s behaviour depends on whatever the provider returns today, which is how a flag flipped for a marketing experiment breaks a pipeline on an unrelated pull request. This guide covers putting flags behind a port, an in-memory provider that makes every value explicit, testing both sides of each flag, using OpenFeature’s in-memory provider where you have adopted that standard, and catching flags that silently fall back to defaults. It sits under third-party SDK isolation.
Root Cause Analysis
Flags break test determinism in three ways. A test that reaches a real provider inherits its current state, so the same test produces different results as flags are toggled in the dashboard — nondeterminism caused by people in another department. A test that reaches no provider at all falls back to the SDK’s default value, which silently tests one branch and never the other. And a provider SDK that streams updates keeps a connection open, leaving handles behind and occasionally flipping a value mid-test.
The more insidious problem is coverage. A flag guarding a new checkout flow means there are now two checkout flows, and until the flag is removed, both ship. A suite that tests only the default branch has tested half the product. The branch nobody tests is usually the new one — the one being rolled out, and therefore the one most likely to have defects.
Flags also accumulate. Each is intended to be temporary, most outlive their rollout, and a codebase with thirty flags has an astronomical number of theoretical combinations. Testing all of them is impossible and unnecessary; testing each flag’s two branches independently, with every other flag at its production value, is both possible and sufficient for almost every real defect.
Reproducible Setup
Declare every flag in one typed registry, so tests and production agree on names, types and production defaults.
// src/flags/registry.ts
export const FLAGS = {
'new-checkout-flow': { type: 'boolean', productionDefault: false },
'checkout-button-copy': { type: 'string', productionDefault: 'Pay now' },
'max-basket-items': { type: 'number', productionDefault: 50 },
} as const;
export type FlagKey = keyof typeof FLAGS;
export type FlagValue<K extends FlagKey> =
(typeof FLAGS)[K]['type'] extends 'boolean' ? boolean :
(typeof FLAGS)[K]['type'] extends 'number' ? number : string;
export interface Flags {
get<K extends FlagKey>(key: K, context: { userId?: string }): Promise<FlagValue<K>>;
}
// src/adapters/launchdarkly-flags.ts — the only file that imports the vendor SDK
import * as ld from '@launchdarkly/node-server-sdk';
import { FLAGS, type Flags } from '../flags/registry';
export function launchDarklyFlags(sdkKey: string): Flags {
const client = ld.init(sdkKey);
return {
async get(key, context) {
await client.waitForInitialization({ timeout: 5 });
return client.variation(key, { kind: 'user', key: context.userId ?? 'anonymous' }, FLAGS[key].productionDefault) as never;
},
};
}
Implementation
Step 1 — Build an in-memory provider that refuses to guess. Every flag a test reads must have been set explicitly or fall back to the declared production default — never to an arbitrary SDK fallback.
// test/fakes/flags.ts
import { FLAGS, type FlagKey, type FlagValue, type Flags } from '../../src/flags/registry';
export function testFlags(overrides: Partial<{ [K in FlagKey]: FlagValue<K> }> = {}) {
const reads: FlagKey[] = [];
const flags: Flags = {
async get(key) {
reads.push(key);
return (key in overrides ? overrides[key] : FLAGS[key].productionDefault) as never;
},
};
return { flags, reads };
}
Defaulting to the production default, rather than to false or undefined, means an unstated flag behaves as it does for real users today — the right baseline for every test that is not about that flag.
Step 2 — Test both branches of each flag explicitly. Parameterising over the flag’s values makes the doubling of paths visible and cheap.
// src/checkout/render-checkout.test.ts
import { test, expect } from 'vitest';
import { testFlags } from '../../test/fakes/flags';
import { buildCheckout } from './build-checkout';
test.each([
[false, 'legacy'],
[true, 'streamlined'],
])('with new-checkout-flow=%s the checkout uses the %s steps', async (enabled, variant) => {
const { flags } = testFlags({ 'new-checkout-flow': enabled });
const checkout = await buildCheckout({ userId: 'u1' }, { flags });
expect(checkout.variant).toBe(variant);
});
Step 3 — Target individual users where the flag does. Percentage rollouts and user targeting are the provider’s job; your code’s job is to pass the right context, which is worth asserting.
test('evaluates the flag for the signed-in user, not anonymously', async () => {
const seen: Array<{ userId?: string }> = [];
const flags = { get: async (_k: string, ctx: { userId?: string }) => (seen.push(ctx), true) } as any;
await buildCheckout({ userId: 'u42' }, { flags });
expect(seen[0]).toEqual({ userId: 'u42' });
});
Step 4 — Use OpenFeature’s in-memory provider if you have adopted OpenFeature. The standard ships a provider designed for exactly this, so tests configure flags through the same API production code evaluates them with.
import { OpenFeature, InMemoryProvider } from '@openfeature/server-sdk';
beforeEach(async () => {
await OpenFeature.setProviderAndWait(new InMemoryProvider({
'new-checkout-flow': { disabled: false, variants: { on: true, off: false }, defaultVariant: 'on' },
}));
});
afterEach(() => OpenFeature.clearProviders());
Step 5 — Fail on an unknown flag key. A typo in a flag name evaluates to the SDK’s fallback everywhere and silently disables a feature; the typed registry catches it at compile time, and a runtime check catches dynamically-built keys.
export async function getFlag(flags: Flags, key: string, ctx: { userId?: string }) {
if (!(key in FLAGS)) throw new Error(`Unknown feature flag "${key}"`);
return flags.get(key as FlagKey, ctx);
}
Step 6 — Remove the flag and its tests together. When a rollout completes, delete the flag from the registry; the compiler then points at every test and call site still referring to it, so the dead branch cannot linger.
One organisational rule makes all of this sustainable: a flag is not “done” when its rollout reaches a hundred per cent, but when its registry entry is deleted. Tracking that as part of the flag’s original ticket, rather than as a cleanup task for later, is what keeps a codebase from carrying thirty dormant flags — and thirty dormant pairs of tested branches — long after anyone remembers why they exist.
Verification
Confirm no test reaches the real provider. Run the suite without an SDK key and with network requests to the vendor treated as errors.
LAUNCHDARKLY_SDK_KEY= npx vitest run --reporter=dot
# no initialisation errors, no requests to *.launchdarkly.com
Then confirm both branches of every flag are exercised. The recording fake’s reads makes a simple coverage check possible: every flag in the registry should be read with both values somewhere in the suite.
// test/flag-coverage.test.ts — run last
import { FLAGS } from '../src/flags/registry';
import { observedFlagValues } from './fakes/flags';
test('every boolean flag is tested in both states', () => {
for (const [key, def] of Object.entries(FLAGS)) {
if (def.type !== 'boolean') continue;
expect(observedFlagValues(key)).toEqual(new Set([true, false]));
}
});
Troubleshooting
Symptom: a test’s outcome changed without any code change. Diagnosis: it reached the real provider, and someone toggled the flag. Fix: route the test through the in-memory provider, and ensure the composition root never constructs the vendor adapter when running tests.
Symptom: the new branch passes tests but fails in production. Diagnosis: the tests set the flag in one place, but production evaluates it in two — the server and the client — with different contexts. Fix: test both evaluation sites, and assert that both pass the same user context so they agree.
Symptom: the test process will not exit. Diagnosis: the vendor SDK’s streaming connection is open. Fix: never initialise the vendor client in tests; if an integration test must, close it in teardown with the SDK’s close method.
Symptom: the coverage check fails for a flag nobody tests any more. Diagnosis: the flag is fully rolled out and its old branch is dead. Fix: that failure is the intended outcome — remove the flag from the registry and delete the dead branch, rather than adding a test for code that no longer ships.
FAQ
Should end-to-end tests run with production flag values?
Run them with the production values for flags they are not about, which is what defaulting to the production default achieves. For the flag being rolled out, run the critical journey in both states — a cheap addition that covers the branch real users are starting to see.
What about client-side flags in React components?
The same principle applies: a provider component that reads from the flags port, and a test wrapper that supplies explicit values. The render helper from writing custom render helpers with providers is the natural place to accept flag overrides.
Should the in-memory provider throw on unset flags instead of defaulting?
Throwing is stricter and forces every test to state every flag it touches, which quickly becomes noise for flags irrelevant to the test. Defaulting to the declared production value is the better balance — explicit where it matters, realistic everywhere else.
How do flags interact with snapshot or visual tests?
Badly, if left to the provider — a flag flip changes the rendered output and fails every snapshot at once. Pin flags explicitly in visual tests, and keep a separate visual case for each state of a flag that changes the interface.
Related
- Back to Third-Party SDK Isolation
- Isolating environment variables per test — the simplest flag mechanism, and its pitfalls.
- Isolating analytics SDKs from test runs — another SDK that must never reach its vendor in tests.
- Mapping user journeys to test layers — deciding where each flagged branch is tested.