A small A/B testing script that gives the same variant everywhere

An A/B testing tool has two promises to keep. Its script must not hold the page up, and a visitor must see the same variant wherever the decision is made. In myab that can be the browser, a React app, Node, a Cloudflare Worker, or Python or PHP on the server. This is how we keep the script small and the SDKs in step.

Why size matters

Our quickstart asks people to put the script high in the page's <head> and load it synchronously. Experiments then change the page before it paints, so visitors never see the original flash up first. The price is that the browser waits for the script on every page view, whether or not the visitor is in a test.

The script carries the project's running experiments, and assignment is a hash computed in the browser, so there is no "which variant?" request. That leaves the script's own size as the cost we control.

A budget the build enforces

The SDK's build bundles it with esbuild, gzips the result and fails if it is too big.

// Gzipped at maximum compression, so the number doesn't move between zlib versions.
// The site says "the SDK is under 4KB" and "4 to 5KB with your experiments":
// keep this budget so both stay true.
const BUDGET_BYTES = 3.75 * 1024;

const size = gzipSync(readFileSync("dist/sdk.js"), { level: 9 }).length;
console.log(`sdk.js: ${size} B gzipped (budget ${BUDGET_BYTES} B)`);
if (size > BUDGET_BYTES) {
  console.error("SDK is over its size budget");
  process.exit(1);
}

That is 3,840 bytes. CI runs the build on every push and pull request, and the deploy workflow builds the SDK before the CDN Worker that serves it. An SDK over budget fails CI and stops the CDN deploy.

The first budget was 3KB, for an SDK of about 1.2KB. Targeting rules pushed it to 3.5KB. Then we found that Node 23 locally and Node 24 in CI produced gzip output 14 bytes apart at the default level. We now gzip at level 9 so the figure doesn't move between zlib versions, and the budget became 3.75KB.

We also tied the budget to what the website says. When we checked, the SDK was 3.7KB gzipped. A project's script adds a 0.2KB visual editor loader, about 70 bytes per flag or experiment, and about 0.8KB for no-code changes while such a test runs. Our own site's script came to 5.0KB. So the site now says the SDK is under 4KB and a project's script is about 4 to 5KB.

One hash for every SDK

Assignment is FNV-1a over UTF-8 bytes, hashed twice for better avalanche on similar ids, then reduced to one of 10,000 buckets.

export const BUCKETS = 10_000;
const encoder = new TextEncoder();

export function fnv1a32(input: string): number {
  let h = 0x811c9dc5;
  for (const byte of encoder.encode(input)) {
    h ^= byte;
    h = Math.imul(h, 0x01000193);
  }
  return h >>> 0;
}

export function bucketOf(experimentId: string, unitId: string): number {
  return fnv1a32(String(fnv1a32(`${experimentId}:${unitId}`))) % BUCKETS;
}

Weights are added up in order, and a visitor gets the first variant where their bucket is below Math.round(cumulative * 10000). Weights that sum to less than 1 leave the rest unenrolled. The answer depends only on the experiment id, the visitor id and the weights, so every SDK should agree.

The browser SDK, @myab/browser and @myab/react all run this code. @myab/server keeps its own copy of the hash and shares the targeting engine. Python and PHP are rewrites.

Where languages disagree

Languages differ on several of these details.

• Integers. JavaScript numbers are doubles. A 32-bit hash times the FNV prime can pass 2^53, where low bits are lost, so we use Math.imul. Python and PHP multiply in full and mask with & 0xFFFFFFFF.
• Sign. Math.imul returns a signed integer. Without >>> 0, String() could add a minus sign and the second hash would see different bytes.
• Strings. JavaScript strings are UTF-16, so we hash TextEncoder output. Python encodes to UTF-8 first. PHP strings are already bytes.
• Rounding. Math.round rounds halves up and Python's round() rounds them to even, so the Python and PHP SDKs carry JavaScript's rule.
• Attribute values. Targeting compares attributes as strings. JavaScript turns true into "true", Python's str() gives "True" and PHP's cast gives "1". Python and PHP convert the JavaScript way.

def _fnv1a32(data: bytes) -> int:
    h = 0x811C9DC5
    for byte in data:
        h = ((h ^ byte) * 0x01000193) & 0xFFFFFFFF
    return h

Test vectors and a conformance file

Two sets of shared cases catch drift. The first is bucketing-vectors.json in the shared package, with 11 cases giving the expected bucket and variant, including non-ASCII and empty ids. The shared and @myab/server tests both check them. The generator warns that changing the vectors changes every live assignment.

The second is sdks/conformance.json, generated from @myab/server, whose own tests compare 500 visitors against the browser SDK. It holds one project config, eight visitor contexts and 13,120 cases.

• 8,704 assignments, from 64 visitor ids, 8 contexts and 17 keys (11 experiments, 5 flags with allocations and one key that doesn't exist)
• 320 on/off flag checks
• 1,536 flag values
• 2,560 assignments for a project at its monthly limit

The visitor ids include a UUID, visitor-émoji-🙂, 日本語 and the empty string. An SDK that hashed UTF-16 code units or characters instead of UTF-8 bytes would get the non-ASCII ones wrong. The contexts include a boolean matched against the string "true", ages as numbers and as strings, and a fractional score.

A TypeScript test fails if the committed file no longer matches the generator's output. A change to assignment logic therefore forces a regeneration, and the new file fails every SDK that hasn't caught up. CI runs the Python tests on 3.9 and PHP on 8.1, the oldest versions we support. The Go, Ruby, Swift and Android SDKs read the same file. It has grown from 6,820 cases when the Python and PHP SDKs arrived to 13,120 now.

What we'd suggest

• Enforce the size budget in the build script, at a fixed compression level.
• Write down which public claim the budget protects, and check it against the whole file visitors download.
• Choose a hash you can write in a few lines in any language, defined over UTF-8 bytes and unsigned 32-bit integers.
• Generate cross-language cases from one reference implementation, commit them, and test that the committed copy is current.
• Include inputs that hit the known traps, such as emoji, empty strings, booleans and numbers sent as strings.