hellosoftware

Case study

myab

myab lets a website run A/B tests and feature flags from one small script. Visitors are split in the browser, results are kept as running totals, and the statistics stay valid however often you check them.

A/B testing and feature flags

myab.co.uk
The myab home page on a laptop: the headline “A/B testing, minus the big bill.”, one version of a live myab test, beside an example experiment card comparing two checkout button wordings.
The myab home page on a phone: the headline “A/B testing, minus the big bill.” with Start free and See how it works buttons.

What it does

  • Browser script held to a 3.75 KB (gzipped) budget on every build
  • Sequential and Bayesian results, CUPED and multi-armed bandits
  • Libraries for React, server-side JavaScript, Python, PHP, Go, Ruby, Swift and Android

Every library is tested against the same 13,120 cases, so a visitor gets the same variant and the same flag values whichever language decides it.

The statistics include a sample ratio mismatch check, which catches tests whose traffic split has gone wrong.

How it fits together

  1. People

    • Visitors' browsers running the SDK
    • Teams using the dashboard

  2. Edge

    • CDN Worker serving the SDK and experiments
    • Ingest Worker receiving results

  3. Processing

    • Cloudflare Queue
    • API Worker (Hono)

  4. Data

    • D1 running totals
    • R2 raw events
How myab fits together, from the people using it to where its data is kept. The website and docs are built with Astro, and the dashboard is a React app.

How it's built

  • Results arrive as small beacons, go through a Cloudflare Queue, and are kept as running totals in D1, with raw events in R2.
  • The API is a Hono app on Workers. The dashboard is a React app, and the website and documentation are built with Astro.
  • One TypeScript monorepo (pnpm and Turborepo), with infrastructure in Terragrunt and deployments through GitHub Actions.
  • Simulations check that the false-positive rate holds even when results are checked repeatedly.
  • TypeScript
  • Astro
  • React
  • Hono
  • Cloudflare Workers
  • Queues
  • D1
  • R2

Services involved

Articles about myab

Tell us what you're building.

Email us about your project: what it needs to do, who it's for and when you'd like it live.