Four example variants, each with its lift and 90% range against the original. A range entirely above zero is a win; a range crossing zero means no difference yet; a range below zero is a loss.

--- description: "How Dalton calculates A/B test results: chance to beat, the 90% range, capped revenue, when a winner is called, sample size and early stopping." --- # How Results Are Calculated ::: info The short answer Every result shows a **chance to beat** the original and a **90% range** for the lift. A winner is only named when the planned window ends, once conversion rate or revenue per visitor clears your bar (90% chance to beat by default). Only clear losers can be stopped early. ::: ## What is counted - **A visit:** one browser over 24 hours. Bots are left out. - **Everyone assigned counts,** even if the variant failed to load for them, so a flaky variant can't look better than it is. - **Conversions count from the moment the visitor saw the test.** ## How each result is calculated | | Conversion rate | Revenue per visitor | | --- | --- | --- | | Method | Bayesian, flat prior | Difference in means (normal approximation), over every visitor | | Shows | Chance to beat and a 90% range | Chance to beat and a 90% range | | Notes | | Orders above the 99th percentile of your last 90 days are capped, so one huge order can't decide a test. Counts once each side has 50 orders. | ## Calling a winner

*Illustration: each bar is a variant's 90% range for the lift over the original, the dot its best estimate.* When the window ends, a variant wins if it clears your bar on **conversion rate or revenue per visitor**, unless it's clearly worse on revenue per visitor. If several win, the one with the highest revenue per visitor is named. Between 80% and the bar it reads **Promising**. ### Choosing your bar The bar is **90% by default**. Change it under **Settings → Experiments → When to call a winner** (Settings is in the account menu, bottom left). | Bar (chance to beat) | Same as confidence in Convert or GA4 | If the change does nothing, still called a winner about | | --- | --- | --- | | 80% | 60% | 1 in 5 tests | | 90% (default) | 80% | 1 in 10 tests | | 95% | 90% | 1 in 20 tests | Pick a lower bar for quick, easy-to-undo changes on quiet pages, and 95% for big or costly ones. The **Significant** badge on the lift cards always uses 95%. ## Sample size Each variant needs about **100 conversions**. From the original's conversion rate, Dalton picks the shortest window of **2, 4 or 6 weeks** that gets there. Paused days don't count. ## Stopping early Winners are never called early. Clear losers can be: after at least 7 days, once every variant's range is below the original's, using a method that stays valid however often it's checked (an always-valid confidence sequence). ## What Dalton doesn't do - **Adjust for several variants.** With four or more, read a narrow win with extra care, or raise the bar. - **Variance reduction** such as CUPED. ## Further reading - Kohavi, Tang and Xu, [*Trustworthy Online Controlled Experiments*](https://experimentguide.com/): the standard reference on A/B testing. - Evan Miller, [How Not To Run an A/B Test](https://www.evanmiller.org/how-not-to-run-an-ab-test.html): why stopping on a good day misleads. - Waudby-Smith et al., [Asymptotic confidence sequences](https://arxiv.org/abs/2103.06476): the method behind early loser stopping.