Appearance

How Results Are Calculated ​

The short answer

Every result shows a chance to beat the original and a 90% range for the lift. A winner is only named when the planned window ends, once conversion rate or revenue per visitor clears your bar (90% chance to beat by default). Only clear losers can be stopped early.

What is counted ​

  • A visit: one browser over 24 hours. Bots are left out.
  • Everyone assigned counts, even if the variant failed to load for them, so a flaky variant can't look better than it is.
  • Conversions count from the moment the visitor saw the test.

How each result is calculated ​

Conversion rateRevenue per visitor
MethodBayesian, flat priorDifference in means (normal approximation), over every visitor
ShowsChance to beat and a 90% rangeChance to beat and a 90% range
NotesOrders above the 99th percentile of your last 90 days are capped, so one huge order can't decide a test. Counts once each side has 50 orders.

Calling a winner ​

Four example variants, each with its lift and 90% range against the original. A range entirely above zero is a win; a range crossing zero means no difference yet; a range below zero is a loss.Winswhole range above 0 · chance to beat ~99%Promisingmostly above 0 · ~88%No differencerange crosses 0 · ~55%Loseswhole range below 0 · ~2%-20%-10%0%+10%+20%+30%

Illustration: each bar is a variant's 90% range for the lift over the original, the dot its best estimate.

When the window ends, a variant wins if it clears your bar on conversion rate or revenue per visitor, unless it's clearly worse on revenue per visitor. If several win, the one with the highest revenue per visitor is named. Between 80% and the bar it reads Promising.

Choosing your bar ​

The bar is 90% by default. Change it under Settings → Experiments → When to call a winner (Settings is in the account menu, bottom left).

Bar (chance to beat)Same as confidence in Convert or GA4If the change does nothing, still called a winner about
80%60%1 in 5 tests
90% (default)80%1 in 10 tests
95%90%1 in 20 tests

Pick a lower bar for quick, easy-to-undo changes on quiet pages, and 95% for big or costly ones. The Significant badge on the lift cards always uses 95%.

Sample size ​

Each variant needs about 100 conversions. From the original's conversion rate, Dalton picks the shortest window of 2, 4 or 6 weeks that gets there. Paused days don't count.

Stopping early ​

Winners are never called early. Clear losers can be: after at least 7 days, once every variant's range is below the original's, using a method that stays valid however often it's checked (an always-valid confidence sequence).

What Dalton doesn't do ​

  • Adjust for several variants. With four or more, read a narrow win with extra care, or raise the bar.
  • Variance reduction such as CUPED.

Further reading ​