Outliers

Gravity Field's personalization platform allows the exclusion of outliers from both conversion and monetary metrics.

When it comes to calculating the financial metrics of campaigns, i.e., when personalization campaigns (or A/B tests) are evaluated based on revenue or average order value, simply summing up all the purchases made within a variation doesn't always show the correct result. For example, in a scenario where Variation A has 10,000 users who made 1,300 purchases, and Variation B also has 10,000 users, but they made 1,200 purchases. Based on the number of purchases, Variation A appears to win with a statistical significance of 98.4% (see the illustration).

Now, when we examine the financial metrics, we see that 1,300 purchases in Variation A brought us $54,600 in revenue, while 1,200 purchases in Variation B brought us $60,158. Does this mean Variation B is the better scenario (see the table with original values)? However, when we take a closer look at the actual orders, we will see that in Variation A, all orders are worth around $43 with an average order value (AOV) equal to $42, while in Variation B, there is one order worth $9,800, while AOV for the rest is also $42.

A/B Test Data (original values)

Variations	Users	Purchase	Revenue	AOV
Variation A	10,000	1,300	$54,600.00	$42.00
Variation B	10,000	1,200	$60,158.00	$50.13

It turns out that a single order significantly impacts the financial metrics of the entire campaign. To avoid such situations, serious A/B testing platforms have the capability to exclude excessively large transactions from the test results.

In our platform, when working with outliers, the following algorithm is applied: the values of all orders in the section for the period in question above the 95th percentile and below the 5th percentile in terms of value are replaced with the 95th or 5th percentile values.

How the 95th percentile is calculated: the values of all the orders in the section are sorted from the largest to the smallest, then the top 5% of the list is taken.

In our case, $9,800 will be replaced with $42, and we will get the following data (see the table with calculated values).

A/B Test Data (calculated values)

Variations	Users	Purchase	Revenue	AOV
Variation A	10,000	1,300	$54,600.00	$42.00
Variation B	10,000	1,200	$50,400.00	$42.00

Links

Statistical Significance Calculator (Bayesian) by Dynamic Yield: https://marketing.dynamicyield.com/bayesian-calculator/