A/B testing Approaches and Framework

Simultaneously Launched Tests

In the modern world, where virtually any change on the website/in the app needs to be A/B tested before full-scale implementation, a limitation of one test per site/page type will lead to serious consequences for the business, making it impossible to move forward at the required pace. So our approach is to allow for multiple simultaneously launched tests on the site with certain limitations.

1. We do not allow simultaneously launched tests for the same site element for the same user audience.

2. We strongly recommend limiting the number of tests to two on the same page type and for one audience (excluding product recommendations).

3. Simultaneous tests on a page should not have a noticeable visual and logical impact on each other.

Test Duration

When using the Gravity Field platform to test changes on our clients' websites, we rely on the Bayesian algorithm to evaluate the statistical significance of running campaigns. The Bayesian approach allows us to monitor working tests in real-time and eliminates the so-called "peeking problem."

To determine the winner, we recommend using a minimum test duration of two weeks, even when there is enough data to make a decision before the end of this period. This accounts for in-week fluctuations in site audience behavior (often, users visiting the site on weekends behave differently from those visiting on weekdays). For the same reason, with a minimum test duration, we do not recommend relying on results obtained during holiday periods when user behavior deviates from the standard.

Any significant changes in test settings or the audience for which the test is active will result in the launch of a new test version, in which variations are redistributed to users, and analytical data is collected from scratch.

Test Launch Framework

https://docs.google.com/spreadsheets/d/1Bq-FlHFGy96ID3GfTYjDTucY2Kk6S6mXJ4teh96oX_4/edit#gid=0