Approach to PoC Analysis

Created: 10.07.2023

Updated: 10.07.2023

Author: Alexander S.

Within the scope of PoC (Proof of Concept) projects / RAMP period, we regularly encounter questions from clients regarding the methodology utilized for calculating metrics and uplifts from personalization campaigns. This document contains the key principles we adhere to when performing calculations.

We rely on three key approaches when calculating PoC results: traffic extrapolation, time extrapolation, and considering only positive tests.

Traffic Extrapolation

Various campaigns are launched with different traffic volumes. In some cases, the campaign's traffic volume decreases because the hypothesis lacks strong credibility in the eyes of some of the client's departments. In other cases, we need to launch a specified number of campaigns within extremely tight time frame and, as a result, have to divide traffic into splits to run multiple campaigns on one page without overlapping.

To fully assess the effectiveness of personalization, we extrapolate the results to the entire traffic, as during the full commercial operation of the personalization platform, the same campaigns will run sequentially one after another. After the winning variation is declared, it will be used on all of the traffic.

Example: A product recommendations campaign on the product page ran on 50% of the site (as another 50% was used for running the social proof campaign). Within the campaign, traffic was distributed as 80% variation and 20% control group. The campaign generated an additional 50,000 in revenue. The calculation runs as follows: 50,000/0.8/0.5, i.e., we extrapolate the incremental gain to 100% of the site's total traffic.

Time Extrapolation

During the PoC, when time is very limited, we launch personalization campaigns for a limited time period and often turn them off as soon as we obtain results to make room for the next campaign.

When calculating results to adequately compare and measure the actual gains from each of the campaigns, we extrapolate all data to a month, extending the period of the effective version of the campaign to 30 days. During full operation of the platform, there is no need to free up space for additional tests, and campaigns can run for a longer duration, which justifies this approach.

Example: The social proof campaign ran for 14 days and showed a significant revenue increase of 20,000. The calculation mechanism is as follows: 20,000/14*30, i.e., we extrapolate the incremental gain to 30 days.

Considering Only Tests with Positive Gains

The idea behind A/B testing is to verify any hypothesis for improving user experience on real users. The experience of Gravity Field team minimizes the number of hypotheses that do not yield positive results. However, in most cases less than all of the hypotheses produce a positive outcome. In this case, the personalization platform allows multiple client teams to test all hypotheses and reject those that do not work.

During the limited PoC period, we launch various personalization cases and, similar to the full commercial operation period, disable or optimize those that do not yield the desired results. However, we do not have an opportunity to spend several months on optimization before considering the final results. As during the full operation, only campaigns that deliver results remain operational, we do not include campaigns that did not meet the expected results in the outcome.