Jun 13, 2021
My very first attempt on AB Testing with Tableau
A/B Testing is a methodology widely used to determine the effectiveness of an experiment on two groups — A and B. Let us try to implement this methodology in Tableau.
To provide a quick introduction on the topic of A/B Testing, I have cited its definition from CrazyEgg as below:
“A/B testing (also known as split testing) is the process of comparing two versions. You do this giving one version to one group and the other version to another group. Then you can see how each variation performs.” — CrazyEgg, 2020
In theory, the concept of this testing is not very complicated to comprehend. However, how these few words are translated into the language of mathematics and statistics is a totally different story. There are different types of statistical hypothesis testing and for the purpose of demonstrating my attempt on carrying out the AB testing in Tableau, a Student’s T-test is used.
A/B Testing in Tableau
For the purpose of A/B testing, there are plenty of tools designed to measure the effectiveness of a campaign on two groups. And as we all know, Tableau is just a visualisation tool which is incredibly interactive and useful in many ways. Even though, there are a few statistical functions embedded in the software it is not very suitable for statistical analysis such as A/B Testing yet.
It took a lot of time to research the methods to overcome this issue and after a few unsuccessful attempts, I managed to get to find a few ways that could get me closer to my desired results. I thought it would be worth writing a short article about this attempt so other readers can probably discuss and contribute to solving it. All credits to Tableau Community and Tableau Software!
Dataset
The data set I used for this article is called “A/B testing” from Mr.Zhang which can be downloaded from Kaggle — please refer to this link: https://www.kaggle.com/zhangluyuan/ab-testing
The dataset contains information about all users in the control and treatment groups and shows all the records of converted actions that the users have made on the old or new page.
Formulas
In order to determine how the experiment is affecting the control and test group, we need to see whether the difference of converted actions between the two groups is significant or not. To do this we need to calculate the P-Value. In this attempt, I will try to calculate the T-Score and calculate the P-Value from the T-Score.
This is the formula for the T-Score (for a 2-sample t-test):
Where X1 — X2 is the difference between the means of two groups; SE is the standard error of difference
The formula for SE can be seen below:
Where s are the standard deviations and n the sample sizes
First Approach
Since the both groups are in the same dataset, I will separate them and calculate the means and standard deviations accordingly.
Once this is ready, I can start using the formula for the T-Statistic as shown in the picture below:
Once the T-Score is calculated, I am using this formula which was obtain from a thread in the Tableau Community (I forgot to save the link to the thread but will attach it in the Reference section when it’s found).
Please take note, this formula does not take into consideration the degree of freedom so it is not 100 percent accurate. It only gives a very close number to what the answer should be.
That was the first approach that I took. I have included a bar chart with the Average Actions from both of the groups with the P-Value in this dashboard. Here users can also choose their Alpha level to see whether the difference was significant enough.
Second Approach
The first few steps are the same for the second approach apart from the last bit where I needed to convert T-Score to P-Value.
The second approach is more precise but it requires TabPy to connect to the scipy library in Python. I already had Anaconda installed in my machine so all I did was to pip install TabPy. I am sure that there must be a lot of videos with instructions on how to install TabPy.
Once Tabpy is activated using Anaconda Command Prompt, I checked that the connection was successful by entering localhost:9004 in the URL tab. After that I could connect Tableau with Tabpy by following these steps:
For the second approach I will need to create another calculation for degree of freedom. First I need to get the sample sizes of the Control and the Test group. I did this by calculating the distinct numbers of users for each group.
Once the calculation for degree of freedom is created, it is time to use the scipy.stats module to convert T-Score to P-Value which requires only a few lines of code as below:
If we compare two numbers together, we can see that the P-Value that was produced by using the formula is a bit lower than the P-Value produced by the Scipy library.
Using the above formula might be faster and easier to implement but it does not give us the accurate answer. Whilst, using Python alongside with Tableau will give us the right figure.
It was a bit of a clumsy attempt at adding P-Value to a Tableau dashboard but it definitely helped me understand the purpose and the A/B Testing itself better. I intend to have another try on this issue so stay tuned and let me know what you think about these two approaches that I took.
