# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python [conda root]
#     language: python
#     name: conda-root-py
# ---

# # AB Test
# This notebook outlines the most basic version of a multivariate test (or AB test), and the ways that the `ab_test` module can be used to calculate useful descriptive statistics about a test's performance. We'll start by importing the source code from [src](src).

from src import *

# Imagine we have a product with a feature whose performance we think we can improve. We have an idea for how the feature be changed to imrpve performance, but no way of knowing whether it would perform better than the original version. This is where AB testing comes in.
#
# We take our 4,000 person strong user base and randomly split it into two sections. Each user is either shown the  new (test) version or the original (control) version of the feature, based on the group of users they were grouped with. The test and control versions will be running side-by-side, but each user only sees one version.  
# We'll show the new test version to just 10% of our audience - this keeps the risk low. If the new version performs badly, only a small proportion of our audience will have had a worse experience. The vast majority of users will just see the same old vanilla feature.
#
# We keep track of the number of conversions in each of the groups, ie the number of times that the users in each group enact a certain behaviour which we're trying to drive - that might be open, click through, purchase...
#
# The original version sees 480 conversions, while the new test version sees 53. 

control = [3600, 480]
test = [400, 55]

# Using the `ab_test` module in the [src](src) folder, we can calculate the `conversion_rate()` for each version of the product, where conversion rate is the ratio of number of users who had the _opportunity_ to enact our chosen behaviour, divided by the number of users who _did_ enact that behaviour.

conversion_rate(control), conversion_rate(test)

# Looks like the test version did very slightly better than our original - lets test that

conversion_rate(test) > conversion_rate(control)

# We can calculate the _uplift_ of the test version compared to the original - the difference between the performance of the two. A positive value for uplift means that our test version is better than the original.

conversion_uplift(control, test)

# However, it's hard to be _sure_ that our test version is actually better than the original. A sample size of 400 isn't that large - can we be sure that the feature's performance will be universally better when we open it up to our whole user base?
#
# We need to calculate the _standard error_, or _uncertainty_ of each measurement:

conversion_rate_uncertainty(control), conversion_rate_uncertainty(test)

# Lets set these rates and uncertaities as variables which we can use as we move on 

# +
conversion_rate_control = conversion_rate(control)
conversion_rate_uncertainty_control = conversion_rate_uncertainty(control)

conversion_rate_test = conversion_rate(test)
conversion_rate_uncertainty_test = conversion_rate_uncertainty(test)
# -

# Next we need to calculate a z-score. This tells us the number of standard deviations away from the mean our value is

z = z_score(conversion_rate_test, conversion_rate_uncertainty_test, 
            conversion_rate_control, conversion_rate_uncertainty_control)
z

# The p-value is what really counts in a hypothesis test, and is defined as follows:
#
# >"The P value is defined as the probability, under the assumption of no effect or no difference (the null hypothesis), of obtaining a result equal to or more extreme than what was actually observed."
#
# In other words, it's an indicator of how confident we can be about rejecting the null hypothesis, or the assumption that our test version has no effect. A smaller p-value indicates that the tester can have greater confidence in that rejection.  
#
# **Note:** It's easy to misinterpret p-values as being more powerful than they really are. [Statistics Done Wrong](https://www.statisticsdonewrong.com/) is an excellent free resource on this topic
#
# Testers usually set a threshold p-value of 0.05 or 0.01. Lets see how our test has done:

p_value = p_value(z)
p_value

# Ouch. Not very significant at all. For us to declare that our test version is worth pursuing, we would want to see a positive uplift (indicating that the test performed better than the control) and a p-value _lower_ than our set threshold (indicating that our assertion of betterness is statistically significant).
#
# # Wrapping up
# We can bundle all of the above into a nice clean function called `ab_test()` which takes as arguments the performance of the control and the test version(s) of the product. It then prints out a nicely formatted version of the statistical values above and a statement about how the test should be interpreted.

ab_test([3600, 480], [400, 55])

# We can expand our test to be _multivariate_ - if the above example with two versions is an AB test, a multivariate test includes C, D, E etc. We bring in more than one test variation at a time and compare their performance all at once. The same code applies here. We just tack on our extra versions at the end

ab_test([2000, 50], [2000, 51], [2000, 20], [2000, 80])

# We can also adjust the p-value of a test to assess its significance to a greater or lesser degree.

ab_test([2000, 50], [2000, 51], [2000, 20], [2000, 80], p=0.01)

StarcoderdataJupyterScriptsDedupFiltered