Skip to content
/ ab Public

A/B Testing Framework for Python (with optional Multi-armed bandit implementation)

License

Notifications You must be signed in to change notification settings

dancrew32/ab

Repository files navigation

A/B Testing for Python

Without the need for a database, deterministically bucket users into A/B tests.

Build Status:

Build Status

Install:

pip install ab

Thanks to Delvian for graciously donating the pip module name at https://pypi.org/project/ab/!

Example A/B Usage:

from ab import ab

# Define test & buckets
TEST_NAME = 'MY_TEST_V1'
TEST_BUCKET_TO_COLOR = {
    'control': 'green',
    'variant1': 'red',
    'variant2': 'blue',
}


# Implemention
def get_button_color(user_id):
    buckets = TEST_BUCKET_TO_COLOR.keys()
    bucket = ab.get_bucket(user_id, test=TEST_NAME, buckets=buckets)
    return TEST_BUCKET_TO_COLOR[bucket]

Thanks to Alexander Ejbekov for the allocation technique:

https://stackoverflow.com/a/23846715/61410

Example Multi-Armed Bandit Usage:

https://en.wikipedia.org/wiki/Multi-armed bandit

from ab import mab

# Define test & buckets
TEST_NAME = 'MY_TEST_V2'
TEST_BUCKET_TO_COLOR = {
    'control': 'green',
    'variant1': 'red',
    'variant2': 'blue',
}


# Implemention
def get_button_color():
    buckets = TEST_BUCKET_TO_COLOR.keys()
    bucket = mab.get_bucket(test=TEST_NAME, buckets=buckets)
    return TEST_BUCKET_TO_COLOR[bucket]


# Record success
def button_clicked(bucket):
    mab.success(test=TEST_NAME, bucket=bucket)

Demo MAB app:

You'll need Docker running and docker-compose.

git clone https://github.com/dancrew32/ab.git ab
cd ab
make up

Then visit http://localhost:5000

Development

Ensure you have python3 and virtualenv installed:

sudo apt install python3.7 python3-venv python3.7-venv

Then make the virtualenv, install any dependencies (there aren't any at the moment), and run the unit tests.

make venv deps test

Publish new version to PyPI

make setup

Releases

No releases published

Packages

No packages published