Skip to content

πŸ§ͺ Source-controlled split testing stack for building, launching and analysing A/B tests.

License

Notifications You must be signed in to change notification settings

mint-metrics/mojito

Folders and files

NameName
Last commit message
Last commit date

Latest commit

Β 

History

9 Commits
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

Mojito experimentation framework

Mojito

A modular, source-controlled split testing framework that lets you build, launch and analyse experiments via Git/CI.

View documentation | Read intro blog post

It's comprised of 3 core modules e.g.:

  1. Mojito JS Delivery: Front-end library for running experiments on your site.
  2. Mojito Snowplow Storage: Data models & events for tracking experiments.
  3. Mojito R Analytics: Templatable RMarkdown experiment reports.

Mojito's 3 components

Features

  • Under 5kb minified & gzipped
  • Define experiments with simple JS or YAML
  • Self-hosted & git-controlled for familiar code review / merging
  • Expressive trigger system & utilities
  • Variant code (JS/CSS) minification & linting
  • Track and handle JS errors caused by your variant code

Mojito vs. [vendor]

Differentiating features between popular vendors' tools and Mojito out of the box:

Feature Optimizely X Google Optimize Mojito
Open-source license ❌ ❌ βœ… BSD3
Light front-end codebase * ❌~80kb ❗~25kb βœ…<5kb
Git source control & CI ❌ ❌ βœ…
Variant error-tracking/handling ❌ ❌ βœ…
Auto CSS/JS minification ❗(not custom code) ❓ βœ…
Self-hosted ❗ (for a fee) ❗(via API) βœ…
Data ownership ❗(via S3 export) ❗(via 360/BigQuery) βœ…
Retroactively add new metrics βœ… ❗(360 only) βœ…
Server-side/App testing βœ… ❗(via API) ❗(via Storage)
WYSIWYG test editor βœ… βœ… ❌

* Tested 2019-07-05

Getting started

Mojito consists of three components, which are often switched out in the course of Mint Metrics' client services:

  1. Delivery: Front-end libraries to reliably control which treatments users are exposed to. e.g. Mojito JS Delivery
  2. Storage: Data collection modules and data modelling steps to power your reports. e.g. Mojito Snowplow Storage
  3. Analytics: Tools to measure & report on the effects caused by your treatments. e.g. Mojito R Analytics

Get up and running quickly with the README files inside each section.

Example experiment

Using Mojito's CI tools, you can set up experiments in YAML & JS:

id: ex1
name: Example test 1
state: live
sampleRate: 0.75
trigger: trigger.js
recipes:
  0:
    name: Original
  1:
    name: Variant
    js: variant.js
    css: variant.css

Where trigger.js activates the experiment when a condition is met and a callback to activate is fired:

function trigger(test) {
    if (document.location.pathname === '/') test.activate();
}

Upon activation, the will include 75% of traffic (sampleRate: 0.75) and split it 50-50 between "Original" and "Variant" groups.

For users assigned to the "Variant" group, we execute a) variant.js and b) variant.css files to transform the page through a a) JS function and b) CSS stylesheet respectively.

After you've defined an experiment YAML...

Run the Gulp pipeline to lint/test/publish your container.

  1. Install the necessary NPM packages: npm install
  2. Build & publish your testing container: gulp scripts-local && gulp publish

Example analytics reports

If you use our Snowplow/Redshift & R Analytics component for reporting, all your metrics can be reported on with a simple array of metrics.

wave_params <- list(
  client_id = "mintmetrics",
  wave_id = "ex1",
  start_date = "2019-05-15 09:19:45",
  stop_date = "2019-06-05 14:29:00",
  time_grain = "hours",
  subject = "usercookie",
  recipes = c("Original", "Variant")
)

goalList <- list(
  list(
    title = "Transactions",
    goal = "purchase",
    operand = "="
  ),
  list(
    title = "Thankyou page views",
    goal = "page_view /contact/thank-you%",
    operand = "like"
  )
)
goalList <- mojitoFullKnit(wave_params, goal_list = goalList)

For this experiment, we'll report on transactions and page views:

Measuring the performance of a treatment relative to the control group in Mojito.

Support for other analytics back-ends

You don't exactly need Snowplow Analytics to use Mojito. You can also track experiments to wherever you like, via a custom storage adapter. E.g. To Google Tag Manager, Adobe etc.

You can even hook Mojito Delivery up to Google Optimize's reports for free.

Credits

Our Delivery JS library is a heavily modified fork of the excellent jamesyu/cohorts lib. Meanwhile we employ heavy use of the Snowplow Analytics event pipeline for our Storage component and RStudio/Knitr for our Analytics reports.

Getting involved

We would love to see PRs! We're able to assist if you hit any snags getting set up.

Reach out to us via:

Learn more

Read the documentation and get Mojito set up. We recommend starting with Mojito JS Delivery.

Releases

No releases published

Packages

No packages published