Experiments

⚠ THIS IS APPROACH IS DEPRECATED! ⚠

We have A/B testing in the extension to perform staged rollouts of new functionality. If a user belongs to an experiment group, they will see some special features which are visible only to the users of that group. This way we can test a new feature on a subset of users and making sure it's stable, before releasing it to all users. Check out Wikipedia for more details on A/B experiments.

We initially implemented our own A/B testing framework, however it turns out there is a common A/B testing framework used by other teams, for example the Docker extension. Now that we've integrated it (see implementation), we're deprecating our custom implementation.

Common key specs include but are not limited to:

• Experiment groups are logged in the beginning when the extension loads
• If the user is in an experiment, ensure that experiments are used in the first session itself
• If the user opted out of telemetry, then they are also opted out of A/B testing
• Conditional behavior depending on whether an experiment is enabled or not: for example if the user is in the “LS – enabled” experiment and they are using the default configuration (Jedi), then the extension activates with the language server

Our custom implementation also includes logic for:

• If the fetch request for new experiments fails, keep the previous configuration around (check the error being logged)
• Checking if you are in an experiment: In Experiment(foo) =HASH>=min and < max. To verify this, you must know the value of HASH (depends on machineId and experiment salt). See the ExperimentsManager section for how to get that.

Setup

Common framework: ExperimentService

Check for an experiment

• Know the experiment name: it should be the "variable name" of the experiment metadata;
• Retrieve the ExperimentService instance
  • In your class constructor: @inject(IExperimentService) private readonly expService : IExperimentService, or
  • Directly from the service container: const expService = serviceContainer.get<IExperimentService>(IExperimentService);
• Check if a user is in an experiment: await expService.inExperiment(experimentName)

⚠️ This call is currently async, it returns a Promise<boolean>, although there are plans to add a synchronous call later.

Opt into/out from experiments

• Add the experiment an an enum in src/client/common/experiments/groups.ts:

// Experiment to turn on the start page
export enum EnableStartPage {
    experiment = 'EnableStartPage'
}

The enum has a single experiment key because only experiment groups are eligible for opt-in/opt-out, not control ones.

• Add the experiment name to the "python.experiments.optInto" and "python.experiments.optOutFrom" keys in package.json:

"python.experiments.optInto": {
                    "type": "array",
                    "default": [],
                    "items": {
                        "enum": [
                            "EnableStartPage",

Once that's done the experiment name should be available in the completion values for the "python.experiments.optInto" and "python.experiments.optOutFrom" user settings on extension activation.

Custom implementation: ExperimentsManager

• Edit experiments.json to contain,

[
    {
        "name": "LS - control",
        "salt": "LS",
        "min": 0,
        "max": 15
    },
    {
        "name": "LS - enabled",
        "salt": "LS",
        "min": 85,
        "max": 100
    },
    {
        "name": "AlwaysDisplayTestExplorer - enabled",
        "salt": "LS",
        "min": 70,
        "max": 90
    },
    {
        "name": "AlwaysDisplayTestExplorer - control",
        "salt": "LS",
        "min": 40,
        "max": 60
    }
]

• To get HASH, set breakpoint at line 154 in src\client\common\experiments.ts and activate the extension. Once you hit the breakpoint, check the value of hash%100 variable.
• Edit experiments.json file to suit your needs. For eg. to make sure you are in experiment xxx, adjust min and max to make sure your HASH lies between min and max.
• Experiments are downloaded using URL specified in constant configUri (Present in src\client\common\experiments.ts). For validating purposes, edit the constant to contain an invalid URI, which makes sure we always use experiments.json to get exps.