Allow Stats Integration to take bundles of Experiments #2367
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview
This PR rewrites the stats integration to allow for multiple bundles of
DataForStatsEngine
s to be passed, each with an id, to the stats engine, to be returned with results alongside that id. The goal is to eventually allow for easier batch creation of multiple snapshots or snapshot analyses when needed (e.g. for impact analysis).Previously the stats engine could take a
snapshot
and an array ofanalyses
for that snapshot to be computed for each metric.Now, the stats engine takes an an array of
{snapshots, analyses}
so that you can compute several analyses (each set of analyses potentially different from one another) for any number of snapshots (within or across experiments).Details
It creates matching input and output objects in
stats.d.ts
and in python for an object with an ID and the payload (either data in or results out).We create a helper that just does one experiment at a time to keep the current snapshot creation flow largely untouched beyond the actual python we're executing.
I also create
createMultipleSnapshotAnalysis
andpostSnapshotsWithScaledImpactAnalysis
for how this new functionality gets used to kick off multiple jobs at once (e.g. the batch update for the experiment impact analysis).This PR also adds default running scaled and absolute difference types to each stats engine call, since this should add minimal python runtime and will help reduce future burden when loading the dashboard for Impact Analysis.