🗺️ Datasets and Experiments

[mikeldking]

As a user, I'd like to have the notion of a dataset of records over which I can run an application or a set of evals. Common dataset purposes are:

• Golden Dataset - contains queries and golden responses
• QA Asset - a set of difficult queries over which you want to perform regression testing
• Sample of production data - pick and choose records from production to convert to an asset (overlaps with above)

Motivation

LLM outputs are non-deterministic and teams need a proper way to evaluate the system. With datasets, teams can select a “test suite” of data points that they can evaluate changes on. This allows them to have trust in their application when they make modifications such as:

• Modifying a prompt template
• Iterate on various components of application and seeing if there are differences in output
• Swapping out for new model releases
• See performance on a cheaper new model or a fine-tuned model

Use-cases

Datasets will contain data from various data sources:

Pre-deployment

• Store and maintain a set of synthetic queries
• Store and maintain a set of hand curated queries
• Make a copy of huggingface / CSV data to be maintained internally

Post-deployment

• Move data from production or staging for regression testing or fine-tuning

Architecture

Dataset

A dataset maintains a set of records. These records are versioned such that if a record is modified (added/edited/deleted) these changes are tracked and versioned. These versions must be immutable such that if there is code that depends on a version, the data does not change.

Dataset Examples

A dataset is a set of examples. These examples contain:

• input - data passed to an LLM, prompt, or function (e.x. a retriever)
• expected / output the result of the invocation of an LLM, prompt, or function
• metadata - any additional information that can be used during experimentation

In addition to the above a dataset record should optionally have

• metadata any additional data associated with the record (e.x. attributes from a span)
• **source span_rowid / trace_rowid ** if the data came from a span, it should link back to the source

Dataset Experiment

A dataset experiment that is run using the examples of a dataset. Experiments are tied to a specific dataset version and have a duration of time. During an experiment, certain parts of an LLM application's components is being modified. This includes:

• Change in LLM or LLM params
• Change in prompt template
• Change in retrieval strategy

Experiment Output

Planning

• [datasets][planning] ERD for datasets, dataset records, audit tables #3043

Infra

• Make Phoenix REST API standardized and to use OPENAPI with exposed documentation #2792
• create an arize-phoenix-client package #2914
• [datasets] database seeding migration of dataset tables #3159

Tables

• [datasets] add indexes to dataset_example_revisions table #3241
• [experiments] experiment tables in datasets migration #3357

Rest API

• [datasets] POST /datasets/upload (CSV) #3045
• [datasets] POST /datasets #3157
• [datasets] PATCH / datasets/:dataset_id #3156
• [datasets] POST /v1/datasets/upload #3158
• [datasets] GET /datasets/:id #3044
• [datasets] DELETE /datasets/:dataset_id #3155
• [datasets] DOWNLOAD /datasets/:dataset_id/openai_ft #3154
• [datasets] GET /datasets/:dataset_id/download/openai_evals #3180
• [datasets][REST] /download/csv #3214
• [datasets] GET /v1/datasets/:dataset_id/versions/ #3215
• [datasets] GET /v1/datasets/:dataset_id/versions/:version_id/examples #3216
• [datasets][REST] example listing endpoint #3227
• [datasets][REST] version parameter on example listing #3228
• [datasets] csv downloads should contain example_id as the index #3311
• [datasets] csv download doesn't prefix input/output/metadata #3310
• [datasets][rest] return dataset ID on CSV upload #3341
• [experiments] POST /datasets/:dataset_id/experiments with a specific version #3355
• ... other rest endpoints @anticorrelator

GraphQL

• [gql] migrate dataset to inferences #3171
• [datasets][gql] migrate connections and node interfaces to strawberry relay #3160
• [datasets][gql] Dataset create #3161
• [datasets][gql] Dataset delete #3162
• [datasets][gql] Dataset Node #3163
• [datasets][gql] Dataset connection #3164
• [datasets][gql] Examples connection #3182
• [datasets][gql] Example node #3177
• [datasets][gql] Mutation createDataset #3196
• [datasets][gql] Mutation createExamples #3179
• [datasets][gql] Mutation patchDatasetExamples #3340
• [datasets][gql] patchDataset #3181
• [datasets][gql] addSpansToDataset #3178
• [datasets][gql] dataset versions connection #3201
• [datasets][gql] mutation upload CSV
• [datasets][gql] refactor mutations #3218
• [datasets][gql] refactor create dataset input and return types #3219
• [datasets][gql] support clientMutationId #3220
• [datasets][gql] standardize error types #3221
• [datasets][gql] search param on datasets for typeaheads etc #3229
• [datasets][gql] add example to dataset #3233
• [datasets][gql] add the ability to sort datasets by name, date created #3252
• [datasets][gql] support explicit version ID passed to the examples connection #3339
• [datasets][gql] add revision resolver to example type #3276
• [datasets][gql] span as example #3278
• [datasets][gql] delete examples mutation #3301
• [datasets] example span optional resolver #3303
• [datasets] span project resolver #3304
• [dataset] [gql] delete dataset #3305
• [dataset] [gql] edit dataset (name, description, metadata) #3306
• [dataset] [gql] add node interface support for dataset example #3326
• [datasets][gql] add revisions resolver to DatasetExample #3331

Experiments SDK

• [experiments] Evaluator Interface / Protocol #3356

UI

• [datasets][gql] Datasets Page #3165
• [datasets] dataset create UI #3209
• [datasets][gql] Dataset Page #3166
• [datasets][ui] Multi-Select on spans / traces table #3202
• [datasets] [ui] dataset examples table #3200
• [datasets] [ui] add span to dataset from trace page #3208
• [datasets] [ui] CSV upload creation of a dataset #3353
• [datasets] [ui] download dataset controls
• [datasets] [ui] view example slideover #3302
• [datasets] [ui] edit example UI
• [datasets] [ui] multi-select on examples and delete
• [datasets] [ui] add the ability to create a new dataset before adding a span
• [datasets] [ui] show latest version in header
• [datasets] [ui] copy to clipboard for dataset id, version id
• [datasets] [ui] delete dataset #3335
• [datasets] [ui] download dataset as CSV button #3307
• [datasets][ui] edit dataset example form
• [datasets][ui] link to span from example details
• [datasets][ui] delete dataset examples UI #3349

Tests

• [datasets] tests for addExamplesToDataset #3275
• [datasets][testing] internal dogfooding round 1 #3333
• [datasets] update patch dataset example test case to check for patched versions #3359

Bugs

• [datasets][bug] dataset examples gql endpoint pulling from all revisions #3300
• [datasets][ui] alert for dataset example created gets hidden under slideovers
• [datasets] don't throw errors on CSV download (just return en empty csv) #3309
• [ENHANCEMENT] use consistent download URLs for dataset downloads #3308
• [datasets][gql] ensure dataset examples query pulls only from dataset id #3332
• [datasets][bug] fix dataset fixtures #3337
• [datasets] [bug] messages in input output is string #3342
• [datasets] make /upload endpoint return dataset in data payload #3363

[mikeldking]

Note that if a span does not meet certain criteria (like embeddings) it might make sense to avoid allowing it to be added to a dataset

[axiomofjoy]

Note that if a span does not meet certain criteria (like embeddings) it might make sense to avoid allowing it to be added to a dataset

What other criteria can we think of?