Request for input: Config-based pipelines

[sryza]

Motivation

YAML. A very polarizing topic in the data community. Many hate it. Many more use it.

Many data organizations define parts of their data pipelines in configuration languages like YAML. Configuration languages like YAML are particularly useful for:

• Users who aren’t familiar with Python, e.g. analysts or frontend engineers who sometimes need to contribute to data pipelines but don’t have it as a primary responsibility.
• Tasks that can & should be heavily standardized within an organization, e.g. moving a file of a certain type from one storage system to another, or refreshing a Databricks notebook.

We've observed that many organizations build layers on top of Dagster that consume YAML. We’re trying to determine whether – and how – to make these uses more first class in Dagster, and to help users avoid reinventing the wheel every time they want to implement them.

What this could look like

What might this look like on top of Dagster? Some basic requirements:

• Customizable - data platform engineers need to be able to determine what fields their stakeholders can set and what level of abstraction to expose them to.
• Escapable - a YAML layer can’t be as expressive as a Turing-complete language, so users need to be able to punch through to Python when they have advanced needs.
• Approachable - to be useful, needs to provide an out of the box experience that allows a user with basic needs to get the job done without complex Python scaffolding.

A high-level sketch:

• A new dagster-yaml library includes a load_defs_from_yaml function, which reads a directory of YAML files and uses their contents to generate a Dagster Definitions object.
• load_defs_from_yaml accepts a list of plugins, which specify YAML interfaces for building Dagster definitions.
• Some Dagster integration libraries, like dagster-dbt and dagster-shell, expose built-in plugins. Users can also define their own plugins.

Example

definitions.py

from dagster_yaml import load_defs_from_yaml
from dagster_dbt import dbt_yaml_plugin
from dagster_shell import shell_yaml_plugin

defs = load_defs_from_yaml(base_dir=".", plugins=[dbt_yaml_plugin, shell_yaml_plugin])

definitions.py (alternative with both YAML defs and Python defs)

from dagster import Definitions
from dagster_yaml import load_defs_from_yaml
from dagster_dbt import dbt_yaml_plugin
from dagster_shell import shell_yaml_plugin

defs_from_yaml = load_defs_from_yaml(base_dir=".", plugins=[dbt_yaml_plugin, shell_yaml_plugin])
defs_from_python = Definitions(...)
defs = defs_from_yaml + defs_from_python

my_analytics_defs.yaml:

defs:
  - type: dbt
    name: cloud_analytics_dbt_assets
    dbt_project_dir: "analytics_dbt/"
    exclude: "+tag:insights,config.materialized:incremental"

my_recommender_model_defs.yaml:

assets:
  - type: shell
    key: recommender_model
    shell_command: "./train_recommender_model.sh"
  - type: python
    key: recommender_model_eval
    module_path: my_org.evaluate_recommender_model

Let us know what you think

As usual, we would love your feedback. Do you write your own definition factories using YAML? What kinds of tasks do you use it for? What kinds of difficulties do you face? What functionality would you find useful?

[zyd14]

Yeah definitely would be useful. That could really help enable folks who aren't as confident with Python programming to use Dagster. I love Dagster, but it is certainly not super accessible to folks who are new to python and just want to string some basic tasks together that are already defined externally.

[astronautas]

How about defining assets as reusable Python components? e.g. make a factory for an asset or smth? So, you have high enough level configuration, but also extendability, if needed, also types.

[zyd14]

Folks are already doing this. I see this proposal as a way to string together pipelines for less technical folks, and/or when building blocks / pipeline steps are already well defined and putting them together using YAML would be less overhead than more code. Also easier to review with non-technical stakeholders as @dduong1603 mentioned.

[astronautas]

Yeah that's fair! For less technical people, a solid option.

[dduong1603]

YAML is also useful for reviewing the pipeline with a non-technical stakeholder. Some industries (e.g. Financial Services) actually require us data folks to walk through the whole pipeline end-to-end with our non-technical counterparts.

But the part I'm actually excited about is the combination of multiple defs (YAML + Python, or multiple YAMLs, or multiple Pythons) 😄 (Edited: I see some of us already asked for combining multiple defs - #21092)

[PedramNavid]

insert yodawg.jpg yaml in your python so you can python your yaml while you yaml your python

[Daniel-Vetter-Coverwhale]

My two cents: YAML is terrible and pushing it as a viable option makes me sad.

If you haven't seen this document, enjoy - https://ruudvanasseldonk.com/2023/01/11/the-yaml-document-from-hell.

For the dbt example this doesn't seem too crazy, because the DBT stuff is already not really in python, so it's just some more yaml to expose yet more yaml that's already done. I guess also for wiring up other non-python assets it could make pipes easier. But then how does dagster know what the inputs and outputs look like? How can it update the catalog? I suppose a lot of that happens inside of pipes, but I'm not sure. Can you reference the assets in YAML from python code and vice versa. I'm pretty sure the answer is yes, by string keys, but you're likely not going to get any assistance from your IDE or your type checker.

Unless dagster is also going to start providing an editor inside of the webserver that uses instance level knowledge rather than source code level knowledge to assist, but I feel like that's potentially playing a dangerous game and could further divide the open source and cloud offerings.

But for general assets this just seems sad. I probably sound like a broken record, but one of the best things about dagster is how it pushes users into better practices. We've already lost some of that in my opinion with pipes and the push away from IO managers I think (though I realize IO managers are often a burden, I think they are a nice concept to default to), but this is even further.

I also feel like if this starts up, you'll shortly be getting requests to add functionality to the YAML, as that seems to always happen. Cloudformation is a particularly egregious example of this.

[nils-borrmann-tacto]

I'm actually doing exactly this, yet I am against this feature (or at best indifferent).

Why? I am doing this because I want to control the API I expose to my stakeholders. A generic API will never be able to give me this kind of control, it will always expose either too little or too less configuration. I doubt it will be possible to hit anyone's sweetspot here. On the contrary, writing a few factory methods is really not that much effort.

[sryza]

@nils-borrmann-tacto - I'm curious about your thoughts on the following very early prototype. The idea is for the yaml format to be fully customizable, with the schema represented using Pydantic models. We will provide some built-in models (DefinitionSetConfigs – name very pending), but users can pick and choose which they make available, as well as write their own.

• Core abstraction – DefinitionSetConfig: https://github.com/dagster-io/dagster/pull/21821/files
• Example with one definition per file: https://github.com/dagster-io/dagster/pull/21822/files
• Example with multiple definitions in one file: https://github.com/dagster-io/dagster/pull/21823/files