Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add existing Dagster project example to dbt quickstart #21817

Conversation

maximearmstrong
Copy link
Contributor

@maximearmstrong maximearmstrong commented May 13, 2024

Summary & Motivation

This PR updates the quickstart example for dbt + Dagster - it adds the steps to update an existing Dagster project to wrap a dbt project.

This work is the continuation of #21240 and #21816

How I Tested These Changes

BK
+
local

make mdx-full-format
make next-watch-build

Copy link
Contributor Author

maximearmstrong commented May 13, 2024

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @maximearmstrong and the rest of your teammates on Graphite Graphite

@maximearmstrong maximearmstrong marked this pull request as ready for review May 13, 2024 18:49
@maximearmstrong maximearmstrong self-assigned this May 13, 2024
@graphite-app graphite-app bot added the area: docs Related to documentation in general label May 13, 2024
@maximearmstrong maximearmstrong requested review from rexledesma, erinkcochran87 and yuhan and removed request for erinkcochran87 May 13, 2024 18:50
@maximearmstrong maximearmstrong force-pushed the maxime/brand-new-project-in-dbt-quickstart branch from 5ff29d3 to 0ddd004 Compare May 13, 2024 19:07
@maximearmstrong maximearmstrong force-pushed the maxime/existing-project-in-dbt-quickstart branch from 3753f75 to 58b6518 Compare May 13, 2024 19:07

### Option 2: Use an existing Dagster project

You can use an existing Dagster project to run your dbt project. To do so, you'll need to add some objects using the [dagster-dbt library](/\_apidocs/libraries/dagster-dbt) and update your `Definitions` object. This example assumes that your existing Dagster project includes both `assets.py` and `definitions.py` files.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should show the file structure that we expect as a directory tree to be explicit about where we expect files to be.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in ce9d1a1

Comment on lines 113 to 137
assets=[my_dbt_assets],
resources={
"dbt": DbtCliResource(project_dir=my_project),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should have placeholders for their existing assets, jobs, resources, etc.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in ce9d1a1

@@ -58,6 +58,67 @@ dagster-dbt project scaffold --project-name my-dagster-project --dbt-project-dir

This creates a directory called `my-dagster-project/` inside the current directory. The `my-dagster-project/` directory contains a set of files that define a Dagster project.

</TabItem>
<TabItem name="Option 2: Use an existing Dagster project">
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
<TabItem name="Option 2: Use an existing Dagster project">
<TabItem name="Option 2: Load your dbt project in an existing Dagster project">

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in ce9d1a1

@maximearmstrong maximearmstrong force-pushed the maxime/brand-new-project-in-dbt-quickstart branch from 0ddd004 to 6e2bb9f Compare May 14, 2024 15:43
@maximearmstrong maximearmstrong force-pushed the maxime/existing-project-in-dbt-quickstart branch from 58b6518 to 76415f8 Compare May 14, 2024 15:43
@maximearmstrong maximearmstrong force-pushed the maxime/brand-new-project-in-dbt-quickstart branch from 6e2bb9f to b75e81f Compare May 14, 2024 16:07
@maximearmstrong maximearmstrong force-pushed the maxime/existing-project-in-dbt-quickstart branch from 76415f8 to 2878c6f Compare May 14, 2024 16:07
@maximearmstrong maximearmstrong force-pushed the maxime/brand-new-project-in-dbt-quickstart branch from b75e81f to ca9f008 Compare May 14, 2024 16:45
@maximearmstrong maximearmstrong force-pushed the maxime/existing-project-in-dbt-quickstart branch from ce9d1a1 to a5952c4 Compare May 14, 2024 16:46
Base automatically changed from maxime/brand-new-project-in-dbt-quickstart to maxime/ds-169/add-full-copy-pasteable-example-to-dbt-integrate-page May 14, 2024 17:02
Copy link
Member

@rexledesma rexledesma left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See comment on how to format the placeholder

Comment on lines 84 to 97
defs = Definitions(
assets=[
# your other assets here,
my_dbt_assets
],
jobs=[
# your jobs here
],
resources={
# your other resources here,
"dbt": DbtCliResource(project_dir=my_project),
},
)
# end_dbt_definitions_example
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use ...

Suggested change
defs = Definitions(
assets=[
# your other assets here,
my_dbt_assets
],
jobs=[
# your jobs here
],
resources={
# your other resources here,
"dbt": DbtCliResource(project_dir=my_project),
},
)
# end_dbt_definitions_example
defs = Definitions(
...,
assets=[
...,
# Add the dbt assets alongside your other assets
my_dbt_assets
],
resources={
...
# Add the dbt resource alongside your other resources
"dbt": DbtCliResource(project_dir=my_project),
},
)
# end_dbt_definitions_example

@maximearmstrong maximearmstrong force-pushed the maxime/existing-project-in-dbt-quickstart branch from a5952c4 to 0fd3a50 Compare May 14, 2024 17:14
@maximearmstrong maximearmstrong force-pushed the maxime/ds-169/add-full-copy-pasteable-example-to-dbt-integrate-page branch from 927bbc5 to cb5b70d Compare May 14, 2024 18:31
@maximearmstrong maximearmstrong force-pushed the maxime/existing-project-in-dbt-quickstart branch from 0fd3a50 to d86c381 Compare May 14, 2024 18:31
@maximearmstrong maximearmstrong merged commit b79360c into maxime/ds-169/add-full-copy-pasteable-example-to-dbt-integrate-page May 14, 2024
1 of 3 checks passed
@maximearmstrong maximearmstrong deleted the maxime/existing-project-in-dbt-quickstart branch May 14, 2024 19:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area: docs Related to documentation in general
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants