Yaml API: Day Zero tutorial notebook #27284

bzablocki · 2023-06-28T10:29:53Z

I've created the first version of Getting Started with Yaml notebooks.
The target group are users who don't have a lot of experience with coding and/or beam; and want to quickly get started with writing pipelines in beam.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

bzablocki · 2023-06-28T10:33:12Z

R: @robertwb

github-actions · 2023-06-28T10:34:23Z

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

codecov · 2023-06-28T10:46:23Z

Codecov Report

Attention: Patch coverage is 28.57143% with 30 lines in your changes are missing coverage. Please review.

Project coverage is 72.15%. Comparing base (3aa78d2) to head (d4d2b4d).
Report is 3 commits behind head on master.

❗ Current head d4d2b4d differs from pull request most recent head 3da0b06. Consider uploading reports for the commit 3da0b06 to get more accurate results

Files	Patch %	Lines
sdks/python/apache_beam/yaml/yaml_provider.py	28.57%	30 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           master   #27284       +/-   ##
===========================================
+ Coverage   38.53%   72.15%   +33.61%     
===========================================
  Files         698      691        -7     
  Lines      102361   101185     -1176     
===========================================
+ Hits        39447    73009    +33562     
+ Misses      61283    26563    -34720     
+ Partials     1631     1613       -18

Flag	Coverage Δ
go	`53.94% <ø> (-0.40%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

svetakvsundhar

Nice tutorial! and cool idea :)

examples/notebooks/get-started/try-apache-beam-yaml.ipynb

amotley

Overall looks great, thanks for writing this Bartosz!

amotley · 2023-06-28T20:04:33Z

examples/notebooks/get-started/try-apache-beam-yaml.ipynb

@@ -0,0 +1,424 @@
+{


Can we add a landing page (similar to https://beam.apache.org/get-started/try-apache-beam/) which jumps to this collab page?

That would be great! Is this something @robertwb can arrange?

I could be wrong, but I imagine it's just a matter of making a new page similar to that one and adding to the TOC

Tech Writers can probably help answer this question too

Me or @svetakvsundhar will work on this once this PR is merged.

Filed #27450

SGTM, can you use the new YAML tags that Jeff created? https://bugdashboard.corp.google.com/app/tree;dashboardId=551658

examples/notebooks/get-started/try-apache-beam-yaml.ipynb

svetakvsundhar

Thanks! Just add in the TODOs :)

examples/notebooks/get-started/try-apache-beam-yaml.ipynb

amotley

Looks great Bartosz, thanks again

bzablocki · 2023-07-11T09:42:19Z

Hi @robertwb, could you take a look at this PR? Thanks!

examples/notebooks/get-started/try-apache-beam-yaml.ipynb

sdks/python/apache_beam/yaml/yaml_provider.py

…ransform in favour of CombinePerKey with a callable argument

examples/notebooks/get-started/try-apache-beam-yaml.ipynb

Polber

Thanks Bartosz! This looks really good. I left a couple comments, but overall seems to provide a great starting point. I think there will need to be some extra blocks around setup (installing dependencies, beam, etc.), but I see you opened a FR for that.

examples/notebooks/get-started/try-apache-beam-yaml.ipynb

sdks/python/apache_beam/yaml/yaml_provider.py

…orial # Conflicts: # sdks/python/apache_beam/yaml/yaml_provider.py

bzablocki · 2023-12-20T11:39:21Z

Hi @Polber, thanks for the review. All the changes are applied now.
@VeronicaWasson, could you perhaps have another look at this notebook?

examples/notebooks/get-started/try-apache-beam-yaml.ipynb

Co-authored-by: Jeff Kinard <35542536+Polber@users.noreply.github.com>

bzablocki · 2024-01-08T09:23:28Z

Hi @Polber, do you think we can merge it now?

Polber

Hey @bzablocki We are currently trying to get some docs onto the Beam website which I think would be a good idea to get submitted first so that users have docs to reference after trying out the day zero notebook.

While playing around with this notebook again, I noticed a couple more minor things, but no rush.

examples/notebooks/get-started/try-apache-beam-yaml.ipynb

Polber · 2024-02-01T23:11:57Z

examples/notebooks/get-started/try-apache-beam-yaml.ipynb

+    "pipeline = '''\n",
+    "pipeline:\n",
+    "  type: chain\n",
+    "  transforms:\n",
+    "    - type: ReadFromCsv\n",
+    "      config:\n",
+    "        path: data/people.csv\n",
+    "    - type: Filter\n",
+    "      config:\n",
+    "        language: python\n",
+    "        keep: \"age >= 18\"\n",
+    "    - type: LogForTesting\n",
+    "'''\n",
+    "save_to_file(pipeline, 'pipelines/pipeline-filter-01.yaml')\n",
+    "! python -m apache_beam.yaml.main --pipeline_spec_file=pipelines/pipeline-filter-01.yaml"


In which case I would move the run command to its own cell.

examples/notebooks/get-started/try-apache-beam-yaml.ipynb

bzablocki · 2024-02-05T14:58:34Z

Thanks for taking another look at this PR @Polber!

Hey @bzablocki We are currently trying to get some docs onto the Beam website which I think would be a good idea to get submitted first so that users have docs to reference after trying out the day zero notebook.

That makes a lot of sense, I agree it's better to release some docs first. Thanks for the update. I'll wait for the documentation and I'll update this PR with the relevant references once that is done.

Polber · 2024-02-28T19:38:16Z

@bzablocki The docs have been pushed to the beam site, if you want to link them, I think we can get this PR merged
https://beam.apache.org/documentation/sdks/yaml/

kennknowles · 2024-03-04T16:08:03Z

Ping! Seems like this is ready to merge?

bzablocki · 2024-03-04T17:21:09Z

Thanks for the ping, I'll submit an updated PR today.

bzablocki · 2024-03-04T19:07:59Z

I updated the PR with documentation and a short explanation on why a pipeline with a transform from an expansion service (Filter-sql) logs output in a different format than a simple pipeline without an expansion service.

robertwb · 2024-03-15T18:34:02Z

As discussed in person, it feels a bit awkward to have every command be a combination of save file + execute command in Python. Could we set the interpreter to shell instead?

bzablocki · 2024-03-18T22:09:54Z

@Polber I converted saving the file to the built-in Jupyter's '%%writefile'. PR is ready for the final review/merge.

Polber

Thanks for working so hard on this!

bzablocki added 8 commits June 7, 2023 12:20

WiP: Day Zero Tutorial

34b6a0c

WiP: Yaml Day Zero tutorial.

6ac7336

Merge branch 'master' into yaml_day_zero_tutorial

141abb5

Python Notebook for getting started with Yaml API

aa4a0de

Add SumPerKey and TopNLargest transforms to Yaml api

89582f2

Small changes to Getting started with yaml notebook

abc777f

Yaml notebooks: formatting

a39362c

Yaml notebooks: change second pipeline to chain

dee90b2

github-actions bot added examples python labels Jun 28, 2023

svetakvsundhar requested changes Jun 28, 2023

View reviewed changes

amotley reviewed Jun 28, 2023

View reviewed changes

bzablocki added 3 commits June 29, 2023 10:00

Small content fixes in the getting started with yaml notebook

a4a9ced

Adding word count example to the yaml notebook.

a6b5c4e

Clear outputs of the notebook

fd00b1f

svetakvsundhar approved these changes Jun 29, 2023

View reviewed changes

amotley reviewed Jun 29, 2023

View reviewed changes

Apply PR comments. Add more information on structure of yaml pipelines

d13f4ed

amotley approved these changes Jun 30, 2023

View reviewed changes

svetakvsundhar mentioned this pull request Jul 11, 2023

[Feature Request]: Create a "Try Apache Beam YAML" landing page #27450

Open

15 tasks

robertwb reviewed Jul 27, 2023

View reviewed changes

bzablocki added 3 commits July 31, 2023 17:19

Switching fromgsutil cp to wget. Dropping introduced SumPer key t…

bbf7f7d

…ransform in favour of CombinePerKey with a callable argument

Merge branch 'master' into yaml_day_zero_tutorial

00ba6c6

Fixing after the merge

781ee6d

VeronicaWasson reviewed Aug 17, 2023

View reviewed changes

examples/notebooks/get-started/try-apache-beam-yaml.ipynb Outdated Show resolved Hide resolved

VeronicaWasson reviewed Aug 17, 2023

View reviewed changes

examples/notebooks/get-started/try-apache-beam-yaml.ipynb Outdated Show resolved Hide resolved

github-actions bot removed timeseries bigtable labels Dec 19, 2023

Polber suggested changes Dec 19, 2023

View reviewed changes

bzablocki added 2 commits December 20, 2023 11:28

Merge remote-tracking branch 'upstream/master' into yaml_day_zero_tut…

f20bfe6

…orial # Conflicts: # sdks/python/apache_beam/yaml/yaml_provider.py

Review comments

bb760ac

github-actions bot removed python yaml labels Dec 20, 2023

Review comments

3242fcb

Polber suggested changes Dec 20, 2023

View reviewed changes

examples/notebooks/get-started/try-apache-beam-yaml.ipynb Outdated Show resolved Hide resolved

examples/notebooks/get-started/try-apache-beam-yaml.ipynb Outdated Show resolved Hide resolved

bzablocki and others added 2 commits December 21, 2023 13:53

Update examples/notebooks/get-started/try-apache-beam-yaml.ipynb

ada5e62

Co-authored-by: Jeff Kinard <35542536+Polber@users.noreply.github.com>

remove the last code block

ccf2a50

Polber suggested changes Feb 1, 2024

View reviewed changes

bzablocki added 2 commits March 4, 2024 18:09

Merge remote-tracking branch 'origin/master' into yaml_day_zero_tutorial

a38fe29

Add a link to documentation. Add an explanation on expansion service.

fad13cd

typo

239c0d5

bzablocki added 4 commits March 18, 2024 10:47

Merge branch 'master' into yaml_day_zero_tutorial

176c835

Use %%writefile instead of custom python function

ee6da7a

Install apache-beam command without --update

60829dc

Install apache-beam[yaml]

3da0b06

Polber approved these changes Mar 28, 2024

View reviewed changes

robertwb merged commit a475fde into apache:master Apr 5, 2024
3 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Yaml API: Day Zero tutorial notebook #27284

Yaml API: Day Zero tutorial notebook #27284

bzablocki commented Jun 28, 2023

bzablocki commented Jun 28, 2023

github-actions bot commented Jun 28, 2023

codecov bot commented Jun 28, 2023 •

edited

svetakvsundhar left a comment

amotley left a comment

amotley Jun 28, 2023

bzablocki Jun 29, 2023

amotley Jun 29, 2023

amotley Jun 29, 2023

bzablocki Jul 11, 2023

svetakvsundhar Jul 11, 2023

amotley Oct 3, 2023

svetakvsundhar left a comment

amotley left a comment

bzablocki commented Jul 11, 2023

Polber left a comment

bzablocki commented Dec 20, 2023

bzablocki commented Jan 8, 2024

Polber left a comment

Polber Feb 1, 2024

bzablocki commented Feb 5, 2024

Polber commented Feb 28, 2024 •

edited

kennknowles commented Mar 4, 2024

bzablocki commented Mar 4, 2024

bzablocki commented Mar 4, 2024

robertwb commented Mar 15, 2024

bzablocki commented Mar 18, 2024

Polber left a comment

Yaml API: Day Zero tutorial notebook #27284

Yaml API: Day Zero tutorial notebook #27284

Conversation

bzablocki commented Jun 28, 2023

GitHub Actions Tests Status (on master branch)

bzablocki commented Jun 28, 2023

github-actions bot commented Jun 28, 2023

codecov bot commented Jun 28, 2023 • edited

Codecov Report

svetakvsundhar left a comment

Choose a reason for hiding this comment

amotley left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

svetakvsundhar left a comment

Choose a reason for hiding this comment

amotley left a comment

Choose a reason for hiding this comment

bzablocki commented Jul 11, 2023

Polber left a comment

Choose a reason for hiding this comment

bzablocki commented Dec 20, 2023

bzablocki commented Jan 8, 2024

Polber left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bzablocki commented Feb 5, 2024

Polber commented Feb 28, 2024 • edited

kennknowles commented Mar 4, 2024

bzablocki commented Mar 4, 2024

bzablocki commented Mar 4, 2024

robertwb commented Mar 15, 2024

bzablocki commented Mar 18, 2024

Polber left a comment

Choose a reason for hiding this comment

codecov bot commented Jun 28, 2023 •

edited

Polber commented Feb 28, 2024 •

edited