Scargo: Transpile a subset of Python to Argo YAML

[Seanny123]

Argo is amazing software. It makes orchestrating distributed workflows way easier than before. However, I do find it difficult to define workflows in YAML with so little editor support. The scientists I work with at ProteinQure have an even harder time prototyping workflows and testing them locally.

This motivated me to create Scargo, which is a subset of Python which can be transpiled to Argo. This allows a scientist to prototype and test a workflow locally using Python, then transpile it to run with Argo.

ProteinQure has decided to open-source the prototype while it is still actively being developed. However, please consider it less of an established product and more of an ambitious demo.

The best way to understand Scargo is to look at the examples. Note the YAML output may look a bit strange and may not run on your cluster, given our cluster is configured to reduce boilerplate. However, configuring this output to match different cluster configurations should be possible in the future.

Please let me know if you have any comments or questions. Especially if you're trying to run the examples and have a different cluster configuration. I'm unclear on what the cluster config of others looks like.

[Seanny123]

@terrytangyuan this project was influenced by your work with Couler and Argo Dataflows. I plan to keep learning from your work and hope this project has some ideas worth your consideration.

[terrytangyuan]

@alexec has been working on Argo Dataflows so he and anyone else who’ve contributed should take all the credit (not me 😄).

It’s good to see work on Python but I don’t fully understand what the use case is for this project from a quick glance. Couler allows to output YAML files for the defined workflows already. What functions does Scargo bring in addition to that?

[Seanny123]

The main feature is I can prototype/test/run a Python workflow locally before generating the YAML, which I wasn't able to do with Couler. This feature/requirement means all functionality is implemented in vanilla Python functions, which is a different approach than Couler and means the syntax is simpler for scientists to use. However, this simpler syntax does mean the implementation complexity is higher.

[terrytangyuan]

There are actually some placeholders and unfinished work in Couler to run this locally with Docker or pure Python (via different submitters). It would've been better if we merged the efforts.

[alexec]

Can I leave you two to chat about this? I’d love to see more 🐍