Install python packages without building a container ( at runtime )

[frank-bee]

In our teams, the data engineers have the following need:
They want to be able to run a a workflow with their business logic ( the actual code ) packed into a package.
But during development they do not want to create a docker image which comes with the package (plus third party packages) but run the workflow "immediately".
Does anybody has the same need?
BTW: This idea is a little bit what Airflow does ( e.g. copying the actual workflow code from git or from AWS S3 - during runtime)''

One of the developers from our teams now found a way to achieve this.
It works somehow as follows:

• they have a special Task which runs before all other Tasks
• this task compresses the code with the business logic and saves it to a K8s Volume /PVC
• all other (actual) workflow tasks get some code injected which downloads the business logic and installs it as python package

It works quite good, that's what I have heard fro the team
Basically any modification in the business logic can be tested immediately by creating a workflow and submitting it to the Argo server (with means of Hera)

I'm a DevOps and my goal is find a more clean way to achieve this.
Ideally Hera itself could be enhanced to support this.
Another approach I thought of, would be to use hera "hooks" ( I have seen that somewhere in the codebase) .
With one hook on workspace level , the code could be injected which runs whenever a workflow starts.
Additionally in each Task ( or actually Script) via another hook, the other code is injected responsible for the download of the code and the installation of the package(s).
Does this make sense? I'm not so familiar with python and have to get into it. Would be great if someone could post a small example for the usage of these hooks.

[flaviuvadan]

Hey @frank-bee, thanks for sharing your use case! This sounds very similar to some problems we have at @dynotx actually, so it's very cool to see the parallels

Another approach I thought of, would be to use hera "hooks"

Hera offers this already! Check out the docs on pre-build and post build hooks here! Let me know if that helps

[vikramsg]

The links seem to not be available anymore

[elliotgunton]

New link here https://hera.readthedocs.io/en/stable/walk-through/advanced-hera-features/

[frank-bee]

Meanwhile, I try to convince our devteams with some other idea. Basically to have fast local docker image build (with proper caching). See here #618 (comment) Flaviu Vadan ***@***.***> schrieb am Mo., 5. Juni 2023, 22:14:

…

Hey @frank-bee <https://github.com/frank-bee>, thanks for sharing your use case! This sounds very similar to some problems we have at @dynotx <https://github.com/dynotx> actually, so it's very cool to see the parallels Another approach I thought of, would be to use hera "hooks" Hera offers this already! Check out the docs on pre-build and post build hooks here! <https://hera.readthedocs.io/en/stable/getting-started/walk-through/advanced-hera-features/> Let me know if that helps — Reply to this email directly, view it on GitHub <#665 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACDG74RBCUX2BINYE5LW53XJY43TANCNFSM6AAAAAAYXIKIWY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[flaviuvadan]

That also works, ofc. Did you look at docker buildx (not build)? Is your team on GCP or AWS? On GCP you have things like image streaming that might help your team with faster job start up times as well, in addition to fast container builds

[frank-bee]

Yes, I use docker buildx in my latest approach
https://github.com/aiknow-public/hera-blueprint

Image streaming sounds interesting! didn't know about that yet - but we are on AWS anyway

[flaviuvadan]

Did the Hera hooks provide the necessary support for your use case?

[frank-bee]

As I said, I think with dockerx, proper caching, this is not needed: https://github.com/aiknow-public/hera-blueprint I looked at the hooks: Is it possible to a)launch a pvc at workflow initialization time and b) In each task /Script of the DAG of that workspace infer some code as well? The hook example was rather simple, so I cannot tell if that usecase would be possible. Maybe worth a try.... Viele Grüße, Frank

…

-- Vom Handy gesendet Flaviu Vadan ***@***.***> schrieb am Di., 6. Juni 2023, 14:50:

Did the Hera hooks provide the necessary support for your use case? — Reply to this email directly, view it on GitHub <#665 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACDG7YN32FKA5WUG3U7NHDXJ4RRRANCNFSM6AAAAAAYXIKIWY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[frank-bee]

I think to be able to copy code to a pvc ,a pod is needed. This is the ugly part, which could be only done with an artifical task in the DAG. Or it has to he implement more in the core of hera....

[flaviuvadan]

What if Hera were to provide you with an option to install something in a script pod after it is initialized? 🤔 Perhaps something like

from hera.workflows import script

@script(..., packages=["numpy"])
def foo(...): ...

Hera can inject calls to install the necessary packages within the pod environment at script startup time. I have to think more carefully about this, ofc.. Hera already injects a piece of code that says "add the current directory as a package to import from" 🤔 similarly, it could run a command to install packages

[frank-bee]

Hi @flaviuvadan , I think that would be very interesting.

Some thoughts:
Having an additional attribute in the @script annotation would be one way to implement it.

Package versions are also important.

Another idea could be to read a requirements.txt
Pro: dependencies do not have to be tracked twice ( avoid code duplication)
Con: maybe not all requirements would be required in each script....)

What about a package with own code (non-third party , not shipped as package), a folder basically?
Our Data Engineers have that usecase: They have quite some code, so it does not fit into one function or so. But if they have the code in a different folder ( classes, functions,..), Hera cannot make use if it , if it is not shipped via container / package.
Something like you wrote about above would be nice: "add a directory as a package to import from"