Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lifecycle Hooks arguments and support for Steps / DAGs #867

Open
2 of 4 tasks
mshatkhin23 opened this issue Nov 14, 2023 · 1 comment
Open
2 of 4 tasks

Lifecycle Hooks arguments and support for Steps / DAGs #867

mshatkhin23 opened this issue Nov 14, 2023 · 1 comment

Comments

@mshatkhin23
Copy link
Contributor

mshatkhin23 commented Nov 14, 2023

Pre-bug-report checklist

1. This bug can be reproduced using YAML

2. This bug occurs when...

  • running Hera code without submitting to Argo (e.g. when exporting to YAML)
  • running Hera code and submitting to Argo

Bug report

Describe the bug
So this issue describes 2 different "issues" - 1 is more of a shortcoming in Argo, the other seems to be a Hera bug:

  1. hooks do not seem to work on Steps / DAGs; I cant seem to find any examples applying hooks to Steps / DAGs in the upstream tests on hooks so I believe this is an upstream shortcoming. This issue comment motivated a workaround by applying the hook on the step of a Steps wrapper, but I guess thats sub-optimal. Is there any reason hooks shouldn't be allowed on Steps / DAGs?
  2. The second issue does seem to be a bug in Hera -- and has to do with building the arguments for a hook. When I try to supply arguments to the hook for a WT or Step / Task, it seems to build an empty arguments object.

To Reproduce
Full Hera code to reproduce the bug:

@script(constructor="inline")
def echo(message: Annotated[str, Parameter(description="A message to print")]) -> None:
    """Prints out a message."""
    print(message)

with WorkflowTemplate(
        name="mlplatform.test-hooks",
        entrypoint="wrapper-steps",
        hooks={
            "wt-exit": m.LifecycleHook(
                template="echo",
                arguments={"message": "on exit"},
            )
        },
    ) as wt:
        with DAG(name="main-dag", inputs=[Parameter(name="a")]) as main_dag:
            task1 = echo(
                name="echo1",
                arguments={"message": "hello main: {{inputs.parameters.a}}"},
            )
            task2 = echo(name="echo2", arguments={"message": "hello main 2"})
            task1 >> task2

        with Steps(name="wrapper-steps"):
            main_dag(
                hooks={
                    "dag-exit": m.LifecycleHook(
                        template="echo", arguments={"message": "on exit"}
                    )
                },
                arguments={"a": "WORLD"},
            )
            echo(
                name="echo3",
                arguments={"message": "hello world 3"},
                hooks={
                    "echo-exit": m.LifecycleHook(
                        template="echo", arguments={"message": "on exit"}
                    )
                },
            )
        return wt

Generates to:

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: mlplatform.test-hooks-v0.0.1-alpha
spec:
  entrypoint: wrapper-steps
  hooks:
    wt-exit:
      arguments: {}
      template: echo
  templates:
  - dag:
      tasks:
      - arguments:
          parameters:
          - name: message
            value: 'hello main: {{inputs.parameters.a}}'
        name: echo1
        template: echo
      - arguments:
          parameters:
          - name: message
            value: hello main 2
        depends: echo1
        name: echo2
        template: echo
    inputs:
      parameters:
      - name: a
    name: main-dag
  - inputs:
      parameters:
      - description: A message to print
        name: message
    name: echo
    script:
      ... (removing for brevity) ...
  - name: wrapper-steps
    steps:
    - - arguments:
          parameters:
          - name: a
            value: WORLD
        hooks:
          dag-exit:
            arguments: {}
            template: echo
        name: main-dag
        template: main-dag
    - - arguments:
          parameters:
          - name: message
            value: hello world 3
        hooks:
          echo-exit:
            arguments: {}
            template: echo
        name: echo3
        template: echo

Expected behavior

  • The Steps / DAGs issue seems to be an upstream shortcoming / issue so it could be nice to just add an example documenting the work-around.
  • The hook arguments should be correctly built when generating the yaml

Environment

  • Hera Version: 5.9.0
  • Python Version: 3.9.18
@elliotgunton
Copy link
Collaborator

For Hera's lifecycle hooks, they currently require a full model LifecycleHook object, as there is no special Hera class for these yet. See example here. I can take a look into the steps/dags exit hooks, but will be an issue upstream if needed, and will close this one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants