New dag decorator #1059
Merged
New dag decorator #1059
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
elliotgunton
added
type:enhancement
elliotgunton
force-pushed
the
new-dag-decorator
branch
from
bc78f34
to
5fb5e2f
Compare
elliotgunton
force-pushed
the
new-dag-decorator
branch
2 times, most recently
from
6b521e2
to
c6a00ca
Compare
elliotgunton
force-pushed
the
refactor-runner-io
branch
from
3c5c81a
to
6ab91e3
Compare
elliotgunton
force-pushed
the
new-dag-decorator
branch
from
c6a00ca
to
2a3962c
Compare
samj1912
pushed a commit
that referenced
this pull request
May 8, 2024
* Use a new pair of mixin classes to hold the functions that are compatible with Pydantic V1 and V2, meaning we no longer need to maintain two sets of classes for each Pydantic version * "RunnerIO" classes to be renamed to `Input` and `Output`, keeping the old name as an alias for one release so we can announce the change This PR splits out parts of commits from #1059 that only refactored RunnerIO --------- Signed-off-by: Elliot Gunton <egunton@bloomberg.net>
elliotgunton
force-pushed
the
new-dag-decorator
branch
from
2a3962c
to
fd0ef94
Compare
elliotgunton
commented
May 9, 2024
elliotgunton
force-pushed
the
new-dag-decorator
branch
from
c4bd9d9
to
bab95e3
Compare
|
I want to add an inner-dag example with associated code fix in |
elliotgunton
commented
May 10, 2024
elliotgunton
force-pushed
the
new-dag-decorator
branch
from
3a5e319
to
70adc40
Compare
samj1912
reviewed
May 11, 2024
elliotgunton
force-pushed
the
new-dag-decorator
branch
6 times, most recently
from
f160b78
to
4b4e283
Compare
samj1912
reviewed
May 14, 2024
elliotgunton
force-pushed
the
new-dag-decorator
branch
from
367f499
to
463db8d
Compare
elliotgunton
force-pushed
the
new-dag-decorator
branch
2 times, most recently
from
cd51680
to
6a2ca54
Compare
samj1912
reviewed
May 14, 2024
elliotgunton
force-pushed
the
new-dag-decorator
branch
from
bbadb06
to
63cd710
Compare
* Within the decorator function call we perform a static inspection of inputs and outputs, add a `DAG` object to the Workflow, and finally do a "run" of the DAG function code using a templated input to collect the tasks * Tasks that pass arguments simply take the templated input strings and pass them around * The dependencies between tasks are inferred automatically via __getattr__ calls on the Task object * Add varname as a dependency to be able to get the python variable name to use as the task name (in the YAML) * Make the _add_type_hints function work for *any* pydantic class Signed-off-by: Elliot Gunton <egunton@bloomberg.net>
Signed-off-by: Elliot Gunton <egunton@bloomberg.net>
* Use a global declaring attribute on the _HeraContext * Avoid recursion in __getattribute__ by using object.__getattribute__, and not flipping the declaring value Signed-off-by: Elliot Gunton <egunton@bloomberg.net>
* `__init__` seemingly wasn't being called correctly in the V1 tests, but this is because of the MRO going through the regular `BaseModel` first and failing validation * This changes v1.py to match v2.py Signed-off-by: Elliot Gunton <egunton@bloomberg.net>
* Makes dags callable within dag-decorated functions Signed-off-by: Elliot Gunton <egunton@bloomberg.net>
Signed-off-by: Elliot Gunton <egunton@bloomberg.net>
* Do not return super __init__ * Remove extra function metadata * make init-files to fix __all__ * Fix example missing required param * Fix create_subnode and regenerate examples Signed-off-by: Elliot Gunton <egunton@bloomberg.net>
* Update github cicd Signed-off-by: Elliot Gunton <egunton@bloomberg.net>
* Also add a BaseModel field to example Signed-off-by: Elliot Gunton <egunton@bloomberg.net>
Signed-off-by: Elliot Gunton <egunton@bloomberg.net>
elliotgunton
force-pushed
the
new-dag-decorator
branch
from
63cd710
to
8de9d3b
Compare
* To protect usage of new decorator_syntax * We should roll the multiple flags required (script_annotations, script_pydantic_io and decorator_syntax) into levels, so that script_annotations < script_pydantic_io < decorator_syntax, i.e. you only need to set one flag to also use the experimental feature required below it. Signed-off-by: Elliot Gunton <egunton@bloomberg.net>
elliotgunton
force-pushed
the
new-dag-decorator
branch
2 times, most recently
from
5b01fb8
to
72d6534
Compare
Signed-off-by: Elliot Gunton <egunton@bloomberg.net>
elliotgunton
force-pushed
the
new-dag-decorator
branch
from
72d6534
to
7cfb6b8
Compare
samj1912
approved these changes
May 15, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Pull Request Checklist
Description of PR
Adds the new dag decorator from HEP0001.
inputs and outputs, add a
DAGobject to the Workflow, and finallydo a "run" of the DAG function code using a templated input to collect the tasks
constructmethod to skip validation, and then pass the strings aroundon the Task object
as the task name (in the YAML)