Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] chained docker tasks integration test #326

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

zachmullen
Copy link
Member

No description provided.

@zachmullen zachmullen force-pushed the docker-chain-example branch 3 times, most recently from ab25298 to d5a17f2 Compare November 2, 2018 13:17
@app.task(base=DockerTask, bind=True)
def docker_run_chained(task, results, container_args=None, *args, **kwargs):
container_args = container_args or []
kwargs['container_args'] = [_maybe_transform_chain_result(a, results) for a in container_args]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a little confused. You shouldn't need to transform these results in the actual task. The transformation happens when the task class (girder_worker.task.Task) calls __call__ here

Ideally we'd have a ResultTransform associated with the first docker task that returns something like a GirderFileId transform. That should get transformed as apart of the above code block before being passed to the second docker_run task. Is there something i'm missing that is preventing that approach?


class ChainedResultItem(ChainedResultTransform):
def transform(self, results):
item_id = super(ChainedResultItem, self).transform(results)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh i see... do you have multiple results being returned from the first docker task? What are they?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure how this actually works, but my understanding of girder_result_hooks was that the return value from a docker_run task was always going to be a list, and I thought it was a list whose values are the resolved result hooks. So in the right-hand task in a chain, we need to be able to select which index into that list is the result we want to use.

I have no idea whether this will actually work. If you have an architecture you think is better, that would be great, but you'd need to write it out in code for me to understand it probably. The code inside test_docker_run_multistep_workflow in this PR is where we can show how this looks from an API perspective.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants