This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Is there a timeline for truly supporting ParallelFor parallelism in Kubeflow v2? We're in the process of adopting Kubeflow and this is a major deal-breaker for us

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

/remove-lifecycle stale

Can someone from the KFP team please comment on this issue? Can users please identify your use case and the priority of this request i.e. is this must have for your use case ?

@jbottum

We use Kubeflow to run massivelly parallel inference pipelines, where we potentially have hundreds / thousands of inputs but we can't feasibly run them all in parallel (due to resource limitations), hence why controlling the parallelism is a must-have for us.

We are successfully doing that with Kubeflow v1, but we feel like we're missing out on more and more on the new functionalities that are offered in v2.

Some users reached out to me about this missing feature in v2 saying this is a blocker to upgrade to that version. I kindly ask for anyone who will comment in this issue to request priority to 👍 the issue description as this is how we rank the most demanding requests from community.

As for the prioritization itself, I'll come up with a plan to the Pipelines WG to speed up the development of this feature. If anyone wants to discuss, join the next Pipelines WG meeting.

I can take this on

/assign @HumairAK

any news about this? it stops be from using Elyra https://elyra.readthedocs.io/en/latest/
I want component A to generate a list of tasks and then parallel execute them with component B

@chanansh Still WIP, but we'll keep updating this issue when we have more information to share.

/assign @gmfrasca

/unassign HumairAK