I can spawn inner instances for a large input collection

[saig0]

Is your feature request related to a problem? Please describe.
At a multi-instance activity, I can define the input collection to iterate over. If the collection holds a large number of elements, the broker might fail to spawn the inner instances. Such a case can only be fixed, by manually decreasing the collection variable.

Describe the solution you'd like

• I can spawn as many instances as defined by the input collection
• instead of spawning the instances all at once on activating the body (current behavior), spawn the instances step-wise until all instances are created (e.g. spawn 10 instances and write a record to spawn the next 10)
• the instance spawning can be interrupted by an event or a terminate command

Describe alternatives you've considered

• increasing buffers and record max length - see Increase buffer size to write more records in a batch #2880
• splitting the creation into multiple steps (e.g. loop or nested multi-instance) or control it externally -

Relates to
https://jira.camunda.com/browse/SUPPORT-16653
https://jira.camunda.com/browse/SUPPORT-16549

[saig0]

Using Zeebe 0.21.0, we can execute a multi-instance activity with an input collection of ~ 5.000 elements (depending on the variables). By increasing the max message size, we could process even larger collections.

Since this is ok for now, we will postpone the issue.

[npepinpe]

Just a side note, let's postpone at least until the new engine is done (i.e. at least Q2 2021). I can see us working on reducing the max message size next year, so we will need to implement this kind of "chunking" - obviously can't guarantee, but it sounds plausible to me.

[saig0]

Update: I tested it again with version 1.0.0-alpha7. We can now iterate over an input collection with up to ~12.000 elements 🎉

[korthout]

Marking priority as later because 'multi-instance for large collections' is not the main concern for the process automation team right now. However, we should probably work on this when we work on

• Multi-instance parallel subprocess times out and never completes for large input collections #8687

Note that the impact of this bug is reduced once we've resolved

• Create incident on multi instance creation to avoid banned instances #5221

Please comment if you think this should have a higher priority.

[epollum]

Hello team, I want to mention that we have a customer who requires the ability to use large multi-instance activities. Please see https://jira.camunda.com/browse/SUPPORT-16499
And #11355

Thank you!

[daniel-ewing]

Hi team, here is another one: https://jira.camunda.com/browse/SUPPORT-16653

[npepinpe]

And another: https://jira.camunda.com/browse/SUPPORT-16549

[abbasadel]

Team meeting: we changed the priority to "Upcoming" to pick this up when we have time

[felix-mueller]

Hey @remcowesterhoud
I saw you worked on this item, there is one more item in the backlog: #8687

Could you perhaps elaborate if this is fixed now as well or is there a case which is not covered?
Could you perhaps explain which case is not covered for #8687?

Thanks
Felix

[remcowesterhoud]

@felix-mueller from what I understand after reading #8687 it is a different issue. This one had to do with instance banning and the original author of the issue explicitly mentioned his instances weren't getting banned.

Whether this is fixed, I am not sure as I'm struggling to understand what is happening in the other issue 😅 @korthout do you have any ideas? It reads to me like it's more about job activation than multi-instance input collections.

If it's stale we can consider closing it and seeing if it ever occurs again.

[korthout]

@remcowesterhoud I've had a look. IMO, that issue is not specifically related to multi-instance, but rather to activating many jobs with large variables in the Go client.

I've provided my thoughts on that issue here