Create incident on multi instance creation to avoid banned instances

[Zelldon]

Description

We still have the issue that we are not able to create large multi instances #2890. We will blacklist the instance quite easy, but this means the user is not able to resolve it. If we would create an incident before, the user can adjust the collection which is used to create the multi instance. It makes the system more usable.

Break down of #5123.

Explanation

If a workflow instance is blacklisted, then this instance can't be used anymore. To avoid that we should try to create incident before that happens. We do this for example before job activation, if we realize that we are not able to activate the job, since it has to large variables (#4420).

Blacklisting should only be used to prevent bugs to cause more issues, but if we know limitations of the system then we should handle them properly.

Support Case: https://jira.camunda.com/browse/SUPPORT-14156

[menski]

Might become more urgent if we offer setups with decreased messages sizes

[Zelldon]

We have seen again some errors in Production:

Error group https://console.cloud.google.com/errors/CKGY9ZmHlJvyQA?service=zeebe&time=P7D&refresh=off&project=camunda-cloud-240911

java.lang.IllegalArgumentException: Expected to claim segment of size 34200920, but can't claim more than 4194304 bytes.
at io.camunda.zeebe.dispatcher.Dispatcher.offer (Dispatcher.java:194)
at io.camunda.zeebe.dispatcher.Dispatcher.claimFragmentBatch (Dispatcher.java:164)
at io.camunda.zeebe.logstreams.impl.log.LogStreamBatchWriterImpl.claimBatchForEvents (LogStreamBatchWriterImpl.java:222)
at io.camunda.zeebe.logstreams.impl.log.LogStreamBatchWriterImpl.tryWrite (LogStreamBatchWriterImpl.java:199)
at io.camunda.zeebe.engine.processing.streamprocessor.writers.TypedStreamWriterImpl.flush (TypedStreamWriterImpl.java:106)
at io.camunda.zeebe.engine.processing.bpmn.behavior.TypedStreamWriterProxy.flush (TypedStreamWriterProxy.java:59)
at io.camunda.zeebe.engine.processing.streamprocessor.ProcessingStateMachine.lambda$writeEvent$6 (ProcessingStateMachine.java:342)
at io.camunda.zeebe.util.retry.ActorRetryMechanism.run (ActorRetryMechanism.java:36)
at io.camunda.zeebe.util.retry.AbortableRetryStrategy.run (AbortableRetryStrategy.java:44)
at io.camunda.zeebe.util.sched.ActorJob.invoke (ActorJob.java:73)
at io.camunda.zeebe.util.sched.ActorJob.execute (ActorJob.java:39)
at io.camunda.zeebe.util.sched.ActorTask.execute (ActorTask.java:122)
at io.camunda.zeebe.util.sched.ActorThread.executeCurrentTask (ActorThread.java:95)
at io.camunda.zeebe.util.sched.ActorThread.doWork (ActorThread.java:78)
at io.camunda.zeebe.util.sched.ActorThread.run (ActorThread.java:192)

2022-01-11 10:51:01.634 CET
zeebe
"Expected to write one or more follow up events for event 'LoggedEvent [type=0, version=16384, streamId=1, position=5537, key=2251799813687658, timestamp=1641894661001, sourceEventPosition=5531] RecordMetadata{recordType=COMMAND, intentValue=255, intent=ACTIVATE_ELEMENT, requestStreamId=-2147483648, requestId=-1, protocolVersion=3, valueType=PROCESS_INSTANCE, rejectionType=NULL_VAL, rejectionReason=, brokerVersion=1.2.9}' without errors, but exception was thrown."
Error
2022-01-11 10:51:01.655 CET
zeebe
"Expected to process event 'TypedEventImpl{metadata=RecordMetadata{recordType=COMMAND, intentValue=255, intent=ACTIVATE_ELEMENT, requestStreamId=-2147483648, requestId=-1, protocolVersion=3, valueType=PROCESS_INSTANCE, rejectionType=NULL_VAL, rejectionReason=, brokerVersion=1.2.9}, value={"bpmnProcessId":"camunda-cloud-quick-start","version":4,"processDefinitionKey":2251799813686918,"processInstanceKey":2251799813687636,"elementId":"Activity_0450a81","flowScopeKey":2251799813687636,"bpmnElementType":"MULTI_INSTANCE_BODY","parentProcessInstanceKey":-1,"parentElementInstanceKey":-1}}' without errors, but exception occurred with message 'Expected to claim segment of size 34200920, but can't claim more than 4194304 bytes.'."
Warning
2022-01-11 10:51:01.656 CET
zeebe
"Blacklist process instance 2251799813687636, due to previous errors."
Warning
2022-01-11 10:51:01.665 CET
zeebe
"Blacklist process instance 2251799813687636, due to previous errors."
Warning
2022-01-11 10:51:01.665 CET
zeebe
"Blacklist process instance 2251799813687636, due to previous errors."

It seems that the user wanted to activate a multi instance with a big collection which caused a record size of 32 MB.

[npepinpe]

Let's leave it in the backlog for now, as there is no quick fix and it's one specific case of two general problems: blacklisting (no visibility) and limitations of the dispatcher. However, once #5525 is fixed, we can revisit this if we can reuse the solution from the PR as a quick fix.

[korthout]

This was raised in a support case. I'd like to raise awareness of this issue to potentially work on it soon.

After #9420, the mentioned incident should be easier to create. We should then be able to activate the inner instances until the limit is reached, handle the allocation error, roll back the transaction and raise an incident.

[Zelldon]

Since we are able now to process large input collection #12692 I would mark this issue as closed.