Workflow activity cannot be run - 'too many activities are running in parallel' error.

[papakhristos-sharpdev]

In what area(s)?

/area runtime

/area operator

/area placement

/area docs

/area test-and-release

What version of Dapr?

1.13

1.1.x
1.0.x
edge: output of git describe --dirty

Expected Behavior

Workflow activities should execute their logic.

Actual Behavior

We receive an error in the side-car log: "execution failed with a non-recoverable error: timed-out trying to schedule an activity execution - this can happen if too many activities are running in parallel or if the workflow engine isn't running: context deadline exceeded"

Steps to Reproduce the Problem

We have a workflow consisting of several activities running consequently step by step. Workflow is a part of the .NET 8 service working inside a Docker container (we run it using Docker Compose).

One of the activities in this workflow is waiting for an external event for 3 days and returning the error output on time. A successful workflow run returns positive output. We catch WorkflowTaskException and return the error output in case of such an exception.

Our UI/tests create workflows in the database and our backend service starts them according to our internal schedule. At some point after the start of our service, our workflows are started but do nothing and we receive 'too many activities' error in side-car logs.
The only solution to fix it is to restart the service and its sidecar. After that, we can re-create workflows and they work until another failure.

We don't have any retry policy attached to our activities.

We found another issue that looks very similar to ours - #6878 - but it is closed.

Release Note

RELEASE NOTE: FIX Bug in runtime.