Pulling agents don't work properly with MemoryStreams on multiple silos

[radovcicmarko]

We're trying to migrate our service to from Orleans 3.x to 7.0. Current solution uses SMS for streaming messages from a producer grain to multiple consumer grains.

I'm currently trying to replace that with MemoryStreams. We have it set up in the way where stream communication is contained within a silo, no messages are streamed between silos. We've achieved that using LocalPlacement and having separate streams per silo.

Trying to do the same thing here using silo port as a part of stream name silo.AddMemoryStreams($"{StreamSettings.MEMORY_STREAM_PROVIDER}_{portInfo.SiloPort}" it seems that pulling agents only deque messages from a single silo although all of them have stream producers (I've tried to run it with 2 and 3 instances).

(Here is the sample repo)[https://github.com/radovcicmarko/orleans-memory-stream-example].

When I start 3 silo instances it seems that pulling agents call dequeue method on MemoryStreamQueueGrain only on one silo (one that started first), although according to logs they have queues they are responsible for on all 3 instances. Also it's weird that first silo has 8 activations of MemoryStreamQueueGrain (partition size is 8), but others start with random number of activations.

After that I run a client app that initializes consumer and producer grains (producer grains are initialized by consumer grains and Local placement is enforced) and only consumer grains on first silo get stream messages even though messages are published on all 3 instances. After some time Can not enqueue since the count has reached its maximum exception will show up on other 2 silos.

Does anyone has any idea on how could this be resolved.