wip: experiment with EntityRef at ActorSystem level #30934

octonato · 2021-11-23T10:08:48Z

This is just an experiment. Still very hacky and very incomplete.

The main idea is to have Entities directly available at ActorSystem level and not only when using cluster sharding.
This is very similar to Orleans' Virtual Actors where actor instantiation and lifecycle is managed by the actor system instead of by the user.

The basic API in this PR look as following:

val typeKey = akka.actor.typed.EntityTypeKey[Foo]("my-foo-entity")
system.initEntity(akka.actor.typed.Entity(typeKey)(ctx => behavior(ctx.shard)))

val entity: EntityRef[Foo] = system.entityRefFor(typeKey, "abc")

Note: akka.actor.typed.EntityTypeKey and akka.actor.typed.Entity. New types in akka-actor-typed, so not reusing the existing sharding types.

Like with cluster sharding, the Entity is an Actor that is managed by Akka. Akka would know if there is one already instantiated for id "abc" and decide if the Actor needs to start or not. Akka would also be responsible for passivating that same Actor.

If the current Actor System is using cluster, then the above would result in a clustered entity (sharded) instead of a local Entity. So there is two possible code paths behind system.initEntity and system.entityRefFor.

For non-clustered actor systems, this entity will be a local actor managed by Akka. Not implemented yet.
For clustered actor system, the akka.actor.typed.Entity needs to be converted to a akka.cluster.sharding.typed.scaladsl.Entity (or javadsl) and registered as a cluster sharding entity. The current conversion is incomplete and only support the most basic configs.

The approach in this PR (duplicating and converting types) is a little heavy and fastidious, but gives a clean API to the users.

Another approach for local entities can be to not try to make then transparent. We could introduce a LocalEntity type to be used directly on the Actor System with similar functionality as a sharded entity (managed instantiation, passivation, etc), but explicitly local.

That will simplify things a lot, IMO. Not sure if it's worth the complexity of duplicating the types and do all the conversion. If an user starts with a LocalEntity and later decide to move to Sharded Entity, the code refactoring should be minimal.

We could also imagine a situation in which we do have an Akka Cluster, but we want a particular actor to be a local entity. In that case, it will be good to have an explicit LocalEntity.

Also, such LocalEntity overlaps with some of the functionality provided by the Receptionist. If we can get an instance of an actor by using a type key and an ID, we may not need to use Receptionist. Or course, their might be valid cases for Receptionist.

octonato · 2021-11-23T10:19:12Z

...sharding-typed/src/test/scala/akka/cluster/sharding/typed/scaladsl/ClusterShardingSpec.scala

+    "EntityRef - tell (using system.entityRefFor) " in {
+      val typeKeyWithEnvelopes = akka.actor.typed.EntityTypeKey[TestProtocol]("envelope-shard")
+      val charlieRef = system.entityRefFor(typeKeyWithEnvelopes, "charlie")
+
+      val p = TestProbe[String]()
+
+      charlieRef ! WhoAreYou(p.ref)
+      p.receiveMessage() should startWith("I'm charlie")
+
+      charlieRef.tell(WhoAreYou(p.ref))
+      p.receiveMessage() should startWith("I'm charlie")
+
+      charlieRef ! StopPlz()
+    }
+
+    "EntityRef - tell (using system.initEntity system.entityRefFor) " in {
+      val typeKeyWithEnvelopes = akka.actor.typed.EntityTypeKey[TestProtocol]("envelope-shard-2")
+      system.initEntity(akka.actor.typed.Entity(typeKeyWithEnvelopes)(ctx => behavior(ctx.shard)))
+      val charlieRef = system.entityRefFor(typeKeyWithEnvelopes, "charlie")
+
+      val p = TestProbe[String]()
+
+      charlieRef ! WhoAreYou(p.ref)
+      p.receiveMessage() should startWith("I'm charlie")
+
+      charlieRef.tell(WhoAreYou(p.ref))
+      p.receiveMessage() should startWith("I'm charlie")
+
+      charlieRef ! StopPlz()
+    }
+


I just added this here to move fast. If we go forward with this idea, it must have it's own tests. Probably repeating everything that's done in this test, but using system.initEntity and system.entityRefFor

patriknw

I like the idea, and maybe it's not so intrusive...

patriknw · 2021-11-28T18:20:43Z

akka-actor-typed/src/main/scala/akka/actor/typed/Entity.scala

+    val messageExtractor: Option[EntityMessageExtractor[E, M]],
+    val allocationStrategy: Option[EntityAllocationStrategy],
+    val role: Option[String],
+    val dataCenter: Option[DataCenter])


There are several things that are only relevant in a clustered setup. role and dataCenter obviously, but also the whole idea of shard and allocationStrategy are not relevant for the local case.

One idea is to only support the very basic init in akka-actor-typed. No custom envelope, no custom message extractor, no allocationStrategy. For more advanced cases the init in akka-cluster-sharding-typed` can be used.

That means that only EntityTypeKey and EntityRef have to be moved to akka-actor-typed.

Yes, everything cluster/sharding related could be removed. But that also means that when moving to sharded entities, the user will need to adapt.

Also, what would that mean to declare an entity using the local API, but on a clustered system? Will it be unique on that node only? Or will it be a sharded one?

So maybe we should take this as a trade-off. We will have some settings that only make sense when clustered and when used in a non-clustered system they are simply ignored.

If we chose to have new APIs and deprecate the exiting sharding entity, then we could make it more explicitly as part of a cluster API. Something like:

Entity(...).withShardingSettings(...)

When not specified, fallback to some defaults. Actually, pretty much like now with Option but behind an explicit sharding settings.

But that also means that when moving to sharded entities, the user will need to adapt.

Also, what would that mean to declare an entity using the local API, but on a clustered system? Will it be unique on that node only? Or will it be a sharded one?

I was thinking that the basic init API would also work for a clustered system, and then it would init ClusterSharding under the hood. All these options are optional in ClusterSharding.

Meaning that the API would only change if user needs something more advanced, then they would have to use the ClusterSharding.init instead.

patriknw · 2021-11-28T18:46:02Z

akka-actor-typed/src/main/scala/akka/actor/typed/internal/entity/EntityProvider.scala

+  def entityRefFor[M](typeKey: EntityTypeKey[M], entityId: String): EntityRef[M]
+}
+
+private[akka] object LocalEntityProvider extends EntityProvider {


I think it would be worth having a new implementation for the local case. It would be much simpler than the cluster variant. No shards at all. Still handling lifecycle and passivation.

The passivation strategies that Peter is working on can probably be used for the local case also.

johanandren · 2021-11-29T07:55:33Z

Looked at this a few times since opened and, I also like the idea. Haven't dug in more to give any detailed feedback about the impl here though.

octonato · 2021-11-30T11:28:31Z

Looked at this a few times since opened and, I also like the idea. Haven't dug in more to give any detailed feedback about the impl here though.

@johanandren, the impl is more a sketch. My goal was to hacking something to find out which parts we will need. Probably everything in this PR should be re-worked.

octonato

I re-worked this a bit.

Now we have a copy of ClusterShardingSpec running entirely with the new API.
Still need lots of code massage, but I think it's proved that this can be done.
MiMa is working as well. So far, nothing broken.

akka-actor-typed/src/main/protobuf/EntityMessages.proto

octonato · 2021-12-09T20:45:09Z

akka-actor-typed/src/main/resources/reference.conf

+    "akka.actor.typed.EntityEnvelope" = akka-actor-typed-entity
+  }
+  serialization-identifiers {
+    "akka.actor.typed.internal.entity.EntityEnvelopeSerializer" = 41


I saw that we just added 39, so I tried 40, but it's already picked.

What are the rules for this? And how can have an overview of all identifiers in use?

they are defined in reference.conf in section serialization-identifiers, so the way is to search for a free spot

by the way, 41 is taken by PingSerializer in some test, but we can change that to a higher number

feels wrong to have serialization in this local module, though. It can be defined in Cluster Sharding (maybe in some of the existing serializers)

Yes, I was not so happy to have it here, but I think we can remove it all together.

@patriknw, it's not possible to remove the serialization. It's possible to workaround it when using the EntityRef, but not when using the shard region ActorRef (the one returned by initEntity). Whatever we pass here can end up on the wire before we can do any conversion.

So, I kept the serialization, but now moved to cluster sharding. It's a little odd, because it's a class defined in akka-actor-typed, but the serialization is akka-cluster-sharding-typed. Anyway, I don't see any other use case where the serialization will be needed while not using sharding.

I think it's fair to put a limit on it and say that the new entity, if used over remote, must be in the context of cluster sharding. In other words, not remote entity if not sharded.

akka-actor-typed/src/main/scala/akka/actor/typed/internal/entity/EntityEnvelopeSerializer.scala

octonato · 2021-12-09T20:47:43Z

...yped/src/main/scala/akka/cluster/sharding/typed/internal/ClusterShardingEntityProvider.scala

+    override def hashCode(): Int = delegate.hashCode()
+    override def equals(obj: Any): Boolean = delegate.equals(obj)


This two are probably not needed. Also not sure if a good idea to do full delegation.

octonato · 2021-12-09T20:48:36Z

...sharding-typed/src/main/scala/akka/cluster/sharding/typed/internal/ClusterShardingImpl.scala

@@ -53,6 +55,7 @@ import akka.util.JavaDurationConverters._
    extends ShardingMessageExtractor[Any, M] {
  override def entityId(message: Any): String = {
    message match {
+      case EntityEnvelope(entityId, _)   => entityId


we will be able to remove this if we manage to do the full conversion and remove the serialization.

octonato · 2021-12-09T20:49:03Z

...ter-sharding-typed/src/main/scala/akka/cluster/sharding/typed/scaladsl/ClusterSharding.scala

@@ -39,7 +41,7 @@ object ClusterSharding extends ExtensionId[ClusterSharding] {
   *
   * Not for user extension.
   */
-  @DoNotInherit trait ShardCommand
+  @DoNotInherit trait ShardCommand extends EntityCommand


this is probably not needed

octonato · 2021-12-09T20:49:41Z

...d/src/test/scala/akka/cluster/sharding/typed/scaladsl/NewEntityWithClusterShardingSpec.scala

+import org.scalatest.wordspec.AnyWordSpecLike
+
+@ccompatUsedUntil213
+object NewEntityWithClusterShardingSpec {


This is a copy of ClusterShardingSpec, but then using the new types. All tests are passing!

We may want to remove this and refactor ClusterShardingSpec to use new API.

...yped/src/main/scala/akka/cluster/sharding/typed/internal/ClusterShardingEntityProvider.scala

lightbend-cla-validator · 2022-09-13T12:20:31Z

Hi @octonato,

Thank you for your contribution! We really value the time you've taken to put this together.

We see that you have signed the Lightbend Contributors License Agreement before, however, the CLA has changed since you last signed it.
Please review the new CLA and sign it before we proceed with reviewing this pull request:

https://www.lightbend.com/contribute/cla

octonato commented Nov 23, 2021

View reviewed changes

patriknw reviewed Nov 28, 2021

View reviewed changes

octonato force-pushed the octonato/non-sharded-entities branch 2 times, most recently from 36d846a to ec227be Compare December 9, 2021 20:32

octonato commented Dec 9, 2021

View reviewed changes

...yped/src/main/scala/akka/cluster/sharding/typed/internal/ClusterShardingEntityProvider.scala Outdated Show resolved Hide resolved

octonato force-pushed the octonato/non-sharded-entities branch from 3941e78 to 3be15c8 Compare December 27, 2021 18:00

octonato force-pushed the octonato/non-sharded-entities branch 2 times, most recently from 866259d to 2371988 Compare February 14, 2022 12:37

octonato force-pushed the octonato/non-sharded-entities branch 3 times, most recently from e8a14d2 to 99039e3 Compare March 29, 2022 07:22

octonato force-pushed the octonato/non-sharded-entities branch from 50d54f1 to 27190a0 Compare September 13, 2022 12:20

octonato closed this Sep 13, 2022

octonato reopened this Sep 13, 2022

octonato force-pushed the octonato/non-sharded-entities branch 2 times, most recently from a80e867 to f49a749 Compare September 28, 2022 14:36

non-sharded entities, aka virtual actors

26b6c75

octonato force-pushed the octonato/non-sharded-entities branch from f49a749 to 26b6c75 Compare September 28, 2022 15:15

leviramsey mentioned this pull request Oct 3, 2022

Introduce mapAsyncPartitioned stream operator #31582

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

wip: experiment with EntityRef at ActorSystem level #30934

wip: experiment with EntityRef at ActorSystem level #30934

octonato commented Nov 23, 2021

octonato Nov 23, 2021

patriknw left a comment

patriknw Nov 28, 2021

patriknw Nov 28, 2021

octonato Nov 30, 2021

patriknw Nov 30, 2021

patriknw Nov 28, 2021

johanandren commented Nov 29, 2021

octonato commented Nov 30, 2021

octonato left a comment

octonato Dec 9, 2021

patriknw Dec 10, 2021

octonato Dec 15, 2021

octonato Dec 27, 2021

octonato Dec 9, 2021

octonato Dec 9, 2021

octonato Dec 9, 2021

octonato Dec 9, 2021

lightbend-cla-validator commented Sep 13, 2022

		override def hashCode(): Int = delegate.hashCode()
		override def equals(obj: Any): Boolean = delegate.equals(obj)

wip: experiment with EntityRef at ActorSystem level #30934

Are you sure you want to change the base?

wip: experiment with EntityRef at ActorSystem level #30934

Conversation

octonato commented Nov 23, 2021

Choose a reason for hiding this comment

patriknw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

johanandren commented Nov 29, 2021

octonato commented Nov 30, 2021

octonato left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lightbend-cla-validator commented Sep 13, 2022