Orleans on Linux times out

[datt265]

I am using Orleans TestClusterBuilder and I have some tests, I am switching from running those tests in a Windows Build agent on TeamCity to a Linux one. The first challenge was to setup the SQLDatabase and for this, I am using Testconinters to setup and initialize my DB.

My problem that on Linux I am experiencing timeouts as below.
System.TimeoutException : Response did not arrive on time in 00:00:30 for message: Request *cli/e41d9395@8de17390->S127.0.0.1:40625:0*grn/Test.Implementation.Grains.ConnectionSessionGrain/0+lVihHsn1rBb InvokeMethodRequest Test.Contracts.IConnectionSessionGrain:HandlePacket #813. Last known status is IsExecuting: True, IsWaiting: False, Diagnostics: .....

From some investigation, I went through I noticed S127.0.0.1:40625 in the error log and I assume this is the culprit of my timeouts.

The code below is how I am setting up my Silos

		public async Task CreateTestCluster(IMessageSink messageSink)
		{
			await DatabaseConfiguration.InitConfiguration(messageSink);

			// Setup and start the cluster
			var builder = new TestClusterBuilder(2);
			builder.AddSiloBuilderConfigurator<DbTestSiloBuilderConfigurator>();

			// Pass the instance name to DbTestSiloBuilderConfigurator
			builder.ConfigureHostConfiguration(configBuilder =>
				configBuilder.AddInMemoryCollection(new[] {
					KeyValuePair.Create(nameof(DbTestSiloBuilderConfigurator), DatabaseConfiguration.DatabaseInstanceName)
				})
			);

			Cluster = builder.Build();

			if (Cluster is { Primary: null })
			{
				await Cluster.DeployAsync();
			}
		}

public void Configure(ISiloBuilder siloBuilder)
{
	var configuration = (ConfigurationRoot)siloBuilder.Properties["Configuration"];

	DatabaseInstanceName = configuration.GetValue<string>(nameof(DbTestSiloBuilderConfigurator));
	string connectionString = $"{OrleansDatabaseConnection}Initial Catalog=casino_game;";

	siloBuilder
		.ConfigureEndpoints(Dns.GetHostName(), 22222, 30000)
		.Configure<ClusterOptions>(options =>
		{
			options.ClusterId = "test";
			options.ServiceId = "test";
		})
		.UseAdoNetClustering(options =>
		{
			options.ConnectionString = connectionString;
			options.Invariant = "System.Data.SqlClient";
		})
		.UseAdoNetReminderService(options =>
		{
			options.ConnectionString = connectionString;
			options.Invariant = "System.Data.SqlClient";
		})
		.ConfigureServices(ConfigureServices)
		.Configure<SchedulingOptions>(options => { options.AllowCallChainReentrancy = false; })
		.Configure<GrainCollectionOptions>(options =>
		{
			// ConnectionSessionGrains are only needed for a very short time. Must be over the CollectionQuantum time that is set at 1 min
			options.ClassSpecificCollectionAge[typeof(ConnectionSessionGrain).FullName] = TimeSpan.FromMinutes(2);
			// We need to take a good guess as to how long a player is inactive before becoming active again
			options.ClassSpecificCollectionAge[typeof(PlayerGrain).FullName] = TimeSpan.FromMinutes(2);
			options.ClassSpecificCollectionAge[typeof(RateLimitGrain).FullName] = TimeSpan.FromMinutes(2);
		})
		.AddStartupTask((services, cancellation) => StartupTask(services))
		.AddIncomingGrainCallFilter<OrleansGrainCallFilter>();
}

Locally [OrleansMembershipTable] looks something like this but I have no access to view when the tests are running on Linux.

From my understanding is that defining .ConfigureEndpoints(Dns.GetHostName(), 22222, 30000) I should see a different IP, and ports.

I have also tried hardcoding IP address instead of Dns.GetHostName() to see if I see a different IP in the table rather than 127.0.0.1

Any ideas of what I am missing ?

[datt265]

@ReubenBond can you help?

[ReubenBond]

Could you paste the rest of the error message and the test code?

[ReubenBond]

I see the request has been executing for about 9 seconds and that it was enqueued 9 seconds ago. Given that, I wonder why a 30s timeout expired. What does this HandlePacket method do? Is it blocking?

[ReubenBond]

The while (true) loop here looks like it could be the issue - it looks like it could loop forever (eg, if custId is not null or connectionId is not null)

[ReubenBond]

I believe this is the most likely reason. Please let us know if fixing this worked for you.