Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: Gateway Thread Stuck Connecting on Oracle Cloud #2750

Open
3 tasks done
winglessraven opened this issue Aug 10, 2023 · 7 comments
Open
3 tasks done

[Bug]: Gateway Thread Stuck Connecting on Oracle Cloud #2750

winglessraven opened this issue Aug 10, 2023 · 7 comments
Assignees
@Misha-133
Labels
bug

Comments

@winglessraven
Copy link

Check The Docs

  • I double checked the docs and couldn't find any useful information.

Verify Issue Source

  • I verified the issue was caused by Discord.Net.

Check your intents

  • I double checked that I have the required intents.

Description

I have a Discord bot plugin for a game server management panel that is compiled as a class library. It works without issues on Windows and most Linux distros.

On Oracle Cloud servers (aarch64 architecture, image: Canonical-Ubuntu-22.04-aarch64-2022.11.06-0, 4 * OCPU, 24GB RAM) the gateway thread seems to be hanging on Connecting.

I have tried to narrow down where the issue is but cannot get anything other than Connecting from the gateway.

I have tested a python bot on the same server to verify it is not a network issue.

Not sure what else I can try to debug the issue and find the root cause.

Version

3.11.0

Working Version

No response

Logs

No stack trace, but debug message I've added shows the connection state being stuck:

[06:44:09] [DiscordBot Info]      : 06:44:09 Discord     Discord.Net v1.0.0 (API v10)
[06:44:09] [DiscordBot Info]      : 06:44:09 Gateway     Connecting
[06:44:09] [DiscordBot Debug]     : Server Status: Stopped || Players: 0/10 || CPU: 0% || Memory: 0MB, Bot Connection Status: Connecting
[06:44:39] [DiscordBot Debug]     : Server Status: Stopped || Players: 0/10 || CPU: 0% || Memory: 0MB, Bot Connection Status: Connecting
[06:45:09] [DiscordBot Debug]     : Server Status: Stopped || Players: 0/10 || CPU: 0% || Memory: 0MB, Bot Connection Status: Connecting
[06:45:39] [DiscordBot Debug]     : Server Status: Stopped || Players: 0/10 || CPU: 0% || Memory: 0MB, Bot Connection Status: Connecting
[06:46:09] [DiscordBot Debug]     : Server Status: Stopped || Players: 0/10 || CPU: 0% || Memory: 0MB, Bot Connection Status: Connecting
[06:46:39] [DiscordBot Debug]     : Server Status: Stopped || Players: 0/10 || CPU: 0% || Memory: 0MB, Bot Connection Status: Connecting
[06:47:09] [DiscordBot Debug]     : Server Status: Stopped || Players: 0/10 || CPU: 0% || Memory: 0MB, Bot Connection Status: Connecting
[06:47:39] [DiscordBot Debug]     : Server Status: Stopped || Players: 0/10 || CPU: 0% || Memory: 0MB, Bot Connection Status: Connecting
[06:48:09] [DiscordBot Debug]     : Server Status: Stopped || Players: 0/10 || CPU: 0% || Memory: 0MB, Bot Connection Status: Connecting
[06:48:39] [DiscordBot Debug]     : Server Status: Stopped || Players: 0/10 || CPU: 0% || Memory: 0MB, Bot Connection Status: Connecting
[06:49:09] [DiscordBot Debug]     : Server Status: Stopped || Players: 0/10 || CPU: 0% || Memory: 0MB, Bot Connection Status: Connecting
[06:49:39] [DiscordBot Debug]     : Server Status: Stopped || Players: 0/10 || CPU: 0% || Memory: 0MB, Bot Connection Status: Connecting
[06:50:09] [DiscordBot Debug]     : Server Status: Stopped || Players: 0/10 || CPU: 0% || Memory: 0MB, Bot Connection Status: Connecting
[06:50:39] [DiscordBot Debug]     : Server Status: Stopped || Players: 0/10 || CPU: 0% || Memory: 0MB, Bot Connection Status: Connecting
[06:51:09] [DiscordBot Debug]     : Server Status: Stopped || Players: 0/10 || CPU: 0% || Memory: 0MB, Bot Connection Status: Connecting
[06:51:39] [DiscordBot Debug]     : Server Status: Stopped || Players: 0/10 || CPU: 0% || Memory: 0MB, Bot Connection Status: Connecting
[06:52:09] [DiscordBot Debug]     : Server Status: Stopped || Players: 0/10 || CPU: 0% || Memory: 0MB, Bot Connection Status: Connecting
[06:52:39] [DiscordBot Debug]     : Server Status: Stopped || Players: 0/10 || CPU: 0% || Memory: 0MB, Bot Connection Status: Connecting
[06:53:09] [DiscordBot Debug]     : Server Status: Stopped || Players: 0/10 || CPU: 0% || Memory: 0MB, Bot Connection Status: Connecting
[06:53:39] [DiscordBot Debug]     : Server Status: Stopped || Players: 0/10 || CPU: 0% || Memory: 0MB, Bot Connection Status: Connecting
[06:54:09] [DiscordBot Debug]     : Server Status: Stopped || Players: 0/10 || CPU: 0% || Memory: 0MB, Bot Connection Status: Connecting
[06:54:39] [DiscordBot Debug]     : Server Status: Stopped || Players: 0/10 || CPU: 0% || Memory: 0MB, Bot Connection Status: Connecting
[06:55:09] [DiscordBot Debug]     : Server Status: Stopped || Players: 0/10 || CPU: 0% || Memory: 0MB, Bot Connection Status: Connecting
[06:55:39] [DiscordBot Debug]     : Server Status: Stopped || Players: 0/10 || CPU: 0% || Memory: 0MB, Bot Connection Status: Connecting
[06:56:09] [DiscordBot Debug]     : Server Status: Stopped || Players: 0/10 || CPU: 0% || Memory: 0MB, Bot Connection Status: Connecting
[06:56:39] [DiscordBot Debug]     : Server Status: Stopped || Players: 0/10 || CPU: 0% || Memory: 0MB, Bot Connection Status: Connecting
[06:57:09] [DiscordBot Debug]     : Server Status: Stopped || Players: 0/10 || CPU: 0% || Memory: 0MB, Bot Connection Status: Connecting
[06:57:39] [DiscordBot Debug]     : Server Status: Stopped || Players: 0/10 || CPU: 0% || Memory: 0MB, Bot Connection Status: Connecting
[06:58:09] [DiscordBot Debug]     : Server Status: Stopped || Players: 0/10 || CPU: 0% || Memory: 0MB, Bot Connection Status: Connecting

Sample

Full code is at https://github.com/winglessraven/AMP-Discord-Bot, but my ConnectDiscordAsync method is below:

public async Task ConnectDiscordAsync(string BotToken)
        {
            DiscordSocketConfig config;

            // Determine the GatewayIntents based on the chat settings
            if (_settings.MainSettings.SendChatToDiscord || _settings.MainSettings.SendDiscordChatToServer)
            {
                // Include MessageContent intent if chat is sent between Discord and the server
                config = new DiscordSocketConfig { GatewayIntents = GatewayIntents.DirectMessages | GatewayIntents.GuildMessages | GatewayIntents.Guilds | GatewayIntents.MessageContent };
            }
            else
            {
                config = new DiscordSocketConfig { GatewayIntents = GatewayIntents.DirectMessages | GatewayIntents.GuildMessages | GatewayIntents.Guilds };
            }

            if(_settings.MainSettings.DiscordDebugMode)
                config.LogLevel = LogSeverity.Verbose;

            // Initialize Discord client with the specified configuration
            _client = new DiscordSocketClient(config);

            // Attach event handlers for logs and events
            _client.Log += Log;
            _client.ButtonExecuted += OnButtonPress;
            _client.Ready += ClientReady;
            _client.SlashCommandExecuted += SlashCommandHandler;
            if (_settings.MainSettings.SendChatToDiscord || _settings.MainSettings.SendDiscordChatToServer)
                _client.MessageReceived += MessageHandler;

            // Login and start the Discord client
            await _client.LoginAsync(TokenType.Bot, BotToken);
            await _client.StartAsync();

            //console output
            await ConsoleOutputSend();

            // Set the bot's status
            await SetStatus();

            // Block this task until the program is closed or bot is stopped.
            await Task.Delay(-1);
        }

Packages

<packages>
  <package id="Discord.Net" version="3.11.0" targetFramework="net48" />
  <package id="Discord.Net.Commands" version="3.11.0" targetFramework="net48" />
  <package id="Discord.Net.Core" version="3.11.0" targetFramework="net48" />
  <package id="Discord.Net.Interactions" version="3.11.0" targetFramework="net48" />
  <package id="Discord.Net.Rest" version="3.11.0" targetFramework="net48" />
  <package id="Discord.Net.Webhook" version="3.11.0" targetFramework="net48" />
  <package id="Discord.Net.WebSocket" version="3.11.0" targetFramework="net48" />
  <package id="ILMerge" version="3.0.41" targetFramework="net48" />
  <package id="ILMerge.MSBuild.Task" version="1.0.7" targetFramework="net48" />
  <package id="Microsoft.Bcl.AsyncInterfaces" version="6.0.0" targetFramework="net48" />
  <package id="Microsoft.Extensions.DependencyInjection.Abstractions" version="6.0.0" targetFramework="net48" />
  <package id="Newtonsoft.Json" version="13.0.3" targetFramework="net48" />
  <package id="System.Buffers" version="4.5.1" targetFramework="net48" />
  <package id="System.Collections.Immutable" version="6.0.0" targetFramework="net48" />
  <package id="System.Interactive.Async" version="6.0.1" targetFramework="net48" />
  <package id="System.Linq.Async" version="6.0.1" targetFramework="net48" />
  <package id="System.Memory" version="4.5.5" targetFramework="net48" />
  <package id="System.Numerics.Vectors" version="4.5.0" targetFramework="net48" />
  <package id="System.Reactive" version="5.0.0" targetFramework="net48" />
  <package id="System.Runtime.CompilerServices.Unsafe" version="6.0.0" targetFramework="net48" />
  <package id="System.Threading.Tasks.Extensions" version="4.5.4" targetFramework="net48" />
  <package id="System.ValueTuple" version="4.5.0" targetFramework="net48" />
</packages>

Plus "ModuleShared" version 2.4.5.4 from the server panel that the plugin is for.
@winglessraven winglessraven mentioned this issue Aug 10, 2023
Closed
@winglessraven
Copy link
Author

Note: changed from Verbose to Debug and the log returns one additional line (10:20:52 Gateway Connecting ApiClient) and then hangs.

If I manually set the gateway host then the following is logged.

config.GatewayHost = "wss://gateway.discord.gg";
[19:24:03] [DiscordBot Info]      : 19:24:03 Discord     Discord.Net v1.0.0 (API v10)
[19:24:03] [DiscordBot Info]      : 19:24:03 Gateway     Connecting
[19:24:03] [DiscordBot Info]      : 19:24:03 Gateway     Connecting ApiClient
[19:24:03] [DiscordBot Info]      : 19:24:03 Gateway     Identifying
[19:24:03] [DiscordBot Info]      : 19:24:03 Gateway     Sent Identify
[19:24:03] [DiscordBot Info]      : 19:24:03 Gateway     Received Hello
[19:24:03] [DiscordBot Info]      : 19:24:03 Gateway     Heartbeat Started
[19:24:03] [DiscordBot Info]      : 19:24:03 Gateway     Sent Heartbeat
[19:24:03] [DiscordBot Info]      : 19:24:03 Gateway     Received HeartbeatAck
[19:24:03] [DiscordBot Info]      : 19:24:03 Gateway     Latency = 100 ms
[19:24:03] [DiscordBot Info]      : 19:24:03 Gateway     Received Dispatch (READY)
[19:24:03] [DiscordBot Info]      : 19:24:03 Gateway     GuildDownloader Started
[19:24:03] [DiscordBot Info]      : 19:24:03 Gateway     Connected
[19:24:03] [DiscordBot Info]      : 19:24:03 Gateway     Raising Event
[19:24:03] [DiscordBot Info]      : 19:24:03 Gateway     Serializer Error:
Newtonsoft.Json.JsonSerializationException: Error converting value "member" to type 'Discord.PermissionTarget'. Path 'channels[16].permission_overwrites[0].type'. ---> System.ArgumentException: Requested value 'member' was not found.
  at Newtonsoft.Json.Utilities.EnumUtils.ParseEnum (System.Type enumType, Newtonsoft.Json.Serialization.NamingStrategy namingStrategy, System.String value, System.Boolean disallowNumber) [0x001f9] in <dd21e08747d94bcebed8b56112226337>:0 
  at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.EnsureType (Newtonsoft.Json.JsonReader reader, System.Object value, System.Globalization.CultureInfo culture, Newtonsoft.Json.Serialization.JsonContract contract, System.Type targetType) [0x00054] in <dd21e08747d94bcebed8b56112226337>:0 
   --- End of inner exception stack trace ---
  at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.EnsureType (Newtonsoft.Json.JsonReader reader, System.Object value, System.Globalization.CultureInfo culture, Newtonsoft.Json.Serialization.JsonContract contract, System.Type targetType) [0x00146] in <dd21e08747d94bcebed8b56112226337>:0 
  at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal (Newtonsoft.Json.JsonReader reader, System.Type objectType, Newtonsoft.Json.Serialization.JsonContract contract, Newtonsoft.Json.Serialization.JsonProperty member, Newtonsoft.Json.Serialization.JsonContainerContract containerContract, Newtonsoft.Json.Serialization.JsonProperty containerMember, System.Object existingValue) [0x000dc] in <dd21e08747d94bcebed8b56112226337>:0 
  at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.SetPropertyValue (Newtonsoft.Json.Serialization.JsonProperty property, Newtonsoft.Json.JsonConverter propertyConverter, Newtonsoft.Json.Serialization.JsonContainerContract containerContract, Newtonsoft.Json.Serialization.JsonProperty containerProperty, Newtonsoft.Json.JsonReader reader, System.Object target) [0x00065] in <dd21e08747d94bcebed8b56112226337>:0 
  at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject (System.Object newObject, Newtonsoft.Json.JsonReader reader, Newtonsoft.Json.Serialization.JsonObjectContract contract, Newtonsoft.Json.Serialization.JsonProperty member, System.String id) [0x00239] in <dd21e08747d94bcebed8b56112226337>:0 
[19:24:03] [DiscordBot Info]      : 19:24:03 Gateway     Received Dispatch (GUILD_AVAILABLE)
[19:24:08] [DiscordBot Info]      : 19:24:08 Gateway     Connected to REQUESTS
[19:24:08] [DiscordBot Info]      : 19:24:08 Gateway     GuildDownloader Stopped
[19:24:08] [DiscordBot Info]      : Base command for bot: projectzomboid
[19:24:11] [DiscordBot Info]      : 19:24:11 Gateway     A Ready handler is blocking the gateway task.
[19:24:27] [DiscordBot Info]      : 19:24:27 Gateway     Received Dispatch (MESSAGE_UPDATE)
[19:24:28] [DiscordBot Info]      : 19:24:28 Gateway     Received Dispatch (MESSAGE_UPDATE)
[19:24:32] [DiscordBot Info]      : 19:24:32 Gateway     Received Dispatch (MESSAGE_UPDATE)
[19:24:33] [DiscordBot Info]      : 19:24:33 Gateway     Sent PresenceUpdate
[19:24:33] [DiscordBot Info]      : 19:24:33 Gateway     Sent PresenceUpdate
[19:24:33] [DiscordBot Error]     : Object reference not set to an instance of an object
[19:24:33] [DiscordBot Info]      : 19:24:33 Gateway     Received Dispatch (INTERACTION_CREATE)
[19:24:36] [DiscordBot Info]      : 19:24:36 Gateway     A SlashCommandExecuted handler is blocking the gateway task.
[19:24:44] [DiscordBot Info]      : 19:24:44 Gateway     Sent Heartbeat
[19:25:03] [DiscordBot Info]      : 19:25:03 Gateway     Sent PresenceUpdate
[19:25:03] [DiscordBot Info]      : 19:25:03 Gateway     Sent PresenceUpdate
[19:25:03] [DiscordBot Error]     : Object reference not set to an instance of an object

@Misha-133
Copy link
Member

Try specifying the api version
wss://gateway.discord.gg/?v=10&encoding=json

Also Discord.Net 1.0.0 in logs looks odd

@Misha-133
Copy link
Member

Misha-133 commented Aug 11, 2023 •
edited

Also
What are you using to run the bot
I see you have .NET framework 4.8 as the target... Do you use Mono? That might also cause issues
I'd suggest moving to .NET 6 if possible, as it has native linux runtimes & sdks

UPD: oops I've missed the fact it's a plugin

@winglessraven
Copy link
Author

winglessraven commented Aug 11, 2023 •
edited

Thanks, I literally just tried the encoding parameter this morning :)

Still seems to be locking somewhere though.

No idea where the version 1.0.0 comes from - I'm using the NuGet package (updated to 3.12.0 today).

[11:51:10] [DiscordBot Info]      : Discord Bot Activated
[11:51:10] [DiscordBot Info]      : 11:51:10 Discord     Discord.Net v1.0.0 (API v10)
[11:51:11] [DiscordBot Info]      : 11:51:11 Gateway     Connecting
[11:51:11] [DiscordBot Info]      : 11:51:11 Gateway     Connecting ApiClient
[11:51:11] [DiscordBot Info]      : 11:51:11 Gateway     Identifying
[11:51:11] [DiscordBot Info]      : 11:51:11 Gateway     Sent Identify
[11:51:11] [DiscordBot Info]      : 11:51:11 Gateway     Received Hello
[11:51:11] [DiscordBot Info]      : 11:51:11 Gateway     Heartbeat Started
[11:51:11] [DiscordBot Info]      : 11:51:11 Gateway     Sent Heartbeat
[11:51:11] [DiscordBot Info]      : 11:51:11 Gateway     Received HeartbeatAck
[11:51:11] [DiscordBot Info]      : 11:51:11 Gateway     Latency = 100 ms
[11:51:11] [DiscordBot Info]      : 11:51:11 Gateway     Received Dispatch (READY)
[11:51:11] [DiscordBot Info]      : 11:51:11 Gateway     GuildDownloader Started
[11:51:11] [DiscordBot Info]      : 11:51:11 Gateway     Connected
[11:51:11] [DiscordBot Info]      : 11:51:11 Gateway     Raising Event
[11:51:11] [DiscordBot Info]      : 11:51:11 Gateway     Received Dispatch (GUILD_AVAILABLE)
[11:51:16] [DiscordBot Info]      : 11:51:16 Gateway     Connected to REQUESTS
[11:51:16] [DiscordBot Info]      : 11:51:16 Gateway     GuildDownloader Stopped
[11:51:16] [DiscordBot Info]      : Base command for bot: projectzomboid
[11:51:19] [DiscordBot Info]      : 11:51:19 Gateway     A Ready handler is blocking the gateway task.
[11:51:31] [DiscordBot Info]      : 11:51:31 Gateway     Received Dispatch (MESSAGE_UPDATE)
[11:51:34] [DiscordBot Info]      : 11:51:34 Gateway     Received Dispatch (MESSAGE_UPDATE)
[11:51:38] [DiscordBot Info]      : 11:51:38 Gateway     Received Dispatch (MESSAGE_UPDATE)
[11:51:41] [DiscordBot Info]      : 11:51:41 Gateway     Sent PresenceUpdate
[11:51:41] [DiscordBot Info]      : 11:51:41 Gateway     Sent PresenceUpdate
[11:51:41] [DiscordBot Info]      : 11:51:41 Gateway     Sent PresenceUpdate
[11:51:42] [DiscordBot Info]      : 11:51:42 Gateway     Received Dispatch (INTERACTION_CREATE)
[11:51:45] [DiscordBot Info]      : 11:51:45 Gateway     A SlashCommandExecuted handler is blocking the gateway task.
[11:51:52] [DiscordBot Info]      : 11:51:52 Gateway     Sent Heartbeat
[11:52:11] [DiscordBot Info]      : 11:52:11 Gateway     Sent PresenceUpdate
[11:52:11] [DiscordBot Info]      : 11:52:11 Gateway     Sent PresenceUpdate

Yep, it's a plugin. The software it's a plugin for targets .NET Framework 4.8. Does Mono conflict with this?

Edit:
Mono isn't installed...
image

@Misha-133
Copy link
Member

Misha-133 commented Aug 11, 2023 •
edited

DNet used to have issues running on mono, so that can be one of the issues

You have a couple of ..... handler is blocking the gateway task
DNet runs events on the gateway thread. You should offload any time consuming code you have in event handlers (like API, DB calls, etc) to a separate task.
The simplest way to do this - using a

_ = Task.Run(async () => 
{
    
});

Wrap the whole handler with this => it should prevent the gateway from being blocked

@winglessraven
Copy link
Author

If I remove the event handlers that it says are blocking it goes back to the original problem, just hangs on 13:28:23 Gateway Connecting ApiClient. If I query the client ConnectionState it just returns Connecting.

Note: This is only specific to this server. On everything else I've tested on (Win & Linux but not aarch64) it progresses to Connected and then Ready.

@Misha-133
Copy link
Member

Well, I don't know then
In your latest log you also had some Received Dispatch (....) - it means that DNet successfully connected to gateway & managed to receive events from the API.
removing event handlers should not any affect on it not being able to establish a WS connection... ig it all comes to whatever .net framework runtime the app your plugin is for uses having some architecture related issues

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Misha-133 Misha-133

Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@winglessraven @Misha-133