Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Possible memory leak in Serenity 0.12 and Songbird 0.4. #205

Open
abokado0107 opened this issue Nov 27, 2023 · 4 comments
Open

Possible memory leak in Serenity 0.12 and Songbird 0.4. #205

abokado0107 opened this issue Nov 27, 2023 · 4 comments
Labels
bug Something isn't working

Comments

@abokado0107
Copy link

abokado0107 commented Nov 27, 2023

Songbird version: (version)
0.4.0-rc

Rust version (rustc -V): (version)
1.74.0

Serenity/Twilight version: (version)
Serenity 0.12.0-rc

Description:
When a large number of small audio files are played back, CPU utilization increases and memory usage explodes after a certain period of time.
image

Steps to reproduce:
Rust installation is omitted.

$ sudo apt update && sudo apt install libopus-dev build-essential autoconf automake libtool m4 pkg-config libssl-dev -y
$ git clone https://github.com/abokado0107/tts-memory-leak
$ cd tts-memory-leak
$ cargo build --release
  1. Build the verification repository.
  2. Enter token in .env and run it.
  3. Type !join on Discord to invite the BOT to the voice channel.
  4. After a few days, the memory usage will explode and crash in a short time.
    It crashes faster if you have it connected to several VCs at the same time.

Verification environment
CPU: ARM Ampere
CPU: 2 core
RAM: 4 GB
OS: Ubuntu 20.04
IaaS: Oracle Cloud Infrastructure
Repositories for verification: https://github.com/abokado0107/tts-memory-leak

@abokado0107 abokado0107 added the bug Something isn't working label Nov 27, 2023
@FelixMcFelix
Copy link
Member

I've been running my own bot continuously for a day or so which also plays back many small audio files; I'm not seeing my CPU or memory substantially increase over time in a single server. I'll keep you posted if anything changes on that front.

@abokado0107
Copy link
Author

I don't know if this is directly related to this issue, but after several days of running my application, I am accumulating tasks that never finish.
This image is displayed by tokio-console.
image

I am not familiar with the structure of serenity or songbird, but it seems that tasks generated by songbird's task.rs and serenity's tokio.rs remain relatively well.

@FelixMcFelix
Copy link
Member

I need a little more context; one of each driver/scheduler/task.rs:112 is expected for every Call without live audio playing, If there's a big difference between those two quantities then that would indicate a task leak. If you're joining/creating Calls but not removing, this would be expected, otherwise I'm very interested.

@abokado0107
Copy link
Author

abokado0107 commented Dec 2, 2023

Thanks for letting me know.
I checked in the production environment and Call was able to remove properly.
I output the following counts to the debug log and the call was increasing and decreasing properly.

if let Some(manager) = songbird::get(ctx).await {
    let call_count = manager.iter().count();
}

image

As for the scale of the application, the number of simultaneous VC connections at 30k guilds is about 2k at peak time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants