Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve stability of virtual threads at scale #4995

Open
bentsherman opened this issue May 13, 2024 · 0 comments
Open

Improve stability of virtual threads at scale #4995

bentsherman opened this issue May 13, 2024 · 0 comments
Assignees

Comments

@bentsherman
Copy link
Member

See https://nextflow.slack.com/archives/C02T98A23U7/p1715351870737659

When a workflow publishes many files using S3-to-S3 copy, virtual threads are needed to maximize request throughput, since each publish task is simply waiting on an HTTP request and doesn't need to occupy an OS thread the whole time.

However, enabling virtual threads currently creates two problems:

  • The AWS Java SDK imposes a separate limit on the number of max HTTP connections (aws.client.maxConnections), and if this limit isn't also increased, Nextflow will likely fail with a message like Timeout waiting for connection from pool

  • Virtual threads have no queue size limit by default, but in reality we might still want to limit the number of in-flight HTTP requests so as to not overload the network

To this end, I propose the following improvements:

  • Allow virtual threads to be used with a queue size (similar to executor.queueSize) to control the number of concurrent publish tasks

  • Automatically increase the AWS connection limit to match the virtual threads queue size so that the user doesn't have to remember it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant