Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Compression cpu thread pool #4970

Merged
merged 4 commits into from May 11, 2024
Merged

Compression cpu thread pool #4970

merged 4 commits into from May 11, 2024

Conversation

fulmicoton
Copy link
Contributor

@fulmicoton fulmicoton commented May 10, 2024 •
edited 

Running small but cpu intensive tasks in a thread pool.

This PR removes usage of spawn_blocking for cpu intensive tasks.

The problem with spawn_blocking is that these tasks get scheduled on a
ever growing thread pool.

For instance, when the server is under load, the GZIP decompression of
payloads could considerably increase the load factor of quickwit,
possibly making it unresponsive to healthcheck.

This PR isolates the thread pool used for the searcher, and instantiates
a second generic thread pool dedicated to those short cpu-intensive tasks.

@fulmicoton fulmicoton force-pushed the compression-cpu-thread-pool branch 17 times, most recently from 6c07817 to 6d65351 Compare May 10, 2024 09:41
@fulmicoton fulmicoton requested a review from guilload May 10, 2024 09:41
@fulmicoton fulmicoton marked this pull request as ready for review May 10, 2024 09:41
@fulmicoton
Running small but cpu intensive tasks in a thread pool.
006eb73 
This PR removes usage of spawn_blocking for cpu intensive tasks.

The problem with spawn_blocking is that these tasks get scheduled on a
ever growing thread pool.

For instance, when the server is under load, the GZIP decompression of
payloads could considerably increase the load factor of quickwit,
possibly making it unresponsive to healthcheck.

This PR isolates the thread pool used for the searcher, and instantiates
a second generic thread pool dedicated to those short cpu-intensive tasks.
@fulmicoton fulmicoton force-pushed the compression-cpu-thread-pool branch from 6d65351 to 006eb73 Compare May 10, 2024 10:11
guilload
guilload approved these changes May 10, 2024
View reviewed changes
quickwit/quickwit-common/src/thread_pool.rs Outdated Show resolved Hide resolved
quickwit/quickwit-common/src/thread_pool.rs Outdated Show resolved Hide resolved
quickwit/quickwit-common/src/thread_pool.rs Outdated Show resolved Hide resolved
quickwit/quickwit-common/src/thread_pool.rs Outdated Show resolved Hide resolved
quickwit/quickwit-common/src/thread_pool.rs Outdated Show resolved Hide resolved
quickwit/quickwit-common/src/thread_pool.rs Outdated Show resolved Hide resolved
quickwit/quickwit-common/src/thread_pool.rs Outdated Show resolved Hide resolved
quickwit/quickwit-search/src/fetch_docs.rs Outdated Show resolved Hide resolved
quickwit/quickwit-search/src/lib.rs Outdated Show resolved Hide resolved
quickwit/quickwit-common/src/thread_pool.rs
let span = tracing::Span::current();
let gauge = self.active_threads_gauge.clone();
let (tx, rx) = oneshot::channel();
self.thread_pool.spawn(move || {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Metric suggestion: number of queued "requests"

@fulmicoton @guilload
Apply suggestions from code review
327a256 
Co-authored-by: Adrien Guillo <adrien@quickwit.io>
@fulmicoton fulmicoton force-pushed the compression-cpu-thread-pool branch from f9f2c88 to 327a256 Compare May 11, 2024 02:05
@fulmicoton fulmicoton merged commit 770d8a7 into main May 11, 2024
4 checks passed
@fulmicoton fulmicoton deleted the compression-cpu-thread-pool branch May 11, 2024 02:34
fulmicoton added a commit that referenced this pull request May 13, 2024
@fulmicoton @guilload
Compression cpu thread pool (#4970)
1dcf943 
* Added a thread pool for ingest decompression

* Running small but cpu intensive tasks in a thread pool.

This PR removes usage of spawn_blocking for cpu intensive tasks.

The problem with spawn_blocking is that these tasks get scheduled on a
ever growing thread pool.

For instance, when the server is under load, the GZIP decompression of
payloads could considerably increase the load factor of quickwit,
possibly making it unresponsive to healthcheck.

This PR isolates the thread pool used for the searcher, and instantiates
a second generic thread pool dedicated to those short cpu-intensive tasks.

* Apply suggestions from code review

Co-authored-by: Adrien Guillo <adrien@quickwit.io>

---------

Co-authored-by: Adrien Guillo <adrien@quickwit.io>
fulmicoton added a commit that referenced this pull request May 14, 2024
@fulmicoton @guilload
Compression cpu thread pool (#4970)
be6c7ad 
* Added a thread pool for ingest decompression

* Running small but cpu intensive tasks in a thread pool.

This PR removes usage of spawn_blocking for cpu intensive tasks.

The problem with spawn_blocking is that these tasks get scheduled on a
ever growing thread pool.

For instance, when the server is under load, the GZIP decompression of
payloads could considerably increase the load factor of quickwit,
possibly making it unresponsive to healthcheck.

This PR isolates the thread pool used for the searcher, and instantiates
a second generic thread pool dedicated to those short cpu-intensive tasks.

* Apply suggestions from code review

Co-authored-by: Adrien Guillo <adrien@quickwit.io>

---------

Co-authored-by: Adrien Guillo <adrien@quickwit.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@guilload guilload guilload approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@fulmicoton @guilload