New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
for_each_init
calls init
several times per thread
#742
Comments
Your The high level of One option for you is to set a manual floor on the job granularity using In the future we might try to have more flexibility, like the static scheduler in #735. |
Thanks, this makes things a little clearer. The I'll do some experiments, but since the elements in |
I originally made the following test to determine whether all inits were dropped before use rayon::prelude::*;
use rayon::ThreadPoolBuilder;
use std::thread::sleep;
use std::sync::{Mutex,Arc};
use std::time::Duration;
struct InitData {
id: u64,
}
impl Drop for InitData {
fn drop(&mut self) {
println!("Dropping init {}.", self.id);
sleep(Duration::from_millis(10+self.id));
println!("Init {} dropped.", self.id);
}
}
fn main() {
let next_id = Arc::new( Mutex::new(0) );
ThreadPoolBuilder::new().num_threads(3).build().unwrap().install(|| {
println!("Calling for each init.");
(0u64..10).into_par_iter().for_each_init(
|| {
let mut l = next_id.lock().unwrap();
let id = *l;
*l += 1;
println!("Initializing init {}.", id);
sleep(Duration::from_millis(10));
println!("Init {} initialized.", id);
InitData { id }
}
, |init_data, item| {
println!("Processing {} with init {}.", item, init_data.id);
sleep(Duration::from_millis(10+init_data.id+item));
println!("Processed {} with init {}.", item, init_data.id);
}
);
println!("For each init returned.");
});
} Is this a bug or still just suboptimal behaviour? |
@chaueve as mentioned above, the adaptive splitting can be overly aggressive, and you may want to tune it manually. Unfortunately you can't use |
Here is my use case, I must process all elements of
ts: Vec<T>
, with a functionf(&mut T, &mut Buffer)
. The Buffer type is long to allocate an initialize, but its state after executingf
is guaranteed to be identical to the initial state. So my (single-threaded) loop looks like this:I would like to make this parallel. So I turned it to:
This runs faster (thanks rayon 😄), but I noticed that
allocate_and_init_buffer
is called more times than the number of threads in the thread pool... I'm guessing that the threads get "reset" every once in a while, making it necessary to call theinit
function again?Is there a way I can control how/when this "reset" occurs, in order to minimize the number of times
allocate_and_init_buffer
is called?The text was updated successfully, but these errors were encountered: