Issues with foreach and SGE

[fkgruber]

furrr works perfectly with future.batchtools. If you have a loop with 3 elements you get 3 jobs on the cluster:

library(furrr)
library(future.batchtools)
plan(batchtools_sge)
nothingness <- future_map(c(2, 2, 2), ~Sys.sleep(.x))

qstat

job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID 
-----------------------------------------------------------------------------------------------------------------
  14375 0.50500 jobb856749 fred         r     06/03/2023 22:23:03 all.q@ip-xxxec2.inte     1        
  14376 0.50500 job2b8fc0d fred         r     06/03/2023 22:23:03 all.q@ip-1xxx.ec2.inte     1        
  14377 0.50500 jobfdf28f1 fred         r     06/03/2023 22:23:03 all.q@ip-xx.ec2.inte     1        

With foreach, however, I only get 1 job:

library(foreach)
library(future)
library(furrr)
library(future.batchtools)
library(doFuture)

mu <- 1.0
sigma <- 2.0
registerDoFuture()
plan(batchtools_sge)
x %<-% {
  foreach(i = 1:3) %dopar% {
    Sys.sleep(3)
    set.seed(123)
    rnorm(i, mean = mu, sd = sigma)
  }
}

qstat

job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID 
-----------------------------------------------------------------------------------------------------------------
  14378 0.50500 job7472e4f fred         r     06/03/2023 22:26:03 all.q@ip-xxx.ec2.inte     1        

and when it return we get the following 2 strange warnings:

Warning messages: 1: executing %dopar% sequentially: no parallel backend registered 2: UNRELIABLE VALUE: Future ( ) unexpectedly generated random numbers without specifying argument 'seed'. There is a risk that those random numbers are not statistically sound and the overall results might be invalid. To fix this, specify 'seed=TRUE'. This ensures that proper, parallel-safe random numbers are produced via the L'Ecuyer-CMRG method. To disable this check, use 'seed=NULL', or set option 'future.rng.onMisuse' to "ignore". >

Why does it say there is no parallel backend registered when I'm running registerDoFuture()?

Alternatively, I tried %dofuture% instead of %dopar% but it still only generates 1 job.

x %<-% {
  foreach(i = 1:10) %dofuture% {
						   Sys.sleep(3)
						   set.seed(123)
						   rnorm(i, mean = mu, sd = sigma)
						 }
  }

f = futureOf(x)
resolved(f)
x

qstat

job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID 
-----------------------------------------------------------------------------------------------------------------
  14379 0.50500 job4b3cd22 fred         r     06/03/2023 22:28:48 all.q@ipxxx     1        

This time I only get the random number warning:

Warning message: UNRELIABLE VALUE: At least one of iterations 1-10 of the foreach() %dofuture% { … }, part of chunk #1 ( doFuture2-1 ), unexpectedly generated random numbers without declaring so. There is a risk that those random numbers are not statistically sound and the overall results might be invalid. To fix this, specify foreach() argument '.options.future = list(seed = TRUE)'. This ensures that proper, parallel-safe random numbers are produced via the L'Ecuyer-CMRG method. To disable this check, set option 'doFuture.rng.onMisuse' to "ignore". >

I also tried other options in foreach like .options.future = list(scheduling=1) but they don't seem to have any effect.

Is it possible that foreach somehow chunks all the iteration in one task? Or is something else not working.

Thanks
Fred

[scottkosty]

TLDR: I was hoping to save Henrik a bit of time but alas I failed.

Thanks for that nice minimal example.

Regarding the issue about the seed, you can read about the purpose of the warning in ?future under the seed argument. The warning even kindly suggests how to add it in your context, by giving the argument to specify to foreach(). However, when I tried that, I still got a warning.

Here is the code I ran, based on your code (note the addition of .options.future = list(seed = TRUE)):

library(foreach)
library(future)
library(furrr)
library(doFuture)

mu <- 1.0
sigma <- 2.0

plan(multisession)

x %<-% {
  foreach(i = 1:10, .options.future = list(seed = TRUE)) %dofuture% {
						   Sys.sleep(3)
						   set.seed(123)
						   rnorm(i, mean = mu, sd = sigma)
						 }
}

f = futureOf(x)
resolved(f)
x

However, I still get the following warning:

UNRELIABLE VALUE: Future (‘<none>’) unexpectedly generated random numbers without specifying argument 'seed'. There is a risk that those random numbers are not statistically sound and the overall results might be invalid. To fix this, specify 'seed=TRUE'. This ensures that proper, parallel-safe random numbers are produced via the L'Ecuyer-CMRG method. To disable this check, use 'seed=NULL', or set option 'future.rng.onMisuse' to "ignore". 

[scottkosty]

By the way, in your real example do you indeed want to set set.seed(123) inside the body, or was this just for debugging purposes? There are use cases (I have one) where you might want to do that, but I think in most cases you do not want to set the seed inside the body, especially after we figure out the seed argument issue.

[fkgruber]

you are right I was trying to get rid of that message about the random number. I was not planning on doing that.

[HenrikBengtsson]

I'm not at a computer right now, but you don't want to use %<-% here (for the same reason you don't use it with future_map()) - just use a regular <- assignment.

[HenrikBengtsson]

If you use %<-%, then that consumes the first layer of your future plan, and %dopar%/%dofuture% will use the second layer. See https://future.futureverse.org/articles/future-3-topologies.html for how to set nested plans (stragegies).

[fkgruber]

thanks.
I think what I was thinking is something a lot simpler.
I think I can do the following. I just have to do the combining on my own. This is non-blocking and correctly submits to the cluster.

library(future.batchtools)
library(listenv)
plan(batchtools_sge)

alltasks = listenv()
for(ii in 1:3){
  alltasks[[ii]]%<-%{
    Sys.sleep(2)
    rnorm(1, 0, 2)
  }
}

f = map(alltasks, futureOf)

for(ff in f)
  print(resolved(ff))

[fkgruber]

The only problem is I'm still getting that warning. Is this something I can safely ignore?
1: UNRELIABLE VALUE: Future ( <none> ) unexpectedly generated random numbers without specifying argument 'seed'. There is a risk that those random numbers are not statistically sound and the overall results might be invalid. To fix this, specify 'seed=TRUE'. This ensures that proper, parallel-safe random numbers are produced via the L'Ecuyer-CMRG method. To disable this check, use 'seed=NULL', or set option 'future.rng.onMisuse' to "ignore".

[fkgruber]

I see that you need to add %seed% 123 to the assignment to remove this message.

[HenrikBengtsson]

You should probably not set a fixed seed. It's better to just declare you're going to use random number generation, i.e. use %seed% TRUE.

Also, note that %<-% is just a fancy wrapper for future() with a delayed assignment calling value(). So, the barebone version is:

library(future.batchtools)
plan(batchtools_sge)

fs <- list()
for(ii in 1:3) {
  fs[[ii]] <- future({
    Sys.sleep(2)
    rnorm(1, 0, 2)
  }, seed = TRUE)
}

for (f in fs) {
  print(resolved(fs))
}