‘error reading from connection’. Post-mortem diagnostic: No process exists with this PID, i.e. the localhost worker is no longer alive.

[anjelinejeline]

Hello,
I have trouble parallelizing my code,
This is the error I got
…
Error in unserialize(node$con) :
MultisessionFuture (doFuture2-1) failed to receive message results from cluster RichSOCKnode #1 (PID 161140 on localhost ‘localhost’). The reason reported was ‘error reading from connection’. Post-mortem diagnostic: No process exists with this PID, i.e. the localhost worker is no longer alive.
…
It seems that it has already occurred before see #474
I tried both multicore and multisession but none of them works

Can you help me to sort this out?

Thank you
Angela

My session info

…

sessionInfo()
R version 4.3.2 (2023-10-31)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 22.04.3 LTS

Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.20.so; LAPACK version 3.10.0

locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C LC_TIME=en_IE.UTF-8 LC_COLLATE=en_US.UTF-8 LC_MONETARY=en_IE.UTF-8
[6] LC_MESSAGES=en_US.UTF-8 LC_PAPER=en_US.UTF-8 LC_NAME=C LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_IE.UTF-8 LC_IDENTIFICATION=C

time zone: Europe/Rome
tzcode source: system (glibc)

attached base packages:
[1] stats graphics grDevices utils datasets methods base

other attached packages:
[1] progressr_0.14.0 doFuture_1.0.0 future_1.33.0 foreach_1.5.2 glue_1.6.2 sfheaders_0.4.3
[7] maptools_1.1-8 sp_1.6-0 spatstat_3.0-7 spatstat.linnet_3.1-3 spatstat.model_3.2-8 rpart_4.1.21
[13] spatstat.explore_3.2-5 nlme_3.1-163 spatstat.random_3.2-2 spatstat.geom_3.2-7 spatstat.data_3.0-3 sf_1.0-14
[19] ggplot2_3.4.4 tmap_3.3-4 dplyr_1.1.4 purrr_1.0.2

loaded via a namespace (and not attached):
[1] tidyselect_1.2.0 viridisLite_0.4.2 farver_2.1.1 fastmap_1.1.1 leaflet_2.2.1 XML_3.99-0.15 digest_0.6.33
[8] lifecycle_1.0.4 terra_1.7-55 magrittr_2.0.3 compiler_4.3.2 progress_1.2.2 rlang_1.1.2 tools_4.3.2
[15] utf8_1.2.4 prettyunits_1.2.0 htmlwidgets_1.6.2 classInt_0.4-10 RColorBrewer_1.1-3 abind_1.4-5 KernSmooth_2.23-22
[22] withr_2.5.2 foreign_0.8-85 leafsync_0.1.0 grid_4.3.2 polyclip_1.10-6 fansi_1.0.5 e1071_1.7-13
[29] leafem_0.2.3 colorspace_2.1-0 globals_0.16.2 scales_1.2.1 iterators_1.0.14 spatstat.utils_3.0-4 dichromat_2.0-0.1
[36] cli_3.6.1 crayon_1.5.2 generics_0.1.3 rstudioapi_0.15.0 future.apply_1.11.0 tmaptools_3.1-1 DBI_1.1.3
[43] proxy_0.4-27 splines_4.3.2 stars_0.6-4 parallel_4.3.2 base64enc_0.1-3 vctrs_0.6.4 Matrix_1.6-3
[50] hms_1.1.3 tensor_1.5 listenv_0.8.0 crosstalk_1.2.0 units_0.8-4 goftest_1.2-3 parallelly_1.36.0
[57] lwgeom_0.2-13 codetools_0.2-19 gtable_0.3.4 deldir_1.0-9 raster_3.6-26 munsell_0.5.0 tibble_3.2.1
[64] pillar_1.9.0 htmltools_0.5.7 R6_2.5.1 lattice_0.22-5 png_0.1-7 class_7.3-22 Rcpp_1.0.11
[71] spatstat.sparse_3.0-3 mgcv_1.9-0 pkgconfig_2.0.3
…

[anjelinejeline]

I also found this answer https://stackoverflow.com/questions/72942390/error-receiving-results-from-r-future-running-on-rstudio-server-on-slurm

I have run the code using the terminal so I guess that the version of R studio is not a problem .. but for information the version is RStudio
2023.06.2 Build 561

with regard to the file sizes: the error message says The total size of the 9 globals exported is 1.01 MiB. is this size a problem?

The last part of the error was a warning

In addition: Warning message:
In mccollect(jobs = jobs, wait = TRUE) :
1 parallel job did not deliver a result
Execution halted

*** caught segfault ***
address 0x562a84ec784a, cause 'memory not mapped'

[HenrikBengtsson]

Hello, I won't be able to respond to this right now, but I'll migrate this issue to the GitHub Discussions, because this is most likely not a bug.

[anjelinejeline]

@HenrikBengtsson thank you,
could you please have a look at the code when you have time?
I really do not understand how to fix it ..
I really would like to understand what is the issue and fix it soon ...
The parallelization works very well in case of a simple task (e.g. works very well with the code ParallelTemplate_doFuture.R I created to test the package and my system environment) .

When I define a more complex function the parallel strategy does not work ..
Could you please run the code Kernelparallel.R
I really appreciate your help

data.zip

Angela

[HenrikBengtsson]

@anjelinejeline , I took a quick look at your Kernelparallel.R script (pasted below).

It could be that you run out of memory. If that happens, your operating system might start terminating processes to rescue itself. So, if one of the parallel workers is killed this way, you get that error. I tried to run your script on my 32-GiB laptop, and it quickly reached 32 GiB of RAM, when I decided to kill it manually.

1. Have you monitored the memory use when you run this?
2. Have you verified that it works even with a single parallel worker? Try with plan(multicore, workers = I(1)) or plan(multisession, workers = I(1)). If that works, try with workers = 2, and so on. If it fails when workers is too large, that suggests you're running out of memory.

Also, what are you intending to return from each foreach() iteration? As it is written now (see below), it looks like different things are returned depending on nrow(outIntersect>1).

# Load packages ----

library("purrr") # Data manipulation and functional programming tools
library("dplyr") # Data manipulation
library("tmap") # World shapefile
library("ggplot2") # Plot
library("sf") # Work with spatial data
library("spatstat") # Compute Kernel 
library("maptools") # To convert density im into raster 
library("sfheaders") # Convert sf to df 
library("glue") # String manipulation 
library("foreach") # Parallelization
library("doFuture") # Parallelization 
library("progressr") # Progress bar


# Compute spatial Kernel for each cell within the buffer that intersects the points -----
# Load the data 

# BW

bw<-readRDS("bw.RDS")


# Buffer grid 
grid_intersection<-st_read("data/buffer/buffergridWHO.shp")

# Adjacent matrix 

adjList<-readRDS("data/grid/adjList.RDS")



# DF

df<-read.csv("df.csv")[,-1]
df<-df |> 
  filter(start %in% c(1975:2020)) # Filter by years

# Transform  into sf 
df <-st_as_sf(df, coords = c("x","y"), 
                           crs="+proj=moll +lon_0=0 +x_0=0 +y_0=0")


cellList<-grid_intersection |> 
  group_split(IDs) |> 
  setNames(sort(unique(grid_intersection$IDs)))

# Try out with two cells
please<-list(cellList[[25]],cellList[[26]])

# Set up the function to apply to each cell 

KernelCell<-function(mylist) {
  
  p <- progressor(steps=length(mylist)) # progress bar
  
  foreach(i=seq_along(mylist), 
          .options.future = list(packages =c("sf", "purrr","spatstat", "progressr"),seed = TRUE)) %dofuture% {
            
            p(sprintf("i=%g", i)) # Print progress 
            
            # Target cell
            targetCell<-mylist[[i]]
            st_crs(targetCell)<-"+proj=moll +lon_0=0 +x_0=0 +y_0=0" # Assign crs
            # Points within the cell 
            
            outIntersect<-st_intersection(st_make_valid(df),
                                          st_make_valid(targetCell))
            outIntersect<-sf_to_df(outIntersect) # Convert to df 
            
            if(nrow(outIntersect>1)){ # If there's no points there will be 1 observation with NA, if points do intersect the observations will be >1
              
              # Neighboring cells
              
              targetID<-targetCell$IDs # ID of the target cell to find its neighboring cells
              neighCellsIdx<-pluck(adjList[targetID]) |> list_c()  # Find the idx of the neighboring cells
              neighCells<-cellList[neighCellsIdx]|> list_rbind() # Extract the neighboring cells 
              neighCells<-st_as_sf(neighCells, crs="+proj=moll +lon_0=0 +x_0=0 +y_0=0" ) # Transform into sf 
              
              # All cells: target + its neighboring cells 
              allCells<-rbind(st_make_valid(targetCell), 
                              neighCells) # Create a sf dataframe
              win<-as.owin(allCells) # Extent to be passed to the ppp
              
              # Transform into ppp and jitter 
              out_ppp<-ppp(outIntersect$x,outIntersect$y,window=win)
              out_jitter<- rjitter(out_ppp, retry=TRUE, nsim=1, drop=TRUE)
              
              Kernel_ppp<-density.ppp(out_jitter, sigma=bw,edge=T, eps=1000)

              
              
            }
            
            
          }
  
}

# Enable progress bar 

handlers(global = TRUE)
handlers("progress")


# Run the function in parallel 
# Parallel strategy 
# N.B. the maximum number of parallel processes running will be decided by availableCores(), since in both cases the evaluation is done on the local machine.

plan(multicore) # FORK strategy on Linux
# plan(multisession) # Windows 


# try out with two cells
system.time({densities.ppp<-KernelCell(please)}) 


# change the plan to sequential
plan(sequential)

[anjelinejeline]

It was definetely an issue with the memory ..
Anyway I am still confused about the parallelization with doFuture as it turns out that the purrr::map is faster and more efficient sometimes .. I also do not know why the progress bar does not always show up ... do you have any ideas of the reason?

[avsdev-cw]

@HenrikBengtsson Before I open another ticket, have you investigated any further into causes of this? I've seen the ticket #474 but that doesn't really solve my issue as I am already using the multisession plan. My issue is also slightly different in that in the short term, there is no problem with the futures, they run and can do the processing consistently. The issue we are seeing is that if the application is left running (in my case it is part of a shiny server application) for a long period (days/weeks), eventually the worker processes seem to vanish and I get the error:

Warning: Error in : MultisessionFuture (<none>) failed to call grmall() on cluster RichSOCKnode #1 (PID 29506 on localhost ‘localhost’). The reason reported was ‘error reading from connection’. Post-mortem diagnostic: No process exists with this PID, i.e. the localhost worker is no longer alive.

[HenrikBengtsson]

Before I open another ticket, have you investigated any further into causes of this?

I just commented on this particular case/script in #705 (reply in thread).

If that doesn't give you ideas how to troubleshoot, please open a new Discussion, because these type of problems are most likely not something specific to the Futureverse, and often case-by-case specific.

[avsdev-cw]

Thanks for the reply. It looks like I might need to open a new discussion but I'll drop a bit further detail here.

In my instance, it shouldn't be a memory issue. The application is running (in an LXC container) on a server with ~10% memory utilization. No other containers exhibit any issues with process counts or memory kills. For the most part, the container is idle for long periods of time, with the occasional request made. When the service is restarted, I can make as many requests as I like without issue. I can even continue to do requests over the next few days. The issue seems to only occur over a longer period, regardless of if any requests are initially made or not.

I am wondering if perhaps I would be better off switching to something which can make the threads/background environments on the fly rather than have them spun up and idle. Is there a back-end/option capable of doing this?