SSH cluster gets only one core running

[iagomosqueira]

I used to setup doFuture to run a foreach::%dopar% loop over another machine using

library(doFuture)
registerDoFuture()
plan(cluster, workers = rep("user@ip", 5))

and the process would be spawned over 5 cores.

It now appears to run on only one, as reported by htop in that machine.

I have tried directly calling

cl <- makeClusterPSOCK(rep("user@ip", 5), revtunnel = TRUE, outfile = "")
plan(cluster, workers = cl)

but the problem persists.

I have tested it on two different servers, with different Linux versions, but got the same result. This was working a couple of months ago.

Anything in the R session I should look for that could be causing it?

> sessionInfo()
R version 4.2.1 (2022-06-23)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 20.04.5 LTS

Matrix products: default
BLAS/LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.8.so

locale:
 [1] LC_CTYPE=en_GB.UTF-8       LC_NUMERIC=C               LC_TIME=en_GB.UTF-8        LC_COLLATE=en_GB.UTF-8     LC_MONETARY=en_GB.UTF-8    LC_MESSAGES=en_GB.UTF-8
 [7] LC_PAPER=en_GB.UTF-8       LC_NAME=C                  LC_ADDRESS=C               LC_TELEPHONE=C             LC_MEASUREMENT=en_GB.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats     graphics  grDevices datasets  utils     methods   base

other attached packages:
 [1] doFuture_0.12.2      future_1.30.0        foreach_1.5.2        mse_2.2.3.9015       testthat_3.1.6       FLBRP_2.5.9.9003     FLasher_0.7.1.9102   FLFishery_0.3.8.9002
 [9] ggplotFL_2.7.0.9102  ggplot2_3.4.0        FLCore_2.6.19.9023   iterators_1.0.14     lattice_0.20-45      SAMtool_1.5.1        MSEtool_3.6.1        snowfall_1.84-6.2
[17] snow_0.4-4           TMB_1.9.2            nvimcom_0.9-140      data.table_1.14.6    rmarkdown_2.20       devtools_2.4.5       usethis_2.1.6        colorout_1.2-2

loaded via a namespace (and not attached):
  [1] colorspace_2.1-0    deldir_1.0-6        ellipsis_0.3.2      rprojroot_2.0.3     corpcor_1.6.10      fs_1.6.0            rstudioapi_0.14     listenv_0.9.0
  [9] remotes_2.4.2       AAP_0.3.2           gsl_2.1-8           ggrepel_0.9.2       mvtnorm_1.1-3       fansi_1.0.4         codetools_0.2-18    splines_4.2.1
 [17] cachem_1.0.6        knitr_1.42          pkgload_1.3.2       png_0.1-8           stabledist_0.7-1    copula_1.1-2        FLa4a_1.8.3         shiny_1.7.4
 [25] compiler_4.2.1      assertthat_0.2.1    Matrix_1.5-3        fastmap_1.1.0       strucchange_1.5-3   cli_3.6.0           later_1.3.0         htmltools_0.5.4
 [33] prettyunits_1.1.1   tools_4.2.1         coda_0.19-4         gtable_0.3.1        glue_1.6.2          dplyr_1.1.0         Rcpp_1.0.10         vctrs_0.5.2
 [41] urca_1.3-3          nlme_3.1-162        progressr_0.13.0    vars_1.5-6          lmtest_0.9-40       xfun_0.37           stringr_1.5.0       globals_0.16.2
 [49] brio_1.1.3          ps_1.7.2            mime_0.12           miniUI_0.1.1.1      lifecycle_1.0.3     gtools_3.9.4        MASS_7.3-58.2       zoo_1.8-11
 [57] scales_1.2.1        promises_1.2.0.1    parallel_4.2.1      sandwich_3.0-2      RColorBrewer_1.1-3  memoise_2.0.1       pbapply_1.7-0       gridExtra_2.3
 [65] latticeExtra_0.6-30 stringi_1.7.12      triangle_1.0        desc_1.4.2          pcaPP_2.0-3         caTools_1.18.2      pkgbuild_1.4.0      rlang_1.0.6
 [73] pkgconfig_2.0.3     bitops_1.0-7        evaluate_0.20       purrr_1.0.1         patchwork_1.1.2     htmlwidgets_1.6.1   cowplot_1.1.1       processx_3.8.0
 [81] tidyselect_1.2.0    parallelly_1.34.0   magrittr_2.0.3      R6_2.5.1            gplots_3.1.3        generics_0.1.3      profvis_0.3.7       ADGofTest_0.3
 [89] DBI_1.1.3           pillar_1.8.1        withr_2.5.0         mgcv_1.8-41         abind_1.4-5         pspline_1.0-19      tibble_3.1.8        crayon_1.5.2
 [97] FLAssess_2.6.3      interp_1.1-3        KernSmooth_2.23-20  utf8_1.2.3          urlchecker_1.0.1    jpeg_0.1-10         grid_4.2.1          callr_3.7.3
[105] digest_0.6.31       xtable_1.8-4        numDeriv_2016.8-1.1 httpuv_1.6.8        stats4_4.2.1        munsell_0.5.0       sessioninfo_1.2.2

[HenrikBengtsson]

(I moved this from the doFuture issue tracker to the future discussion forum)

I don't think this has anything to do with doFuture per se, so let's make sure you do indeed get five independent workers. You can do this using:

> library(future)
> plan(cluster, workers = rep("user@ip", 5))
> futureSessionInfo()

At the end of the output, you should see 'Worker R session details:' listing five workers, and the last line should be:

Number of unique worker PIDs: 5 (as expected)

If you get that, then we know for sure that there are five workers running, and what you observe is due to something else.

[iagomosqueira]

Yes, it does report 5 workers being available

[...]
*** Backends
Number of workers: 5
[...]
Number of unique worker PIDs: 5 (as expected)

[HenrikBengtsson]

Good. Next, please confirm that you get five unique PIDs when you do:

library(doFuture)
registerDoFuture()
plan(cluster, workers = rep("user@ip", 5))
pids <- foreach(kk = 1:5) %dopar% { Sys.getpid() }
print(unlist(pids))

[iagomosqueira]

Yes, it works. I have also tried adding some of the arguments I use in the foreach call, like .combine and '.multicombine', and it still works

> print(unlist(pids))
[1] 74450 74526 74601 74676 74751

So what else could be a problem in the full call?

[HenrikBengtsson]

what else could be a problem in the full call?

So, I'm 99.999% sure that nothing in the future ecosystem has changed that causes your code going from running in parallel on multiple cores to a single core.

It could be that your code changed, or some of your dependencies changed. Another alternative could be that some type of CGroups policy has changed on both of your servers, e.g. a CPU Cgroups setting restricting all processes running on a single core. To test the latter, create a minimal example like the PID one above, but where each iteration also runs some heavy computations for ~30 seconds. That should help you narrow in on the real cause.

[iagomosqueira]

Thanks. I will keep trying to find the reason. If I launch one the the processes I can only see one CPU running, despite receiving back still the 5 PIDs.

I am also testing it on two home computers, so permissions is not the issue.

Thanks so much for your help