Plan multicore workers fall off near end of function's completion

[rafev]

Hello all!
I have a function I wrote that executes several hundred thousand operations in forked parallel [plan(multicore)] and tracks progress using progressr. It runs for about a week and works quite well on my M1 mac in base R, but I noticed at around 80% the number of backend workers begins to fall off. Before that point all eight workers are chugging along fine and good CPU utilization, but at 80% completion it drops to five workers, and at above 90% complete it drops to just two workers. Why is this happening? It severely slows down the end of my functions operation time.

Below is a snippet of my function and code:

my_1way.mods = function(modules_being_tested) {
p = progressor(along = modules_being_tested)
foreach (i = modules_being_tested) %dopar% {

set.seed(32)
Net.fit = train(x = train_grp[, module_df[module_df$colors == i, 1]],
y = as.vector(train_pheno[, ..covariate_interest])[[1]],
method = "glmnet",
trControl = cctrl1,
metric = "RMSE",
tuneGrid = expand.grid(.alpha = seq(.05, 1, length = 15), .lambda = seq(0.01, 1, 0.005)))

p(sprintf("i=%s", i))

my_list[[i]] = Net.fit
}
}

registerDoFuture()
plan(multicore)

options(future.globals.maxSize = 943718400) # changes the maxsize for globals to be 900 MiB for future objects
options(doFuture.rng.orMisuse = "ignore")   # avoid error message for random number generation being used within function

#Set up progress bar
handlers(global = TRUE)
handlers("cli")

##-- Running single module loop
start.time = Sys.time()
my_list = my_1way.mods(modules_being_tested)
end.time = Sys.time()

#time spent processing loop
difftime(end.time, start.time)

Any help would be greatly appreciated!

[HenrikBengtsson]

See Section 'Load balancing ("chunking")' in https://dofuture.futureverse.org/reference/registerDoFuture.html The default is .options.future = list(scheduling = 1.0), which will chunk up the iterations into equal chunks and then launch them on the parallel workers. If some of these chunks finish sooner than others, then those workers will just sit idle until the other workers finishing up their chunks. If you use .options.future = list(scheduling = 5.0), each worker will on average get 5 chunks, which means that if some chunks take longer than others, then the workers who finish quickly will be able to work on additional chunks. This will give a better load balancing.