Send tasks to free processes to allow nested futures

[bakaburg1]

Hello,

Probably there's a very good reason why this has not been done yet but I'll try my luck:
The future documentation says that it automatically switches to sequential on nested future calls, to protect from out-of-control spawning of processes.

I was thinking, is it not possible to keep the number of processes fixed and just send them the tasks as soon a process is free?

for example:

library(future)
library(furrr)
library(tictoc)

plan(multisession(workers = 5))

tic()
future_map(1:2, \() {
    future_map(1:4, ~ Sys.sleep(1))
})
toc()

This code takes 4 seconds to run because the inner level is sequential. What if the allocation continues for nested futures using the already allocated processes, so that tasks 1.1, 1.2, 1.3, 1.4, and 2.1 are processed in parallel, and then 2.2, 2.3, and 2.4 are sent to the processes as soon as they get free? With this approach the processing should take ~ 2 sec.
This implementation should avoid the risk of recursive generation of processes. Of course, keeping track of resolutions and returned values could be tough.

[sounkou-bioinfo]

Reference here https://henrikbengtsson.github.io/talk-user2024-future.mirai/#/nested-parallelization-with-some-care indicating using the Mirai back-end allows nested parallelization