[help] tar_map_rep() generates the same random numbers across reps when using rep_workers argument

[solmos]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

In the following reproducible example I use tar_map_rep() to simulate many data sets (using rnorm()) for several combination of scenarios. When I don't use rep_workers I get the expected result: each data set (uniquely defined by tar_group, tar_batch and tar_rep) has different random variables.

targets::tar_dir({
  targets::tar_script({
    sim_data <- function(n, mu, sigma) {
      data.frame(x = rnorm(n, mu, sigma), y = rnorm(n, mu, sigma))
    }
    tarchetypes::tar_map_rep(
      data_sim,
      command = sim_data(100, mu, sigma),
      values = expand.grid(mu = c(2, 10), sigma = c(0.5, 1)),
      batches = 2,
      reps = 30
    )
  })
  targets::tar_make()
  targets::tar_load(data_sim)
  # Get two of the simulated data sets
  data1 <- subset(data_sim, tar_group == 1 & tar_batch == 1 & tar_rep == 1)
  data2 <- subset(data_sim, tar_group == 1 & tar_batch == 1 & tar_rep == 2)
  cat("data1_seed: ", unique(data1$tar_seed), "\n")
  cat("data2_seed: ", unique(data2$tar_seed), "\n")
  identical(data1$x, data2$x)
})
#> data1_seed:  -536056082 
#> data2_seed:  -1437990038 
#> [1] FALSE

However when I use the rep_workers in tar_map_rep() some replicated data sets contain the same random vectors (even though the seed seems to be different).

targets::tar_dir({
  targets::tar_script({
    sim_data <- function(n, mu, sigma) {
      data.frame(x = rnorm(n, mu, sigma), y = rnorm(n, mu, sigma))
    }
    tarchetypes::tar_map_rep(
      data_sim,
      command = sim_data(100, mu, sigma),
      values = expand.grid(mu = c(2, 10), sigma = c(0.5, 1)),
      batches = 2,
      reps = 30,
      rep_workers = 2
    )
  })
  targets::tar_make()
  targets::tar_load(data_sim)
  data1 <- subset(data_sim, tar_group == 1 & tar_batch == 1 & tar_rep == 1)
  data2 <- subset(data_sim, tar_group == 1 & tar_batch == 1 & tar_rep == 2)
  cat("data1_seed: ", unique(data1$tar_seed), "\n")
  cat("data2_seed: ", unique(data2$tar_seed), "\n")
  identical(data1$x, data2$x)
})
#> data1_seed:  -536056082 
#> data2_seed:  -1437990038 
#> [1] TRUE

Is this the expected behavior of rep_workers?

This issue was hard to debug because even when I removed the rep_workers argument from tar_map_rep(), the pipeline was skipping the target since it was using the same batches * reps. It was only until I looked at the data that I found the problem.

packageVersion("targets")
#> [1] ‘1.3.0’
packageVersion("tarchetypes")
#> [1] ‘0.7.8’
sessionInfo()
#> R version 4.2.2 (2022-10-31)
#> Platform: aarch64-apple-darwin20 (64-bit)
#> Running under: macOS Ventura 13.5.2
#
#> Matrix products: default
#> BLAS:   /Library/Frameworks/R.framework/Versions/4.2-arm64/Resources/lib/libRblas.0.dylib
#> LAPACK: /Library/Frameworks/R.framework/Versions/4.2-arm64/Resources/lib/libRlapack.dylib
#
#> locale:
#> [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
#
#> attached base packages:
#> [1] stats     graphics  grDevices datasets  utils     methods   base     
#
#> loaded via a namespace (and not attached):
#>  [1] pillar_1.9.0      compiler_4.2.2    highr_0.9         tools_4.2.2       targets_1.3.0    
#>  [6] digest_0.6.33     evaluate_0.18     lifecycle_1.0.3   tibble_3.2.1      pkgconfig_2.0.3  
#> [11] rlang_1.1.1       reprex_2.0.2      igraph_1.3.5      cli_3.6.1         yaml_2.3.7       
#> [16] xfun_0.35         fastmap_1.1.1     withr_2.5.0       dplyr_1.1.2       knitr_1.41       
#> [21] generics_0.1.3    fs_1.6.3          vctrs_0.6.3       tidyselect_1.2.0  glue_1.6.2       
#> [26] data.table_1.14.8 R6_2.5.1          processx_3.8.0    fansi_1.0.3       base64url_1.4    
#> [31] rmarkdown_2.25    callr_3.7.3       magrittr_2.0.3    backports_1.4.1   ps_1.7.3         
#> [36] codetools_0.2-18  htmltools_0.5.6   renv_1.0.0        utf8_1.2.2       

[wlandau]

Thanks for reporting this. rep_workers relies on furrr::future_map(), which has its own opinions about setting seeds. tarchetypes tries to set the correct seed inside future_map(), but it looks like something is preventing this. I think it is an issue with furrr:

library(future)
library(furrr)
plan(sequential)
future_map_dfr(
  .x = seq_len(2L),
  .f = function(seed) {
    set.seed(seed)
    tibble::tibble(user_seed = seed, rng_state = digest::digest(.Random.seed))
  },
  .options = furrr::furrr_options(seed = NULL)
)
#> # A tibble: 2 × 2
#>   user_seed rng_state                       
#>       <int> <chr>                           
#> 1         1 f659f2cdd57f16488358b216b60e824a
#> 2         2 f659f2cdd57f16488358b216b60e824a

Even stranger, the problem goes away when I run this example inside reprex::reprex().

[wlandau]

I see discussions at DavisVaughan/furrr#251 and https://stackoverflow.com/questions/75521357/ensure-reproducibility-across-purrrmap-and-furrrfuture-map. The discussion at ropensci/targets#1139 and the chapter at https://books.ropensci.org/targets/random.html shows what targets is trying to do.

[wlandau]

I submitted DavisVaughan/furrr#265, and we will see where it goes.

[wlandau]

In 0a112b5, I switched from furrr to parallel to implement rep_workers. I think this solves the issue for tarchetypes.

[solmos]

@wlandau, does this mean that we should avoid using furrr to parallelize within a target?

[wlandau]

I am not sure if DavisVaughan/furrr#265 extends to every case, but if your work relies on pseudo-random number generation, it might be safer to use something like parallel::parLapply() for the time being. You might be okay though if you are using an earlier version of future or furrr and the reprex I posted in DavisVaughan/furrr#265 does not give you identical results across tasks.