Expected behavior

SRW contains the config variable OMP_NUM_THREADS_RUN_FCST which should activate threading when set greater than 1 (default in SRW is 2, so should be every run unless specified otherwise).

Current behavior

Threading is not activated for the weather model, no matter what OMP_NUM_THREADS_RUN_FCST is set to. You can observe this by opening the run_fcst log file for any SRW run; here is just one example from the WE2E test grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_WoFS_v0:

* . * . * . * . * . * . * . * . * . * . * . * . * . * . * . * . * . * . * . * .
     PROGRAM ufs-weather-model HAS BEGUN. COMPILED       0.00     ORG: np23
     STARTING DATE-TIME  OCT 14,2024  17:32:13.156  288  MON   2460598
...
...
atmosphere_init: current_time_seconds =       0.0
Using n_split from the namelist: 005
 Off center implicit scheme param=   1.000000
  p_fac=  0.1000000
Using n_sponge : 024
Using non_ortho :       T
 Starting PEs :           25
 Starting Threads :            1
 Cubed-sphere, single face domain decomposition
whalo =    3, ehalo =    3, shalo =    3, nhalo =    3
...

Note the line "Starting Threads : 1", which indicates only a single thread is being used, despite us requesting 2.

Machines affected

Confirmed on Orion, Hera, and Derecho, but this should affect all machines.

Steps To Reproduce

1. Run any SRW test that contains a run_fcst task
2. View the run_fcst log file, and confirm that threading is not being used, despite OMP_NUM_THREADS_RUN_FCST=2 being the default

Detailed Description of Fix

I honestly don't know how to fix this. I am confused by the latest instructions on how to set OMP_NUM_THREADS for the weather model. Currently, the app sets the threading value in the model_configure file (atmos_nthreads: 1, but this variable has been deprecated for years (as a side note, so has PE_MEMBER01, which is also still being set) so the setting has no effect. However, if I follow the instructions in the weather model users guide (setting ATM_omp_num_threads: 2 in ufs.configure), I get behavior that I don't understand from the model: it seems to be using way fewer MPI tasks than expected, and so fails with a corresponding error message:

* . * . * . * . * . * . * . * . * . * . * . * . * . * . * . * . * . * . * . * .
     PROGRAM ufs-weather-model HAS BEGUN. COMPILED       0.00     ORG: np23
     STARTING DATE-TIME  OCT 14,2024  17:23:00.856  288  MON   2460598
...
...
atmosphere_init: current_time_seconds =       0.0
Using n_split from the namelist: 005
 Off center implicit scheme param=   1.000000
  p_fac=  0.1000000
Using n_sponge : 024
Using non_ortho :       T
 Starting PEs :           11
 Starting Threads :            2

FATAL from PE     3: mpp_domains_define.inc: not all the pe_end are in the pelist


FATAL from PE     1: mpp_domains_define.inc: not all the pe_end are in the pelist


FATAL from PE     2: mpp_domains_define.inc: not all the pe_end are in the pelist


FATAL from PE     4: mpp_domains_define.inc: not all the pe_end are in the pelist
...
...

Note the entry Starting PEs : 11, which doesn't make any sense to me with the settings provided (identical to the first test except for setting ATM_omp_num_threads: 2 in ufs.configure) and with 30 cores allocated.

Clearly there is something I am missing, but I don't have the time to dig deeper into this. However, this is something that should be solved sooner rather than later, as it likely represents a huge waste of resources due to not using threads that we expect to use (and likely impacts any external groups using recent SRW code to run the weather model).

Additional Information (optional)

I believe this bug was introduced in #1050, but I can not confirm that that is the case.