Nexus: Set workstation thread binding to none #5240
Conversation
|
Is it possible to test if it is OpenMPI in use and add the binding option if the answer is yes? |
|
A fair question for @jtkrogel My take is that since the current workstation class assumes you are using some flavor of MPI, we could have different variants for different MPI implementations (OpenMPI, MPICH, even no MPI [but set threads properly]). Hopefully this current change can be merged quickly if there is not a near ready-to-go better route. I'll note though that improvements to the Workstation class are not academic -- if we are to run the Nexus examples in the nightlies or even CI, efficiency will be key. |
|
Test this please |
|
@prckent nexus tests failing on sulfur |
|
But why those sulfur configurations? |
|
I discovered that some old but not too-old mpiruns don't take the thread binding settings. Rather than make assumptions, a better alternative might be to use environment variables to allow the user to build up an mpirun-type command, similar to how CMake does it. e.g. https://cmake.org/cmake/help/latest/module/FindMPI.html#usage-of-mpiexec . This would allow any MPI to be supported within the workstation class. "${MPIEXEC_EXECUTABLE} ${MPIEXEC_NUMPROC_FLAG} ${MPIEXEC_MAX_NUMPROCS} ${MPIEXEC_PREFLAGS} EXECUTABLE ${MPIEXEC_POSTFLAGS} ARGS" @jtkrogel Do you know if anyone has ever looked at doing this? |
|
Being addressed by #5255 to allow+document more flexibility in mpirun options. |
Proposed changes
The Nexus workstation configuration, e.g. ws16, implicitly assumes codes are built with OpenMPI for MPI runs. When also used with threading, the threads will be wrongly bound for our uses, giving poor performance. The binding is the default behavior of all recent OpenMPI releases and is easily verified in a system with hyperthreading enabled where the per process CPU% will show as 50% instead of the expected 100%, or in a 16 "core" run, 16 cores will not be used. Performance is consequently halved. Added --bind-to none, similar to what has already been done in some of the Supercomputer job definitions.
Open to changing to more optimal settings or moving the location of the setting, but for everyday workstation runs I think this is close enough and simple.
Noticed while testing #5214.
What type(s) of changes does this code introduce?
Does this introduce a breaking change?
What systems has this change been tested on?
nitrogen2, GCC14 + OpenMPI, standard nightly configuration.
Checklist