MILC with QUDA

These instructions are intended to be a quick start guide to getting MILC running with GPUs using the QUDA library.

Obtaining and compiling QUDA

You can obtain QUDA using the following:

git clone https://github.com/lattice/quda.git

At the time of writing, the current stable release of QUDA is the 1.0 pre-release, located in the release/1.0.x branch. Further improvements are merged into the develop branch (which is the git default) so if you want to live on the bleeding edge, you can use this branch, else use the 1.0 branch.

cd quda
git checkout release/1.0.x
cd ..

QUDA uses cmake to set compilation options. For running with HISQ fermions, e.g., the su3_rhmc_hisq test that is commonly used in MILC, you should do something like the following

mkdir build
cd build
cmake ../quda -DQUDA_GPU_ARCH=sm_70 -DQUDA_DIRAC_DEFAULT_OFF -DQUDA_DIRAC_STAGGERED=ON -DQUDA_FORCE_HISQ=ON -DQUDA_FORCE_GAUGE=ON -DQUDA_BUILD_SHAREDLIB=ON -DQUDA_QMP=ON -DQUDA_QIO=ON -DQUDA_DOWNLOAD_USQCD=ON
cmake .

The final cmake command above is often required to ensure that cmake completes its configuration fully, and is a known issue. Above, we implicitly assume that the CUDA and MPI compilers are present in the $PATH. Here we are setting the the GPU architecture to sm_70 which corresponds to Volta. Choices include:

• sm_35 for Kepler (Tesla K20 / K40 / K80)
• sm_60 for Pascal (Tesla P100, Quadro GP100)
• sm_70 for Volta (Tesla V100, Quadro V100)
• sm_80 for Ampere (NVIDIA A100)

Here we are disabling unnecessary parts of QUDA when used with MILC, in order to reduce compilation time. The final three arguments concern the installation of the USQCD companion libraries QMP and QIO. QUDA can automate their download and installation, and that is what we have enabled here.

Then finally to build QUDA, you should use a parallel build as QUDA can take a long time to build,

make -j N

where N is the number of cores / threads that the compilation node has. We typically recommend setting this to the number of hardware threads (e.g., hyperthreads) in the system. If you have set an install path when running cmake, then to complete the installation

make install

Finally note that when building with OpenMPI 4.x and above, due to the use of the deprecated MPI_Type_struct, QMP will fail to build unless OpenMPI has been configured with the MPI-1 compatibility option --enable-mpi1-compatibility. The solution is to either enable this option in the OpenMPI build or trivially edit the QMP source code to change the single occurrence of MPI_Type_struct to MPI_type_create_struct in usqcd/src/QMP/lib/mpi/QMP_mem_mpi.c. Fixing this issue in QMP is tracked here.

Obtaining and compiling MILC

For use with QUDA we recommend the present develop branch of MILC. This enables the maximum benefit of QUDA acceleration.

git clone https://github.com/milc-qcd/milc_qcd.git
cd milc_qcd
git checkout develop

To aid compilation of MILC with QUDA support, there is a provided helper script for the su3_rhmd_hisq application ks_imp_rhmc/compile_su3_rhmd_hisq_quda.sh. Editing this script as appropriate and executing from its directory should result in a full build of MILC with QUDA acceleration for the desired application. For a standard build the important settings are CUDA_HOME, QUDA_HOME, QIOPAR and QMPPAR. Note QUDA_HOME should point to the build or install directory for QUDA, and not the source directory. It is trivial to modify this script to accommodate building different executables, e.g., replacing the su3_rhmd_hisq executable name in the script with the desired one. Note that we need to point MILC to the installed QMP and QIO as part of the QUDA installation, these will be located in the usqcd directory in the QUDA build directory: simply setting both QMPPAR and QIOPAR to ${QUDA_HOME}/usqcd is sufficient.

cd ks_imp_rhmc
cp ../Makefile .
./compile_su3_rhmd_hisq_quda.sh

The build of MILC should now be complete.

Running MILC with QUDA

Typically, running MILC with QUDA is exactly like running MILC without QUDA. There is a one-to-one mapping between the number of GPUs and the number of MPI processes in the system. If you have followed the above instructions to build QUDA, then QUDA will have been built as a shared library, and so you will need to include the path to the QUDA library in the LD_LIBRARY_PATH.

Typically, the CUDA Multi-Process Service (MPS) should not be enabled as this will only decrease performance. An exception to this could be if running on a system with many CPU cores, and MPI performance is superior to OpenMP performance. Otherwise just set OMP_NUM_THREADS (or equivalent) to the number of cores available per process (per GPU).

Set a location for QUDA to write out its autotuning cache: e.g.,

export QUDA_RESOURCE_PATH=/tmp

On the first run QUDA will dump the kernel launch parameters here, for use in later runs. Thus to get optimum performance you should do first a tuning run, and then do a benchmarking run afterwards. This path should be set to a location that is accessible by whichever nodes are running the executable.

Performance Tuning

For guidelines in how to improve strong scaling performance (fixed problem size as the number of GPUs is increased) you can refer to these pages quick-start and multi-gpu.

By default MILC will attempt to split the problem between processes in order to minimize the surface-to-volume ratio of the local problem size. This is in general a good thing to do, however, MILC favours partitioning the fastest running X dimension rather than that slowest running T dimension first. This is bad for running on modern architectures since it leads to strided memory accesses when doing the X-face halo update. The process grid topology can be set manually making it easy to override this, using the command-line option

-qmp-geom MX MY MZ MT

to specify a partitioning of the X axis in MX equal segments, the Y axis into MY segments, etc. So, for example, with a lattice size 32x32x32x64 and 8 MPI ranks the command

mpirun -np 8 ./su3_rhmc_hisq -qmp-geom 1 1 2 4 ...(other options)

would result in local volumes of 32x32x16x16 on a 1x1x2x4 grid of virtual processors. Without this additional flag, the process topology would default to local problem size 16x16x32x32 (partitioning in T first since it has length 64, then split from the X dimension upwards) which leads to strided memory accesses.

With QMP-2.5.1 and above, users can control the logical topology, helping improve inter/intra node layout. In addition to the regular QMP args (-qmp-geom x y z t), one can also pass two new args, -qmp-logic-map and -qmp-alloc-map, which control the process coordinate to rank mapping, for the following two examples

mpirun -np 8 ./su3_rhmc_hisq -qmp-geom 1 1 2 4 -qmp-logic-map 0 1 2 3 -qmp-alloc-map 0 1 2 3 ...(other options)
mpirun -np 8 ./su3_rhmc_hisq -qmp-geom 1 1 2 4 -qmp-logic-map 3 2 1 0 -qmp-alloc-map 3 2 1 0 ...(other options)

If we assume that the MPI launcher packs adjacent ranks onto the same node, the first invocation would result in the T process coordinate equal to rank/2 and the Z coordinate equal to rank%2. Conversely, the second invocation would have time as T = rank%4 and Z = rank/4. In general the user should ensure that the two map arguments have identical parameters.

See also the NERSC-MILC page for further details about launching large-scale jobs.