MILC with QUDA

These instructions are intended to be a quick start guide to getting MILC running with GPUs using the QUDA library.

Obtaining and compiling QUDA

You can obtain QUDA using the following:

git clone https://github.com/lattice/quda.git

At the time of writing, the current recommended branch of QUDA is the develop branch, which is the git default. The current stable release branch, release/1.0.x, does not compile, which is something we are actively addressing.

QUDA uses cmake to set compilation options. For running with HISQ fermions, e.g., the su3_rhmc_hisq test that is commonly used in MILC, you should the minimal suggested configuration is

mkdir build
cd build
cmake ../quda -DCMAKE_BUILD_TYPE=RELEASE -DQUDA_GPU_ARCH=sm_70 -DQUDA_DIRAC_DEFAULT_OFF=ON -DQUDA_DIRAC_STAGGERED=ON \
  -DQUDA_FORCE_HISQ=ON -DQUDA_FORCE_GAUGE=ON -DQUDA_BUILD_SHAREDLIB=ON \
  -DQUDA_QMP=ON -DQUDA_QIO=ON -DQUDA_DOWNLOAD_USQCD=ON

Above, we implicitly assume that the CUDA and MPI compilers are present in the $PATH. Here we are setting the the GPU architecture to sm_70 which corresponds to Volta. Choices include:

• sm_35 for Kepler (Tesla K20 / K40 / K80)
• sm_60 for Pascal (Tesla P100, Quadro GP100)
• sm_70 for Volta (Tesla V100, Quadro V100)
• sm_80 for Ampere (NVIDIA A100)

Here we are disabling unnecessary parts of QUDA when used with MILC, in order to reduce compilation time. The final three arguments concern the installation of the USQCD companion libraries QMP and QIO. QUDA can automate their download and installation, and that is what we have enabled here. You can optionally specify an install directory with -DCMAKE_INSTALL_PREFIX=[path], though for MILC bindings it's sufficient to just work from the build directory.

To build QUDA, you should use a parallel build as QUDA can take a long time to build,

make -j N

where N is the number of cores / threads that the compilation node has. We typically recommend setting this to the number of hardware threads (e.g., hyperthreads) in the system. If you have set an install path when running cmake (-DCMAKE_INSTALL_PREFIX=[path]), then to complete the installation run

make install

Finally note that when building with OpenMPI 4.x and above, due to the use of the deprecated MPI_Type_struct, QMP will fail to build unless OpenMPI has been configured with the MPI-1 compatibility option --enable-mpi1-compatibility. The solution is to either enable this option in the OpenMPI build or trivially edit the QMP source code to change the single occurrence of MPI_Type_struct to MPI_type_create_struct in usqcd/src/QMP/lib/mpi/QMP_mem_mpi.c. Fixing this issue in QMP is tracked here.

Obtaining and compiling MILC

For use with QUDA we recommend the present develop branch of MILC. This enables the maximum benefit of QUDA acceleration.

git clone --branch develop https://github.com/milc-qcd/milc_qcd.git

To aid compilation of MILC with QUDA support, there is a provided helper script for the su3_rhmd_hisq application ks_imp_rhmc/compile_su3_rhmd_hisq_quda.sh. Editing this script as appropriate and executing from its directory should result in a full build of MILC with QUDA acceleration for the desired application. For a standard build the important settings are CUDA_HOME, QUDA_HOME, QIOPAR and QMPPAR, where QUDA_HOME can point to the build or install directory for QUDA. It is trivial to modify this script to accommodate building different executables, e.g., replacing the su3_rhmd_hisq executable name in the script with the desired one.

Note that we need to point MILC to the installed QMP and QIO as part of the QUDA installation, these will be located in the usqcd directory in the QUDA build directory.

After modifying ks_imp_rhmc/compute_su3_rhmd_hisq_quda.sh appropriately, the MILC RHMC driver can be built via

cd ks_imp_rhmc
cp ../Makefile .
./compile_su3_rhmd_hisq_quda.sh

The build of MILC should now be complete.

Running MILC with QUDA

Typically, running MILC with QUDA is exactly like running MILC without QUDA. There is a one-to-one mapping between the number of GPUs and the number of MPI processes in the system. If you have followed the above instructions to build QUDA, then QUDA will have been built as a shared library, and so you will need to include the path to the QUDA library in the LD_LIBRARY_PATH.

Typically, the CUDA Multi-Process Service (MPS) should not be enabled as this will only decrease performance. An exception to this could be if running on a system with many CPU cores, and MPI performance is superior to OpenMP performance. Otherwise just set OMP_NUM_THREADS (or equivalent) to the number of cores available per process (per GPU).

Set a location for QUDA to write out its autotuning cache: e.g.,

export QUDA_RESOURCE_PATH=/tmp

On the first run QUDA will dump the kernel launch parameters here, for use in later runs. Thus to get optimum performance you should do first a tuning run, and then do a benchmarking run afterwards. This path should be set to a location that is accessible by whichever nodes are running the executable.

Performance Tuning

For guidelines in how to improve strong scaling performance (fixed problem size as the number of GPUs is increased) you can refer to these pages quick-start and multi-gpu.

By default MILC will attempt to split the problem between processes in order to minimize the surface-to-volume ratio of the local problem size. This is in general a good thing to do, however, MILC favours partitioning the fastest running X dimension rather than that slowest running T dimension first. This is bad for running on modern architectures since it leads to strided memory accesses when doing the X-face halo update. The process grid topology can be set manually making it easy to override this, using the command-line option

-qmp-geom MX MY MZ MT

to specify a partitioning of the X axis in MX equal segments, the Y axis into MY segments, etc. So, for example, with a lattice size 32x32x32x64 and 8 MPI ranks the command

mpirun -np 8 ./su3_rhmc_hisq -qmp-geom 1 1 2 4 ...(other options)

would result in local volumes of 32x32x16x16 on a 1x1x2x4 grid of virtual processors. Without this additional flag, the process topology would default to local problem size 16x16x32x32 (partitioning in T first since it has length 64, then split from the X dimension upwards) which leads to strided memory accesses.

With QMP-2.5.1 and above, users can control the logical topology, helping improve inter/intra node layout. In addition to the regular QMP args (-qmp-geom x y z t), one can also pass two new args, -qmp-logic-map and -qmp-alloc-map, which control the process coordinate to rank mapping, for the following two examples

mpirun -np 8 ./su3_rhmc_hisq -qmp-geom 1 1 2 4 -qmp-logic-map 0 1 2 3 -qmp-alloc-map 0 1 2 3 ...(other options)
mpirun -np 8 ./su3_rhmc_hisq -qmp-geom 1 1 2 4 -qmp-logic-map 3 2 1 0 -qmp-alloc-map 3 2 1 0 ...(other options)

If we assume that the MPI launcher packs adjacent ranks onto the same node, the first invocation would result in the T process coordinate equal to rank/2 and the Z coordinate equal to rank%2. Conversely, the second invocation would have time as T = rank%4 and Z = rank/4. In general the user should ensure that the two map arguments have identical parameters.

See also the NERSC-MILC page for further details about launching large-scale jobs.