PASC24: Controlling Parallel CFD Simulations in Julia from C/Fortran with libtrixi

This is the companion repository for the poster

Controlling Parallel CFD Simulations in Julia from C/Fortran with libtrixi
Gregor Gassner, Benedict Geihe, Michael Schlottke-Lakemper

PASC24, ETH Zurich, June 3 to 5, 2024

Abstract

With libtrixi we present a software library to control complex Julia code from a main program written in a different language. Specifically, libtrixi provides an API to Trixi.jl, a Julia package for adaptive numerical simulations of conservation laws, used to accurately predict naturally occurring processes in various areas of physics. Here a broad range of spatial and temporal length scales render finely resolved computational grids indispensable and call for high-performance computing techniques. Consequently, many simulation tools are written in traditional HPC languages such as C, C++, or Fortran, which offer high computational performance, but are often complex to learn and maintain. The Julia programming language aims to combine convenience with performance by providing an accessible, high-level syntax together with fast, just-in-time-compiled execution. With libtrixi we present a blue print for connecting established research codes to modern software packages written in Julia. We will give details on the implementation of the interface library and show numerical applications in earth system modeling. These are controlled by a Fortran code and employ Trixi.jl’s distributed CPU compute capabilities.

Poster

The poster can be downloaded here.

Reproducibility

The instructions are written based on the supercomputer JUWELS, operated at Jülich Supercomputing Centre, on which the results presented on the poster were obtained.

Getting started

The following standard software packages need to be made available locally before installing libtrixi:

• C compiler with support for C11 or later (e.g., GCC or Clang) (we used GCC v12.3.0)
• Fortran compiler with support for Fortran 2018 or later (e.g., Gfortran) (we used Gfortran v12.3.0)
• CMake (we used cmake v3.26.3)
• MPI (e.g., Open MPI or MPICH) (we used Open MPI v4.1.5)
• HDF5 (we used parallel HDF5 v1.14.2)
• Julia (it is advised to use tarballs from the official website or the juliaup manager, we used v1.10.3)
• Paraview for visualization (we used v5.12.0)

The following software packages require manual compilation and installation. To simplify the installation instructions, we assume that the environment variable WORKDIR was set to the directory, where all the code is downloaded to and compiled, and PREFIX was set to an installation prefix. For example,

export WORKDIR=$HOME/repro-poster-2024-pasc-libtrixi
export PREFIX=$WORKDIR/install

t8code

t8code is a meshing library which is used in Trixi.jl. The source code can be obtained from the official website. Here is an example for compiling and installing it (we used v2.0.0):

cd $WORKDIR
git clone --branch 'v2.0.0' --depth 1 https://github.com/DLR-AMR/t8code.git
cd t8code
git submodule init
git submodule update
./bootstrap
./configure --prefix="${PREFIX}/t8code" \
            CFLAGS="-Wall -O2" \
            CXXFLAGS="-Wall -O2" \
            --enable-mpi \
            CXX=mpicxx \
            CC=mpicc \
            --enable-static \
            --without-blas \
            --without-lapack
make
make install

libtrixi

Here is an example for compiling and installing libtrixi (we used v0.1.5):

cd $WORKDIR
git clone --branch 'v0.1.5' --depth 1 https://github.com/trixi-framework/libtrixi.git
cd libtrixi
mkdir build
cd build
cmake -DCMAKE_BUILD_TYPE=Release \
      -DCMAKE_INSTALL_PREFIX=${PREFIX}/libtrixi ..
make
make install

A separate Julia project for use with libtrixi has to be set up:

mkdir ${PREFIX}/libtrixi-julia
cd ${PREFIX}/libtrixi-julia
${PREFIX}/libtrixi/bin/libtrixi-init-julia \
  --t8code-library ${PREFIX}/t8code/lib/libt8.so \
  ${PREFIX}/libtrixi

To exactly reproduce the results presented on the poster, the provided Manifest.toml can be used to install the same package versions for all Julia dependencies:

cp Manifest.toml ${PREFIX}/libtrixi-julia/
cd ${PREFIX}/libtrixi-julia
JULIA_DEPOT_PATH=./julia-depot \
  julia --project=. -e 'import Pkg; Pkg.instantiate()'

Alternatively all packages can be updated to the most recent version on demand:

cd ${PREFIX}/libtrixi-julia
JULIA_DEPOT_PATH=./julia-depot \
  julia --project=. -e 'import Pkg; Pkg.update()'

Trixi2Vtk

Trixi2Vtk needs to be installed for later post processing (we used v0.3.15):

cd ${PREFIX}/libtrixi-julia
JULIA_DEPOT_PATH=./julia-depot \
  julia --project=. -e 'import Pkg; Pkg.add(name="Trixi2Vtk", version="0.3.15")'

Running the baroclinic instability simulation

The julia scripts for setting up simulations (libelixirs) can be found in the examples folder:

export EXAMPLES=$PREFIX/libtrixi/share/libtrixi/LibTrixi.jl/examples

For the results on the poster the resolution of the computational mesh was increased. For this, modify line 128 of ${EXAMPLES}/libelixir_p4est3d_euler_baroclinic_instability.jl such that initial_refinement_level = 1.

We used the following SLURM script to launch the simulation on JUWELS:

#!/bin/bash -x
#SBATCH --nodes=128
#SBATCH --ntasks-per-node=48

module purge
module load GCC
module load OpenMPI
module load HDF5/1.14.2

srun -n "${SLURM_NTASKS}" \
  ${PREFIX}/libtrixi/bin/trixi_controller_simple_f \
  ${PREFIX}/libtrixi-julia \
  ${EXAMPLES}/libelixir_p4est3d_euler_baroclinic_instability.jl

Output files will be written to ./out_baroclinic and need to be post processed:

cd ${PREFIX}/libtrixi/out_bubble
JULIA_DEPOT_PATH=${PREFIX}/libtrixi-julia/julia-depot \
   julia --project=${PREFIX}/libtrixi-julia -e 'using Trixi2Vtk; trixi2vtk("solution*.h5")'

The results can now be viewed using, e.g., paraview. We used slice_surface.pvsm to generate our results.

Authors

This repository was initiated by Benedict Geihe and Michael Schlottke-Lakemper.

License

The contents of this repository are licensed under the MIT license (see LICENSE.md).