Work-Around: Segfault in MPI_Init with HIP

[ax3l]

See:
https://docs.olcf.ornl.gov/systems/crusher_quick_start_guide.html#olcfdev-1655-occasional-seg-fault-during-mpi-init

Proposed work-around for @tmsclark in #4236.

I think this might be a general defect of GPU-aware MPI implementations from HPE/Cray at this point, explicit device context init before MPI init can help establishing GPU-aware MPI init assumptions at this point.
Clarifying with AMD if we can have a check for an already existing HIP/ROCm initialized runtime, in case we want to move this safety net to AMReX at some point, too. (Also, not clear if hipInit is idempotent.)

• make part of ablastr::parallelization::mpi_init?
• test on AMD GPUs, i.e. Frontier, LUMI

[lucafedeli88]

Thanks really a lot for this PR! I've left a small comment. I'll test these changes as soon as possible.

[lucafedeli88]

Is there a reason for not using ABLASTR_ALWAYS_ASSERT_WITH_MESSAGE here?

[ax3l]

Yes: anything calling into AMReX functions cannot be used (safely) before AMReX is initialized. amrex::Assert implements multiple things, not all of them work with an uninitialized AMReX context.

For instance, not even amrex::Print() works before init, I had to develop a work-around falling back to all-print in pyAMReX: AMReX-Codes/pyamrex#174

Raising a standard exception is a clean thing to do here - we init MPI and technically there is no AMReX involved at this point in time yet.

[ax3l]

I tested this on Frontier and it does not cause issues.

[ax3l]

@tmsclark @lucafedeli88 let me know if it fixes your LUMI issue :)