Wrong numerical result with BiCGStab in OpenMP

[blegouix]

Hello,

Unfortunatly I cannot provide a minimal reproducer atm so I guess it will be almost impossible to debug for you, but just to let you know we have a project in which we observe a wrong result for a multiple-rhs solve with BiCGStab in OpenMP even with OMP_NUM_THREADS=1. The workaround forcing the use of serial ginkgo executor in place of omp one "fixes" the problem:

https://github.com/CExA-project/ddc/pull/319/files

Here, create_gko_exec<Kokkos::Serial>() is a gko::ReferenceExecutor::create(); while gko_exec is a gko::OmpExecutor::create();.

If you can't tell what the problem could be I will update this discussion when I will have more substantial informations.

I tried with gcc and clang, with Ginkgo 1.7.0 and current develop branch.

The workaround is not necessary (result is correct) with cpu serial or cuda executor.

Valgrind does not reveals anything and both cases (with or without fix) "converge" in 3 iterations.