UCP worker progress assertion failures

[miEsMar]

Hi all,

I am working on a MPI based project. The main implementation being OpenMPI.
In trying to test some things, the MPI interafce (via the mpi.h file) has been extended to provide some functionality, among which querying the underlying UCP worker (version 1.15), to be used internally by the tool, if the UCX PML MCA module is effectively loaded (which for testing purposes, it is always).

However, under some tool configuration, the runtime experiences some UCP level assertion failures:

ucp_worker.c:2888 Assertion `worker->inprogress++ == 0' failed
...
ucp_worker.c:2893 Assertion `--worker->inprogress == 0' failed

providing different backtraces (why don't all MPI ranks return the same backtrace?):

==== backtrace (tid:2924686) ====
 0  /apps/GPP/UCX/1.15.0/GCC/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f5191333964]
 1  /apps/GPP/UCX/1.15.0/GCC/lib/libucs.so.0(ucs_fatal_error_message+0xb2) [0x7f5191330ad2]
 2  /apps/GPP/UCX/1.15.0/GCC/lib/libucs.so.0(ucs_fatal_error_format+0xd9) [0x7f5191330bb9]
 3  /apps/GPP/UCX/1.15.0/GCC/lib/libucp.so.0(ucp_worker_progress+0x1c8) [0x7f51913cc1e8]
 4  ../ompi/install/lib/openmpi/mca_pml_ucx.so(mca_pml_ucx_progress+0x1a) [0x7f5192005d7d]
 5  ../ompi/install/lib/libopen-pal.so.40(opal_progress+0x30) [0x7f5216721750]
[...]

*** Process received signal ***
Signal: Aborted (6)
Signal code:  (-6)
[ 0] 
==== backtrace (tid:2924679) ====
 0  /apps/GPP/UCX/1.15.0/GCC/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f5191333964]
 1  /apps/GPP/UCX/1.15.0/GCC/lib/libucs.so.0(ucs_fatal_error_message+0xb2) [0x7f5191330ad2]
 2  /apps/GPP/UCX/1.15.0/GCC/lib/libucs.so.0(ucs_fatal_error_format+0xd9) [0x7f5191330bb9]
 3  /apps/GPP/UCX/1.15.0/GCC/lib/libucp.so.0(ucp_worker_progress+0x198) [0x7f51913cc1b8]
 4  ../ompi/install/lib/openmpi/mca_pml_ucx.so(mca_pml_ucx_progress+0x1a) [0x7f5192005d7d]
 5  ../ompi/install/lib/libopen-pal.so.40(opal_progress+0x30) [0x7f5216721750]
 6  ../ompi/install/lib/libmpi.so.40(+0x623f4) [0x7f52169713f4]
 7  ../ompi/install/lib/libmpi.so.40(ompi_request_default_wait+0x27) [0x7f521697142f]
 8  ../ompi/install/lib/libmpi.so.40(ompi_coll_base_sendrecv_actual+0x10a) [0x7f5216a1b48f]
 9  ../ompi/install/lib/libmpi.so.40(ompi_coll_base_allreduce_intra_recursivedoubling+0x343) [0x7f5216a1df44]
10  ../ompi/install/lib/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allreduce_intra_do_this+0x13f) [0x7f5190115727]
11  ../ompi/install/lib/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allreduce_intra_dec_fixed+0x485) [0x7f519010de1a]
12  ../ompi/install/lib/libmpi.so.40(PMPI_Allreduce+0x405) [0x7f521698e69c]
[...]

Inspecting the UCP source code, I see that those assertions are to check whether the progress procedure is called recursively.
What are the circumstances under which this happens?

Has anyone ever encountered such issues before?

Thanks.

[miEsMar]

Also, in the stdout file, the following is reported:

   debug.c:1294 UCX  WARN  ucs_debug_disable_signal: signal 8 was not set
   debug.c:1294 UCX  WARN  ucs_debug_disable_signal: signal 7 was not set
spinlock.c:29   UCX  WARN  ucs_recursive_spinlock_destroy() failed: busy
   debug.c:1294 UCX  WARN  ucs_debug_disable_signal: signal 8 was not set
spinlock.c:29   UCX  WARN  ucs_recursive_spinlock_destroy() failed: busy
   debug.c:1294 UCX  WARN  ucs_debug_disable_signal: signal 8 was not set
   debug.c:1294 UCX  WARN  ucs_debug_disable_signal: signal 1 was not set
spinlock.c:29   UCX  WARN  ucs_recursive_spinlock_destroy() failed: busy
   debug.c:1294 UCX  WARN  ucs_debug_disable_signal: signal 8 was not set
spinlock.c:29   UCX  WARN  ucs_recursive_spinlock_destroy() failed: busy

[miEsMar]

Can it be related to Multi-threading?
I.e. multiple threads are calling to ucp_worker_progress() concurrently with the same worker?
Though a sync mechanism is in place, so I don't see why it's failing..