What's the thread-safe model relationship between ucx & ibverbs?

[Yangfisher1]

Recently, I've been using UCP APIs for my own project and have benefited from its RMA operations. Before that, I used ibverbs to develop RDMA-related applications, and as far as I know, the ibverbs APIs are totally thread-safe, meaning that you can use the same resources in multiple threads.

However, when I try to use UCP rather than directly using ibverbs, things turn out differently. It turns out that the ucp_context has a flag called mt_workers_shared to indicate whether different workers from different threads would share the same context or not. Additionally, at the ucp_worker level, it uses UCS_THREAD_MODE_XXX to control the thread-safe level. So, I'm a little bit confused about whether it's necessary to set the thread safety level of these UCX objects if I'm only using the RMA operations and the underlying protocol should be IB interface. Can I directly share it within multiple threads since the underlying ibverbs objects are completely thread-safe?

Another interesting thing I learned is that the ib interface does not support UCS_THREAD_MODE_MULTI, as the code shows:

    if (ucs_derived_of(worker, uct_priv_worker_t)->thread_mode == UCS_THREAD_MODE_MULTI) {
        ucs_error("IB transports do not support multi-threaded worker");
        return UCS_ERR_INVALID_PARAM;
    }

[gleon99]

Hi @Yangfisher1 ,
UCT is always passed SINGLE or SERIALIZED from UCP, unless you use UCT directly, see also:
https://github.com/openucx/ucx/blob/master/src/ucp/core/ucp_worker.c#L2505

[Yangfisher1]

I see, but I'm asking whether it is possible to directly share resources with RDMA components since the ibverbs APIs are totally thread-safe.

[gleon99]

You can share UCP objects across threads if the correct thread mode is set upon creation.

[Yangfisher1]

the correct thread mode is set upon creation.

I just don't understand the relationship between the UCT's thread mode and the ibverbs API's. As I mentioned above, ibverbs APIs are thread-safe, so I can directly use UCT with UCS_THREAD_MODE_SINGLE flag and share the underlying resources with ib inferaces?

[gleon99]

I can directly use UCT with UCS_THREAD_MODE_SINGLE flag and share the underlying resources with ib inferaces

No, because of MMIO usage by UCT, which is different from ibverbs.
We recommend using the UCP layer APIs, which ensure MT safety.
In case you need to share UCT resources between threads, use UCS_THREAD_MODE_SERIALIZED and ensure their synchronization.

[Yangfisher1]

@gleon99 Got it, thanks! In this case maybe I should directly use ibverbs API to minimize the resource overhead.