ib_write_bw works fine without -R. But only very small bandwidth when using -R option

[chgdragon2023]

When I use ib_write_bw without -R option, everything looks fine to me. However, if -R is used. only very small bandwidth can be achieved. In addition, when -R option is used, the CPU usage on the server side is very high

            root@loving-insect:/home/ocp_user# ib_write_bw -d mlx5_0 -i 1 -D 1 -m 4096 -q 1 -s 1000000 -t 10 --report_gbits
            
            ************************************
            * Waiting for client to connect... *
            ************************************
            ---------------------------------------------------------------------------------------
                                RDMA_Write BW Test
             Dual-port       : OFF          Device         : mlx5_0
             Number of qps   : 1            Transport type : IB
             Connection type : RC           Using SRQ      : OFF
             PCIe relax order: ON
             ibv_wr* API     : ON
             CQ Moderation   : 1
             Mtu             : 4096[B]
             Link type       : Ethernet
             GID index       : 3
             Max inline data : 0[B]
             rdma_cm QPs     : OFF
             Data ex. method : Ethernet
            ---------------------------------------------------------------------------------------
             local address: LID 0000 QPN 0x2da6 PSN 0x38f9aa RKey 0x201200 VAddr 0x007f67c0644240
             GID: 00:00:00:00:00:00:00:00:00:00:255:255:11:00:00:72
             remote address: LID 0000 QPN 0x3606 PSN 0xf0086a RKey 0x200c00 VAddr 0x007f0a53d57240
             GID: 00:00:00:00:00:00:00:00:00:00:255:255:11:00:00:88
            ---------------------------------------------------------------------------------------
             #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
             1000000    48233            0.00               385.86             0.048233
            ---------------------------------------------------------------------------------------
            root@loving-insect:/home/ocp_user# ib_write_bw -d mlx5_0 -i 1 -D 1 -m 4096 -q 1 -s 1000000 -t 10 --report_gbits -R
            
            ************************************
            * Waiting for client to connect... *
            ************************************
            ---------------------------------------------------------------------------------------
                                RDMA_Write BW Test
             Dual-port       : OFF          Device         : mlx5_0
             Number of qps   : 1            Transport type : IB
             Connection type : RC           Using SRQ      : OFF
             PCIe relax order: ON
             ibv_wr* API     : ON
             CQ Moderation   : 1
             Mtu             : 4096[B]
             Link type       : Ethernet
             GID index       : 3
             Max inline data : 0[B]
             rdma_cm QPs     : ON
             Data ex. method : rdma_cm
            ---------------------------------------------------------------------------------------
             Waiting for client rdma_cm QP to connect
             Please run the same command with the IB/RoCE interface IP
            ---------------------------------------------------------------------------------------
             local address: LID 0000 QPN 0x2da8 PSN 0x4fc3c9
             GID: 00:00:00:00:00:00:00:00:00:00:255:255:11:00:00:72
             remote address: LID 0000 QPN 0x2c24 PSN 0xac206a
             GID: 00:00:00:00:00:00:00:00:00:00:255:255:11:00:00:95
            ---------------------------------------------------------------------------------------
             #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
             1000000    6           0.000000            0.048000            0.000006
            ---------------------------------------------------------------------------------------
            root@loving-insect:/home/ocp_user# 

[Lyflihai]

you may use ZTRCC as your congestion control protocol