I'm trying to test the rdma bandwith using ib_send_bw. I found that it does not work with message size between 1025~1344.
Here is an example:
Server Side:
weijia@compute29:~$ ib_send_bw -s 1025
************************************
* Waiting for client to connect... *
************************************
---------------------------------------------------------------------------------------
Send BW Test
Dual-port : OFF Device : mlx5_1
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
RX depth : 512
CQ Moderation : 100
Mtu : 4096[B]
Link type : Ethernet
Gid index : 0
Max inline data : 0[B]
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
local address: LID 0000 QPN 0x0720 PSN 0x1b9abd
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:09:29
remote address: LID 0000 QPN 0x0593 PSN 0xee06cc
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:09:31
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps]
mlx5: compute29: got completion with error:
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 9d003304 00000720 020192e0
Completion with error at server
Failed status 4: wr_id 0 syndrom 0x33
rcnt=513
Client Side:
weijia@compute31:~$ ib_send_bw compute29 -s 1025
---------------------------------------------------------------------------------------
Send BW Test
Dual-port : OFF Device : mlx5_1
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
TX depth : 128
CQ Moderation : 100
Mtu : 4096[B]
Link type : Ethernet
Gid index : 0
Max inline data : 0[B]
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
local address: LID 0000 QPN 0x0593 PSN 0xee06cc
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:09:31
remote address: LID 0000 QPN 0x0720 PSN 0x1b9abd
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:09:29
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps]
mlx5: compute31: got completion with error:
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00008914 0a000593 020128d0
Completion with error at client
Failed status 11: wr_id 0 syndrom 0x89
scnt=628, ccnt=500
======================================================
My server setup:
OS: ubuntu linux 12.04
OFED: MLNX_OFED_LINUX-3.2-2.0.0.0-ubuntu12.04-x86_64
NIC: Mellanox Connect X-4 VPI dual port NIC MCX456A-ECAT