I am hosting an Infiniband server in a linux machine and also I have created a client and connected to that service in the same machine.This works fine most of the time. But in one instance when I was trying to connect to that server from the same client (there is no prior connectivity with that server), It is throwing RDMA_CM_EVENT_ROUTE_ERROR and the connection couldn't be established.

I don't know the root cause of this error and it is not 100% recreatable. This made my application unreliable. I want to know the root cause of it..

Hi

To answer the below:

1. Will this optical QSFP+ module (40GbE) work on each switch ? If so, on what port ? Or any port will be OK ?

Ans.  It will work on all ports.

2. What does mean the '' WT '' letters on the MSX6012 model ?  (Wide Transceiver ??) If so, does that mean that i need to buy an another MSX6012F_WT ? Or the MSX1012B will also work fine ?

Ans.  MSX6012 if by default an infiniband switch and MSX1012 is by default an ethernet switch.  If you have an ethernet license (which can be purchased from Mellanox) installed on the MSX6012 switch, and the profile is now ethernet, the modules should work with both switches.

3. Actually, my MSX6012F is in VPI profil mode. Do i need to put it in single_ethernet mode ? (i don't do infiniband on the network for now, but it is envisaged).

Ans.  I would suggest to change is to single_ethernet mode.

4. As our needs are evolving, and my two switches already have almost all their ports in use, I plan to buy two MSX6036F to replace the MSX1012B and MSX6012F. Will both MC2210511-LR4 modules will work and, if so, on which ports?

Ans.  It will work on ports 1,3,33,&35.

Hello-

In case someday anyone else would like to try, we went ahead and upgraded to Ubuntu 18 and OFED 4, and our old ConnectX-2 IB cards are still working fine with this setup, despite the release notes indicating support was dropped.

-Lewis

ConnectX-5 supports PTP.

The Release Notes wording should be fixed to explain the limitation better, it is referring to "service type" in the QP; this is an advanced feature that does not affect working with PTP today.

Sorry for the confusion.

Erez.

Do you want passthrough/sriov to the VM or have a IB mode card acting as a network device for a virtual switch?

The former definitely should work the latter I think as well. Have not been able to get it to go with RDMA but basic IB should be simple enough I think.

When I rebuild the packages, the debuginfo package is there! Then I check the building logs and files, it seems that all the user-space libs are not re-built at all. Only the kernel rpms are built.

I didn't find the switch in the script to rebuild the user-space libs, then could any expert help this?

Dear community,

Pretty simple question: Is it possible to disable the hardware tag matching capabilities for MPI on the ConnectX-5 adapters? If yes, how?

Thank you,

Karlo

Need help, I'm running out of ideas!

I have a Dell M1000e blade chassis with M3601Q 40gbps Mellanox infiniband switches in I/O slot B1C1, connects to Midplane on C1. I have M910 Poweredge blades with J05yt connectX3 mezzanine card plugged. I have installed latest MLNX OFED 4.4. The OS is based on CentOS7.4 within Rocks Manzanita cluster. Since it is a blade, connection is via midplane. Switch lights are steady and good.

After following prior posts, executing the commands such as ibhosts, ibstat, lspci | grep Mell, lspci -Qvvs 07:00.0, ifcong -a, HCA_self_test.ofed, and mstflint -d 07:00.0 q, the best I can tell is my port is down/Initializing andI have subnet manager issue.  I cannot get it Active or an IP show. Can you please help me diagnose? I'll post some needed output, let me know what else is required.

Thank you much!

[root@headnode /]# hca_self_test.ofed

---- Performing Adapter Device Self Test ----

Number of CAs Detected ................. 2

PCI Device Check ....................... PASS

Kernel Arch ............................ x86_64

Host Driver Version .................... MLNX_OFED_LINUX-4.4-2.0.7.0 (OFED-4.4-2.0.7): 3.10.0-693.el7.x86_64

Host Driver RPM Check .................. PASS

Firmware on CA #0 HCA .................. v2.10.2132

Firmware on CA #1 HCA .................. v2.10.2132

Host Driver Initialization ............. PASS

Number of CA Ports Active .............. 0

Port State of Port #1 on CA #0 (HCA)..... DOWN (InfiniBand)

Port State of Port #2 on CA #0 (HCA)..... DOWN (InfiniBand)

Port State of Port #1 on CA #1 (HCA)..... INIT (InfiniBand)

Port State of Port #2 on CA #1 (HCA)..... DOWN (InfiniBand)

Error Counter Check on CA #0 (HCA)...... FAIL

    REASON: found errors in the following counters

      Errors in /sys/class/infiniband/mlx4_0/ports/1/counters

         link_error_recovery: 93

         symbol_error: 65535

Error Counter Check on CA #1 (HCA)...... PASS

Kernel Syslog Check .................... PASS

Node GUID on CA #0 (HCA) ............... 00:02:c9:03:00:f9:2e:80

Node GUID on CA #1 (HCA) ............... 00:02:c9:03:00:f9:32:f0

------------------ DONE ---------------------

[root@headnode /]# ibhosts

Ca    : 0x0002c90300f92e80 ports 2 "headnode HCA-1"

[root@headnode /]# ibstat

CA 'mlx4_0'

    CA type: MT4099

    Number of ports: 2

    Firmware version: 2.10.2132

    Hardware version: 0

    Node GUID: 0x0002c90300f92e80

    System image GUID: 0x0002c90300f92e83

    Port 1:

        State: Down

        Physical state: Polling

        Rate: 10

        Base lid: 0

        LMC: 0

        SM lid: 0

        Capability mask: 0x02514868

        Port GUID: 0x0002c90300f92e81

        Link layer: InfiniBand

    Port 2:

        State: Down

        Physical state: Polling

        Rate: 10

        Base lid: 0

        LMC: 0

        SM lid: 0

        Capability mask: 0x02514868

        Port GUID: 0x0002c90300f92e82

        Link layer: InfiniBand

CA 'mlx4_1'

    CA type: MT4099

    Number of ports: 2

    Firmware version: 2.10.2132

    Hardware version: 0

    Node GUID: 0x0002c90300f932f0

    System image GUID: 0x0002c90300f932f3

    Port 1:

        State: Initializing

        Physical state: LinkUp

        Rate: 40

        Base lid: 0

        LMC: 0

        SM lid: 0

        Capability mask: 0x02514868

        Port GUID: 0x0002c90300f932f1

        Link layer: InfiniBand

    Port 2:

        State: Down

        Physical state: Polling

        Rate: 10

        Base lid: 0

        LMC: 0

        SM lid: 0

        Capability mask: 0x02514868

        Port GUID: 0x0002c90300f932f2

        Link layer: InfiniBand

[root@headnode /]#

[root@headnode /]# mstflint -d 05:00.0 q

Image type:            FS2

FW Version:            2.10.2132

Device ID:             4099

Description:           Node             Port1            Port2            Sys image

GUIDs:                 0002c90300f92e80 0002c90300f92e81 0002c90300f92e82 0002c90300f92e83

MACs:                                       000000000000     000000000000

VSD:                  

PSID:                  DEL0A10210018

[root@headnode /]# lspci -Qvvs 05:00.0

05:00.0 Infiniband controller: Mellanox Technologies MT27500 Family [ConnectX-3]

    Subsystem: Mellanox Technologies ConnectX-3 IB QDR Dual Port Mezzanine Card

    Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+

    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

    Latency: 0, Cache Line Size: 64 bytes

    Interrupt: pin A routed to IRQ 34

    Region 0: Memory at fb100000 (64-bit, non-prefetchable) [size=1M]

    Region 2: Memory at f4800000 (64-bit, prefetchable) [size=8M]

    Capabilities: [40] Power Management version 3

        Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)

        Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-

    Capabilities: [48] Vital Product Data

        Product Name: DELL ConnectX-3 Mezz

        Read-only fields:

            [PN] Part number: 0J05YT              

            [EC] Engineering changes: A00

            [SN] Serial number: IL0J05YT7403125S000Q

            [V0] Vendor specific: DDR/QDR SFF mezz

            [RV] Reserved: checksum good, 0 byte(s) reserved

        Read/write fields:

            [V1] Vendor specific: N/A  

            [YA] Asset tag: N/A                        

            [RW] Read-write area: 107 byte(s) free

            [RW] Read-write area: 253 byte(s) free

            [RW] Read-write area: 253 byte(s) free

            [RW] Read-write area: 253 byte(s) free

            [RW] Read-write area: 253 byte(s) free

            [RW] Read-write area: 253 byte(s) free

            [RW] Read-write area: 253 byte(s) free

            [RW] Read-write area: 253 byte(s) free

            [RW] Read-write area: 253 byte(s) free

            [RW] Read-write area: 253 byte(s) free

            [RW] Read-write area: 253 byte(s) free

            [RW] Read-write area: 253 byte(s) free

            [RW] Read-write area: 253 byte(s) free

            [RW] Read-write area: 253 byte(s) free

            [RW] Read-write area: 253 byte(s) free

            [RW] Read-write area: 252 byte(s) free

        End

    Capabilities: [9c] MSI-X: Enable+ Count=128 Masked-

        Vector table: BAR=0 offset=0007c000

        PBA: BAR=0 offset=0007d000

    Capabilities: [60] Express (v2) Endpoint, MSI 00

        DevCap:    MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 unlimited

            ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 116.000W

        DevCtl:    Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported+

            RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- FLReset-

            MaxPayload 256 bytes, MaxReadReq 512 bytes

        DevSta:    CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

        LnkCap:    Port #8, Speed 8GT/s, Width x8, ASPM L0s, Exit Latency L0s unlimited, L1 unlimited

            ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+

        LnkCtl:    ASPM Disabled; RCB 64 bytes Disabled- CommClk+

            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-

        LnkSta:    Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

        DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported

        DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis-, LTR-, OBFF Disabled

        LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-

             Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-

             Compliance De-emphasis: -6dB

        LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-

             EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-

    Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI)

        ARICap:    MFVC- ACS-, Next Function: 0

        ARICtl:    MFVC- ACS-, Function Group: 0

    Capabilities: [148 v1] Device Serial Number 00-02-c9-03-00-f9-2e-80

    Capabilities: [154 v2] Advanced Error Reporting

        UESta:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-

        UEMsk:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt+ UnxCmplt+ RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-

        UESvrt:    DLP+ SDES- TLP+ FCP+ CmpltTO+ CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol-

        CESta:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-

        CEMsk:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+

        AERCap:    First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-

    Capabilities: [18c v1] #19

    Kernel driver in use: mlx4_core

    Kernel modules: mlx4_core

[root@headnode ~]# sminfo -p 1

ibwarn: [8670] _do_madrpc: recv failed: Connection timed out

ibwarn: [8670] mad_rpc: _do_madrpc failed; dport (DR path slid 0; dlid 0; 0)

sminfo: iberror: failed: query

[root@headnode ~]# sminfo -p 2

ibwarn: [8684] _do_madrpc: recv failed: Connection timed out

ibwarn: [8684] mad_rpc: _do_madrpc failed; dport (DR path slid 0; dlid 0; 0)

sminfo: iberror: failed: query

[root@headnode ~]#

Opensm

******************************************************************

****************** ERRORS DURING INITIALIZATION ******************

******************************************************************

Sep 12 11:18:51 735239 [A3E15700] 0x01 -> state_mgr_check_tbl_consistency: ERR 3322: lid 1 is wrongly assigned to port 0x0002c90300f92e81 ('headnode HCA-1' port 1) in port_lid_tbl

Sep 12 11:18:51 735367 [A3E15700] 0x02 -> state_mgr_check_tbl_consistency: Clearing Lid for port 0x0002c90300f92e81

Sep 12 11:18:51 735375 [A3E15700] 0x01 -> state_mgr_check_tbl_consistency: ERR 3322: lid 3 is wrongly assigned to port 0x0002c90300f932f1 ('headnode HCA-2' port 1) in port_lid_tbl

Sep 12 11:18:51 735392 [A3E15700] 0x02 -> state_mgr_check_tbl_consistency: Clearing Lid for port 0x0002c90300f932f1

Sep 12 11:18:51 735430 [A3E15700] 0x01 -> osm_ucast_port_is_zero_lid: ERR 3A04: Port 0x2c90300f932f1 (headnode HCA-2 port 1) has LID 0. An initialization error occurred. Ignoring port

Sep 12 11:18:51 735449 [A3E15700] 0x01 -> osm_ucast_port_is_zero_lid: ERR 3A04: Port 0x2c90300f92e81 (headnode HCA-1 port 1) has LID 0. An initialization error occurred. Ignoring port

Sep 12 11:18:51 735462 [A3E15700] 0x01 -> osm_ucast_port_is_zero_lid: ERR 3A04: Port 0x2c90300f92e81 (headnode HCA-1 port 1) has LID 0. An initialization error occurred. Ignoring port

Sep 12 11:18:51 735468 [A3E15700] 0x01 -> osm_ucast_port_is_zero_lid: ERR 3A04: Port 0x2c90300f932f1 (headnode HCA-2 port 1) has LID 0. An initialization error occurred. Ignoring port

Sep 12 11:18:51 735480 [A3E15700] 0x02 -> osm_ucast_mgr_process: minhop tables configured on all switches

Sep 12 11:18:51 740351 [A3E15700] 0x80 -> Errors during initialization

Sep 12 11:18:51 740385 [A3E15700] 0x01 -> do_sweep:

[root@headnode ~]# nmcli connection show ib0

connection.id:                          ib0

connection.uuid:                        65aec7ac-2335-44aa-b9c2-0945379d8111

connection.stable-id:                   --

connection.interface-name:              ib0

connection.type:                        infiniband

connection.autoconnect:                 yes

connection.autoconnect-priority:        0

connection.autoconnect-retries:         -1 (default)

connection.timestamp:                   0

connection.read-only:                   no

connection.permissions:                 --

connection.zone:                        --

connection.master:                      --

connection.slave-type:                  --

connection.autoconnect-slaves:          -1 (default)

connection.secondaries:                 --

connection.gateway-ping-timeout:        0

connection.metered:                     unknown

connection.lldp:                        -1 (default)

ipv4.method:                            auto

ipv4.dns:                               --

ipv4.dns-search:                        --

ipv4.dns-options:                       (default)

ipv4.dns-priority:                      0

ipv4.addresses:                         --

ipv4.gateway:                           --

ipv4.routes:                            --

ipv4.route-metric:                      -1

ipv4.ignore-auto-routes:                no

ipv4.ignore-auto-dns:                   no

ipv4.dhcp-client-id:                    --

ipv4.dhcp-timeout:                      0

ipv4.dhcp-send-hostname:                yes

ipv4.dhcp-hostname:                     --

ipv4.dhcp-fqdn:                         --

ipv4.never-default:                     yes

ipv4.may-fail:                          yes

ipv4.dad-timeout:                       -1 (default)

ipv6.method:                            link-local

ipv6.dns:                               --

ipv6.dns-search:                        --

ipv6.dns-options:                       (default)

ipv6.dns-priority:                      0

ipv6.addresses:                         --

ipv6.gateway:                           --

ipv6.routes:                            --

ipv6.route-metric:                      -1

ipv6.ignore-auto-routes:                no

ipv6.ignore-auto-dns:                   no

ipv6.never-default:                     no

ipv6.may-fail:                          yes

ipv6.ip6-privacy:                       0 (disabled)

ipv6.addr-gen-mode:                     stable-privacy

ipv6.dhcp-send-hostname:                yes

ipv6.dhcp-hostname:                     --

ipv6.token:                             --

infiniband.mac-address:                 80:00:02:08:FE:80:00:00:00:00:00:00:00:02:C9:03:00:F9:32:F1

infiniband.mtu:                         auto

infiniband.transport-mode:              connected

infiniband.p-key:                       default

infiniband.parent:                      --

proxy.method:                           none

proxy.browser-only:                     no

proxy.pac-url:                          --

proxy.pac-script:                       --

Could any expert give a hand?

Hi,

   We purchased ConnectX-4 Lx Mellanox cards. I wanted to configure the tc flow for vxlan type packets. I am following https://www.mellanox.com/related-docs/prod_software/ASAP2_Hardware_Offloading_for_vSwitches_User_Manual_v4.4.pdf. The document tries to program tc flows between the PF and representer netdevs. I was able program the flow from representer netdev to PF(encapsulation flow) but when I try to program the tc flow from PF to representer netdev(decapsulation flow), it fails with the following error. RTNETLINK answers: Invalid argument We have an error talking to the kernel. I tried looking at the dmesg logs and journalctl -r but in vain. Below are the commands that I tried.

representer netdev to PF:  tc filter add dev enp6s0_1 protocol 0x806 parent ffff: flower skip_sw src_mac 36:ca:3d:75:01:f6 dst_mac 56:9a:db:7c:4c:25 action tunnel_key set src_ip 192.168.6.61 dst_ip 192.168.6.9 id 98 action mirred egress redirect dev enp6s0

PF to representer netdev: tc filter add dev enp6s0 protocol 0x806 parent ffff: flower skip_sw dst_mac 36:ca:3d:75:01:f6 src_mac 56:9a:db:7c:4c:25 enc_src_ip 192.168.6.9 enc_dst_ip 192.168.6.61 enc_key_id 98 enc_dst_port 4789 action tunnel_key unset action mirred egress redirect dev enp6s0_1

I enabled the switchdev-mode, basic-enacp & transport-inlink.

I am assuming there is no need of explicit vxlan type interface to be created in the host. Can any one please point me what field I am missing to program the flow or why the flows is failing. It is extremely painful as the tc logs doesn't give any information.

The command details as follows:

     192.168.6.9 & 192.168.6.61 are my underlay network ips.

     enp6s0 - PF

     enp6s0_1 - representer netdev

     src_mac & dst_mac : underlay networks macs if PF to rpresentor netdev else overlay VM macs.

Thanks.

Hi Brian,

Did you start ibping on the server side using different device? It is '-C' option.

Hi,

In general, the question is not related to Mellanox, but more to Infiniband SPEC and linux-rdma mailing list could be a better choice. And seems that you already discussing the subject using another forum, which is great by the way - https://www.rdmamojo.com/2013/02/15/ibv_poll_cq/

Instead of re-compiling RPMS, I would suggest to have a folder with sources of the libraries compiled with debug symbols and use LD_LIBRARY_PATH to load files from this folder. That gives you much more flexibility when debugging the issue as you can make the changes on the fly, recompile in a second and use it. Also it doesn't require root access in order to install RPM. You might use newer version of libibverbs, libxml5 and other libraries in that way.

The general instruction is to open a RPM, extract archive, configure with customer prefix and run make install. For run time, use LD_LIBRARY_PATH. For example,

$rpm2cio -id <RPM>

$tar xf <TAR>

$cd <CREATED FOLDER>

$./autogen.sh ( if necessary)
$./configure --prefix=<PREFIX> --enable-debug$make -j <N> install

export LD_LIBRARY_PATH=<PREFIX>/lib (or lib64)

Check if using latest firmware - 2.36.5000 for you device helps? 2.10.XXXX is extremely outdated. http://www.mellanox.com/page/firmware_table_dell_archive

As additional step, after installing 2.36 firmware, check if using MOFED-4.0 (or even 3.4 or using Inbox Infiniband package) makes the issue go away. Try to explicitly specify guid in opensm.conf configuration file or on the command line (opensm --guid <GUID>)

Thank you, appreciate the help! I'll work on this today and report. I'm using an older infiniband QDR switch M3601Q than the new M4001Q. I do know the standard firmware update failed when I was installing OFED, had to do a force install.

Using 4.16.13.5 on ESXi 6.5 with SR-IOV allows the guest to access IB fabric.

http://www.mellanox.com/related-docs/prod_software/Mellanox_MLNX-NATIVE-ESX-ConnectX-4-5_Driver_for_VMware_ESXi_6.5_Rele…

Hi Joel,

When running OEM HCA, using Mellanox firmware is not supported and Mellanox OFED has no firmware images for OEM cards. Hopefully, you didn't burn it and that's why there is a link to Dell archive.

Makes sense, thank you. I have downloaded everything I might need. The PSID DEL0A10210018 doesn't have a match, since the switch is older, picked couple to try. The firmware link for M3601Q switch leads to the newest OEM link you shared as well.

I'll give it all a good shot and report through weekend. Hoping this does the magic!

Hi, all

I have a trouble on CentOS 6.5

I create a scsi target used tgtd and iser, BUT disconnected frequently , its log like this:

Sep 15 12:16:09 almstor01 tgtd: iser_cm_disconnected(1632) conn:0x7f1ee0 cm_id:0x81b460 event:10, RDMA_CM_EVENT_DISCONNECTED

Sep 15 12:16:09 almstor01 tgtd: iser_conn_close(1291) conn:0x7f1ee0 cm_id:0x0x81b460 state: CLOSE, refcnt:395

Sep 15 12:16:09 almstor01 tgtd: iser_cm_disconnected(1632) conn:0x7f06a0 cm_id:0x7d29b0 event:10, RDMA_CM_EVENT_DISCONNECTED

Sep 15 12:16:09 almstor01 tgtd: iser_conn_close(1291) conn:0x7f06a0 cm_id:0x0x7d29b0 state: CLOSE, refcnt:395

Sep 15 12:16:09 almstor01 tgtd: iser_cm_disconnected(1632) conn:0x7f00e0 cm_id:0x807ce0 event:10, RDMA_CM_EVENT_DISCONNECTED

Sep 15 12:16:09 almstor01 tgtd: iser_conn_close(1291) conn:0x7f00e0 cm_id:0x0x807ce0 state: CLOSE, refcnt:395

Sep 15 12:16:09 almstor01 tgtd: iser_cm_disconnected(1632) conn:0x7f0f00 cm_id:0x7f0c60 event:10, RDMA_CM_EVENT_DISCONNECTED

Sep 15 12:16:09 almstor01 tgtd: iser_conn_close(1291) conn:0x7f0f00 cm_id:0x0x7f0c60 state: CLOSE, refcnt:389

Sep 15 12:16:09 almstor01 tgtd: iser_cm_disconnected(1632) conn:0x7d71c0 cm_id:0x7ed170 event:10, RDMA_CM_EVENT_DISCONNECTED

Sep 15 12:16:09 almstor01 tgtd: iser_conn_close(1291) conn:0x7d71c0 cm_id:0x0x7ed170 state: CLOSE, refcnt:390

Sep 15 12:16:09 almstor01 tgtd: iser_cm_disconnected(1632) conn:0x7ff380 cm_id:0x7ff0e0 event:10, RDMA_CM_EVENT_DISCONNECTED

Sep 15 12:16:09 almstor01 tgtd: iser_conn_close(1291) conn:0x7ff380 cm_id:0x0x7ff0e0 state: CLOSE, refcnt:391

Sep 15 12:16:09 almstor01 tgtd: iser_cm_disconnected(1632) conn:0x81bb10 cm_id:0x75c5e0 event:10, RDMA_CM_EVENT_DISCONNECTED

Sep 15 12:16:09 almstor01 tgtd: iser_conn_close(1291) conn:0x81bb10 cm_id:0x0x75c5e0 state: CLOSE, refcnt:403

Sep 15 12:16:09 almstor01 tgtd: iser_cm_disconnected(1632) conn:0x828100 cm_id:0x7ed800 event:10, RDMA_CM_EVENT_DISCONNECTED

Sep 15 12:16:09 almstor01 tgtd: iser_conn_close(1291) conn:0x828100 cm_id:0x0x7ed800 state: CLOSE, refcnt:416

Sep 15 12:16:10 almstor01 tgtd: iser_cm_conn_established(1612) conn:0x7d7780 cm_id:0x7d6f70, 192.168.100.4 -> 192.168.100.101, established

Sep 15 12:16:10 almstor01 tgtd: iser_cm_conn_established(1612) conn:0x8641b0 cm_id:0x7eceb0, 192.168.200.3 -> 192.168.200.101, established

Sep 15 12:16:11 almstor01 tgtd: iser_cm_conn_established(1612) conn:0x85fee0 cm_id:0x7d8650, 192.168.100.1 -> 192.168.100.101, established

Sep 15 12:16:11 almstor01 tgtd: iser_cm_conn_established(1612) conn:0x870dc0 cm_id:0x7d8b00, 192.168.200.1 -> 192.168.200.101, established

Sep 15 12:16:11 almstor01 tgtd: iser_cm_conn_established(1612) conn:0x8619e0 cm_id:0x861740, 192.168.200.2 -> 192.168.200.101, established

Sep 15 12:16:11 almstor01 tgtd: iser_cm_conn_established(1612) conn:0x862da0 cm_id:0x862b00, 192.168.100.2 -> 192.168.100.101, established

Sep 15 12:16:11 almstor01 tgtd: iser_cm_timewait_exit(1646) conn:0x7f1ee0 cm_id:0x81b460

Sep 15 12:16:11 almstor01 tgtd: iser_cm_timewait_exit(1646) conn:0x7f06a0 cm_id:0x7d29b0

Sep 15 12:16:11 almstor01 tgtd: iser_cm_timewait_exit(1646) conn:0x7f00e0 cm_id:0x807ce0

Sep 15 12:16:11 almstor01 tgtd: iser_cm_timewait_exit(1646) conn:0x7f0f00 cm_id:0x7f0c60

Sep 15 12:16:11 almstor01 tgtd: iser_cm_timewait_exit(1646) conn:0x7d71c0 cm_id:0x7ed170

Sep 15 12:16:11 almstor01 tgtd: iser_cm_timewait_exit(1646) conn:0x7ff380 cm_id:0x7ff0e0

Sep 15 12:16:11 almstor01 tgtd: iser_cm_timewait_exit(1646) conn:0x81bb10 cm_id:0x75c5e0

Sep 15 12:16:11 almstor01 tgtd: iser_cm_timewait_exit(1646) conn:0x828100 cm_id:0x7ed800

Sep 15 12:16:11 almstor01 tgtd: iser_cm_conn_established(1612) conn:0x7eeae0 cm_id:0x7d80d0, 192.168.100.3 -> 192.168.100.101, established

Sep 15 12:16:11 almstor01 tgtd: iser_cm_conn_established(1612) conn:0x7f00c0 cm_id:0x8636e0, 192.168.200.4 -> 192.168.200.101, established

what can I do something for fix the issue.

Thanks.

Please use latest version , 3.6.8010 ,  which is the best version