I am trying to connect two machines with infiniband (MCX353A-QCBT) in Ubuntu 14.04. But I failed.
This is my state. (lspci, ifconfig, ibstat, ibnodes, /sbin/connectx_port_config, dmesg). Does anyone have any directions, links, or advice?
sangmuk@sclab1:~$ lspci | grep Mellanox
83:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]
sangmuk@sclab1:~$ ifconfig
eth0 Link encap:Ethernet ..
eth1 Link encap:Ethernet ..
eth2 Link encap:Ethernet HWaddr 7c:fe:90:16:8b:50
inet6 addr: fe80::7efe:90ff:fe16:8b50/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:126402 errors:0 dropped:0 overruns:0 frame:0
TX packets:126585 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:23725053 (23.7 MB) TX bytes:23771526 (23.7 MB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:390193 errors:0 dropped:0 overruns:0 frame:0
TX packets:390193 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:26791107 (26.7 MB) TX bytes:26791107 (26.7 MB)
sangmuk@sclab1:~$ ibstat
CA 'mlx4_0'
CA type: MT4099
Number of ports: 1
Firmware version: 2.36.5000
Hardware version: 1
Node GUID: 0x7cfe900300168b50
System image GUID: 0x7cfe900300168b53
Port 1:
State: Active
Physical state: LinkUp
Rate: 10
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x0c010000
Port GUID: 0x7efe90fffe168b50
Link layer: Ethernet
sangmuk@sclab1:~$ ibnodes
ibwarn: [27593] mad_rpc_open_port: client_register for mgmt 1 failed
src/ibnetdisc.c:766; can't open MAD port ((null):0)
/usr/sbin/ibnetdiscover: iberror: failed: discover failed
ibwarn: [27598] mad_rpc_open_port: client_register for mgmt 1 failed
src/ibnetdisc.c:766; can't open MAD port ((null):0)
/usr/sbin/ibnetdiscover: iberror: failed: discover failed
sangmuk@sclab1:~$ ibhosts
ibwarn: [27635] mad_rpc_open_port: client_register for mgmt 1 failed
src/ibnetdisc.c:766; can't open MAD port ((null):0)
/usr/sbin/ibnetdiscover: iberror: failed: discover failed
sangmuk@sclab1:~$ sudo /sbin/connectx_port_config
ConnectX PCI devices :
|----------------------------|
| 1 0000:83:00.0 |
|----------------------------|
Before port change:
eth
|----------------------------|
| Possible port modes: |
| 1: Infiniband |
| 2: Ethernet |
| 3: AutoSense |
|----------------------------|
Select mode for port 1 (1,2,3): 1
WARNING: Illegal port configuration attempted,
Please view dmesg for details.
sangmuk@sclab1:~$ dmesg
...
[338561.776021] mlx4_core 0000:83:00.0: Requested port type for port 1 is not supported on this HCA