If using HPC-X or OpenMPI from the Mellanox OFED, you might add '-mca btl self -mca pml ucx'
↧
Re: How can I add a timestamp in Roce?
↧
Let me know which os is stable for SB7800
Hello
Recently delivered sb7800 to our customers.
The customer requested that the SB7800 OS be installed with the most stable version of OS and wants to be recommended by the vendor.
Please advise me of the most stable os of the sb7800 devices that have come to date.
Thank you.
↧
↧
How to enable the MT4103(connectx-3 pro) physical port
It is found that when configuring the adapter as RoCEv2 by “options mlx4_core roce_mode=0000:81:00.0-1,0000:84:00.0-1”, after OS rebooting, one MT4103 can not work well. But there is nothing wrong for another one with the same firmware.
The port state always shows “Disabled" as below:
# ibstat mlx4_1
CA ‘mlx4_1’
CA type: MT4103
Number of ports: 1
Firmware version: 2.42.5000
Hardware version: 0
Node GUID: 0xe41d2d030029ecb0
System image GUID: 0xe41d2d030029ecb0
Port 1:
State: Down
Physical state: Disabled
Rate: 10
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x04010000
Port GUID: 0xe61d2dfffe29ecb0
Link layer: Ethernet
And related ethnet interface’s link can not be detectable, but actually the cable is connected.
# ethtool ens259
Settings for ens259:
Supported ports: [ FIBRE ]
Supported link modes: 1000baseKX/Full
Supported pause frame use: Symmetric Receive-only
Supports auto-negotiation: No
Advertised link modes: 1000baseKX/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: No
Speed: Unknown!
Duplex: Unknown! (255)
Port: FIBRE
PHYAD: 0
Transceiver: internal
Auto-negotiation: off
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000014 (20)
link ifdown
Link detected: no
I have tried OFED driver of MLNX_OFED_LINUX-4.4-1.0.0.0-rhel7.4-x86_64 and MLNX_OFED_LINUX-4.4-2.0.7.0-rhel7.4-x86_64. Neither can work.
The switch is Dell 4048-on.
So, what is the problem with one MT4103? How can I enable the physical port?
↧
testing beegfs with rdma using connectx-3 pro
I have used vdbench and fio to test for RDMA connections and TCP connections.
But the weird is ,RDMA performence is worse than TCP.
Hardware and Software Used:
two machines with no switch and the cards are plugged in back to back
HCA: Ethernet controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]
os: CentOS Linux release 7.4.1708 (Core) 3.10.0-693.el7.x86_64
OFED: MLNX_OFED_LINUX-4.4-2.0.7.0-rhel7.4-x86_64.iso
the parameter file is:
messagescan=no
hd=default,vdbench=/root/download/vdbench50407,user=root,shell=ssh
hd=hd1,system=stor01
hd=hd2,system=stor02
fsd=fsd1,anchor=/mnt/beegfs/largetest2,depth=1,width=5,files=10,size=100m,shared=yes
fwd=format,threads=16,xfersize=1m
fwd=default,xfersize=1m,fileio=random,fileselect=random,rdpct=60,threads=16
fwd=fwd1,fsd=fsd1,host=hd1
fwd=fwd2,fsd=fsd1,host=hd2
rd=rd1,fwd=fwd*,fwdrate=max,format=restart,elapsed=600,interval=1
the result is:
| largetest 100M | RDMA | TCP |
| read(MB/s) | 1322 | 2008 |
| write(MB/s) | 883.5 | 1342 |
| total(MB/s) | 2205.9 | 3351.5 |
anyone has tested beegfs for the difference between RDMA and TCP?
↧
Up to $200 off! sfpcables lower cost shop for datacenter
SFPcables.com is an official store of 10Gtek company. It offers one-stop Datacenter solution and products, including compatible Transceivers, DAC cables, AOC cables, Fiber Optical, Patch-cord, CWDM/DWDM passive/active systems, Network Adapters and more. As a commitment to offering high-quality, reliable and customized products, SFPcables.com develops and tests new products to meet the growing Data Centers and High Performance Computing market needs. We guarantee only premium materials and quality engineering are used in the designing and manufacturing for the fiber and cooper products of SFPcables.com and the whole production process are under the highest quality control.
In addition to our extensive product line, our customer service has earned us a reputation of trust that is unparalleled in the industry. SFPcables.com holds thousands of cables and transceivers in stock to ensure 24-hour shipping for all orders.
SFPcables.com invests heavily in technology, with a Compatibility Test Lab full of the latest brands of equipment, SFPcables.com ensures its precise programming for various brands of switches, servers and routers.
We help clients navigate the complexities of their hardware architecture to guarantee compliance throughout the network.
Our service success is proven by the loyalty and support of some of the largest data centers in the world, including:telecommunications, corporations, government agencies and reputable distributors.
we have a feedback for our customer’s support. as you see on the pic.
up to $200 off !!!
Take more save more, sfpcables is a shop store of 10Gtek. we always have exhabition all over the world, all our products are with Certification. 100% high quality guaranteed.
Our special sell:
↧
↧
Re: How to configure host chaining for ConnectX-5 VPI
Just some due diligence here.
We put our ConnectX5 cards in our 3 host vmware 6.5 stack, and did not get it to work with host_chaining. We ended up contacting support about it, and the reply we got wasn't optimistic.
"Host-chaining is currently not supported as it is not GA for ESXi."
So my previous post was a grain of salt, and marked out accordingly.
I have yet to see *any* documentation on host_chaining specifically; which is really sad, since As far as I know, my post above is the best available.
↧
Re: How to configure host chaining for ConnectX-5 VPI
You're welcome!
I'm glad I helped someone after all the headache I went through for it.
I have no hard experience with VMWare, and so take all of this with a grain of salt.
First thought is vlan tags. I was told that VMWare tags by default.
From my (limited) understanding and thoughts, host chaining inside VMware is not a good idea.
If you setup a virtual switch (on the vmware side) and put both ports of the card on the switch, give that switch an IP, that would allow for vmotion and such over the link at close to line speed. Letting the switch (analogous to openvswitch) do all of the routing, and fast pathing.
Thoughts - If there was host chaining:
Vmware still sees both ports (we can't assign IPs to raw port interfaces to start with.)
It doesn't really know which port to send out, so it could take the extra hop before it gets to the destination.
Three node, desired going from A -> B might take the path of A -> C -> B
Where I can talk is non-chaining speed.
We did try using openswitch and the cards with chaining off. So long as the stp stuff is turned on; we got nearly line speed.
We opened a support ticket for our problems with MTU. It took a while, but we found the problem.
They have a nice little utility (sysinfo-snapshot) for seeing the card internals and OS config options which helped us (by looking through it.)
See my post below. Host_chaining is not supported on ESXi at this time.
↧
How can I enable "packet pacing" on connectX-5 ?
How can I enable "packet pacing" on the connectX-5 ? With the command ibv_devinfo -v I get Packet_pacing_caps: qp_rate_limits_max: 0kbps I expect here the wire rate of the card.
Thanks Patrick
↧
Re: Can't ibping Lid or GUID but can ping by ip
Thank you for responding quickly.
I am able to ibping to the gid on first dev but not on the second one:
SERVER:
-----------------------
# show_gids
DEV PORT INDEX GID IPv4 VER DEV
--- ---- ----- --- ------------ --- ---
mlx5_0 1 0 fe80:0000:0000:0000:248a:0703:0014:f9ac v1
mlx5_1 1 0 fe80:0000:0000:0000:248a:0703:0014:f850 v1
n_gids_found=2
CLIENT:
name@server:/etc/infiniband$ ibping --dgid fe80:0000:0000:0000:248a:0703:0014:f9ac 8
Pong from centos-dgx1.brane.systems.(none) (Lid 8 Gid fe80::248a:703:14:f9ac): time 0.109 ms
Pong from centos-dgx1.brane.systems.(none) (Lid 8 Gid fe80::248a:703:14:f9ac): time 0.095 ms
Pong from centos-dgx1.brane.systems.(none) (Lid 8 Gid fe80::248a:703:14:f9ac): time 0.139 ms
Pong from centos-dgx1.brane.systems.(none) (Lid 8 Gid fe80::248a:703:14:f9ac): time 0.174 ms
Pong from centos-dgx1.brane.systems.(none) (Lid 8 Gid fe80::248a:703:14:f9ac): time 0.159 ms
Pong from centos-dgx1.brane.systems.(none) (Lid 8 Gid fe80::248a:703:14:f9ac): time 0.190 ms
Pong from centos-dgx1.brane.systems.(none) (Lid 8 Gid fe80::248a:703:14:f9ac): time 0.169 ms
Pong from centos-dgx1.brane.systems.(none) (Lid 8 Gid fe80::248a:703:14:f9ac): time 0.163 ms
^Z[6] Killed ibping 8
[7] Killed ibping -S
[8]+ Stopped ibping --dgid fe80:0000:0000:0000:248a:0703:0014:f9ac 8
name@server:/etc/infiniband$ ibping --dgid fe80:0000:0000:0000:248a:0703:0014:f850 8
ibwarn: [47999] mad_rpc_rmpp: _do_madrpc failed; dport (Lid 8 Gid fe80::248a:703:14:f850)
ibwarn: [47999] mad_rpc_rmpp: _do_madrpc failed; dport (Lid 8 Gid fe80::248a:703:14:f850)
ibwarn: [47999] mad_rpc_rmpp: _do_madrpc failed; dport (Lid 8 Gid fe80::248a:703:14:f850)
ibwarn: [47999] mad_rpc_rmpp: _do_madrpc failed; dport (Lid 8 Gid fe80::248a:703:14:f850)
ibwarn: [47999] mad_rpc_rmpp: _do_madrpc failed; dport (Lid 8 Gid fe80::248a:703:14:f850)
ibwarn: [47999] mad_rpc_rmpp: _do_madrpc failed; dport (Lid 8 Gid fe80::248a:703:14:f850)
ibwarn: [47999] mad_rpc_rmpp: _do_madrpc failed; dport (Lid 8 Gid fe80::248a:703:14:f850)
^Z
[9]+ Stopped ibping --dgid fe80:0000:0000:0000:248a:0703:0014:f850 8
How can ibping the other gids?
Thanks
Brian
↧
↧
LLDP via SNMP Windows 2012 Server Ethernet NIC
i have a Windows server 2012 R2 machine with one of your Mellanox NIC and I want to be able to access LLDP information via SNMP, do you supply a MIB for this purpose or is there another way to get this information?
↧
mlx5 IPoIB not working in connected mode
Hello,
I am trying to have my hosts connected on my infiniband network with mlx5 cards in connected mode but IPoIB is not working.
CONNECTED MODE is mandatory in my environment.
I disabled IPoIB enhanced mode
options ib_ipoib ipoib_enhanced=0
in this way I configured ib0 as a connected mode IPoIB
ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 qdisc pfifo_fast state UP group default qlen 256
link/infiniband 80:00:00:86:fe:80:00:00:00:00:00:00:50:6b:4b:03:00:42:e7:b4 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
inet 172.21.52.144/22 brd 172.21.55.255 scope global ib0
while ibping works, ping itself does not work and I Am not able to use the interfaces.
server 1: 172.21.52.144
server 2: 172.21.52.145
they cannot ping each other while they can ibping each other.
my systems are RHEL 7.5 3.10.0-862.11.6.el7.x86_64
here is the info of my mlx5 card.
how can I enable connected mode to work on these interfaces?
CA 'mlx5_0'
CA type: MT4115
Number of ports: 1
Firmware version: 12.23.1020
Hardware version: 0
Node GUID: 0x506b4b030042e7b4
System image GUID: 0x506b4b030042e7b4
Port 1:
State: Active
Physical state: LinkUp
Rate: 100
Base lid: 3
LMC: 0
SM lid: 1
Capability mask: 0x2659e848
Port GUID: 0x506b4b030042e7b4
Link layer: InfiniBand
thank you
↧
Here free to take 10 OM3-LC-LC-D.
if you need purchase some optical items. you may have a check. factory outlets! 100% quality certification!
even save up to $200 off!
↧
Re: mlx5 IPoIB not working in connected mode
Hello Riccardo,
Thank you for posting your question on the Mellanox Community.
As you also opened a support case with use regarding this issue, we will continue to update you through the support case.
Thanks and regards,
~Mellanox Technical Support
↧
↧
ConnectX-4 in IB mode in ESXi
Hi.
Can I connect InfiniBand Adapter in ESXi 6.X in IB mode? Not ethernet.
So that, VM can work with this adapter.
I can't find information.
Or, which virtualization better use for this task?
↧
Hardware support for PTP in ConnectX-5
Hi,
ConnectX-5 firmware release notes indicate that PTP is an unsupported feature (e.g., http://www.mellanox.com/pdf/firmware/ConnectX5-FW-16_23_1020-release_notes.pdf ). Does this mean there is no hardware support for PTP in ConnectX-5? I'd like to synchronize NIC device clocks in my network that uses ConnectX-5.
Thanks
Radhika
↧
Let me know which os is stable for SB7800
Hello
Recently delivered sb7800 to our customers.
The customer requested that the SB7800 OS be installed with the most stable version of OS and wants to be recommended by the vendor.
Please advise me of the most stable os of the sb7800 devices that have come to date.
Thank you.
↧
Re: Let me know which os is stable for SB7800
Hi Shin,
The latest Mellanox OS available for SB7800 is 3.6.8010
You can find the image on myMellanox.
Thanks,
Pratik Pande
↧
↧
When will an ACK generated in RDMA write?
Hi,
Recently, I am doing some RDMA write latency test with ConnectX-4 Lx 25G NIC.
And I have two questions about the testing:
- What is the version of RDMA? Is the default RoCEv2?
- By default, the RC(reliable) mode is chosen. As required, an ACK is needed from the remote to local, then the local will add an entry to the CQ and the software then know the data has arrived at the remote. I searched with google but didn't find when a MLNX NIC will generate the ACK. I mean, is it generated when the data received by the HCA or after DMA to the host memory? And I assume the ACK is automatically generated by the NIC adapter without any SW involvement. Is my understanding right?
Many thanks
↧
Re: ASAP2 Live Migration & H/W LAG
Hi Zhang,
I'm looking for the best way of using ASAP2 with OpenStack where NIC has bonded two ports.
Best regards,
↧
How to enable the debuginfo for the libraries of OFED?
Hi,
I am debugging an issue and the function call tracing is needed. But with some debugging tools like perf, no symbol could be found. "nm" tool couldn't get any symbol.
I checked the mlnxofedinstall script and there is a list of the debuginfo files for CentOS, but in the OFED downloaded from the official website, no such package could be found and installed, even with the "--add-kernel-support". (e.g. the libs of mlx5 and ibverbs)
Several years ago, someone asked such question but no answer. So could some expert share the step to rebuild the packages with debuginfo to make the debugging convenient.
Many thanks
↧