Thanks. After a clean install with Ubuntu 13.04 and Mellanox OFED 2.0.5 I was able to get pinging working. When I try to run OpenSM on the second port with the port 0 to port 0 and port 1 to port 1 configuration, iblinkinfo seems to always show only 1 set of links. When I try to use port 1 to port 1 ( which works by itself) it says
ibwarn: [3161] ib_path_query_via: sa call path_query failed
ibwarn: [3161] main: Failed to resolve 0x0002c903009f5742; attempting full scan
I have assigned two ib0 and ib1 to two different subnets. I will try the cross-over configuration and see if that helps.