Just to add to yairi's note, you can also flip from IB to Eth by setting this module parameter:
options mlx4_core port_type_array="2,2"
or write directly in procfs:
echo eth > /sys/bus/pci/devices/0000\:20\:00.0/mlx4_port2
echo eth > /sys/bus/pci/devices/0000\:20\:00.0/mlx4_port1
