See sections 5.1, 5.3 and 5.4 for a good step by step guidance for IB performance troubleshooting:
http://www.mellanox.com/related-docs/prod_software/MLNX_VPI_WinOF_User_Manual_v4.80.pdf
Have you done it? Also what are versions of fw/driver/switch you have and p/n for cables?
Ntttcp uses IPoIB and winsock and the actual problem can be masked in multiple software layers and might have nothing to do with your IB fabric. To exclude or pinpoint a problem with IB (if there is – with bad cables, bad adapters, etc.) you should start with pure IB tests using utilities like ib_send_bw (you should have similar results with nd_send/read utilities but I haven’t use them myself). As far as what can be tuned in BIOS and OS – see performance guide at http://www.mellanox.com/related-docs/prod_software/Performance_Tuning_Guide_for_Mellanox_Network_Adapters.pdf