The word Gateway in general used in many cases across the industry. in this case we use it when connecting IB networks to Ethernet networks. The gateway is not IP router (nor IB router) and have QPs needed to the IB side.
Assuming your servers send IPoIB traffic, You need a gateway to transfer IPoIB traffic to IPoEth and a router (default gateway) to terminate the IP packets and pass it to a different network (e.g. in the Enterprise, probably you will have some kind of firewall or NAT services before going into the ISP.
See also this document - see the ARP flow.
Thanks,
Ophir.