This article will help you enable Infiniband and RDMA support in Linux using the upstream drivers or the MLNX OFED drivers from Mellanox.
RHEL/CentOS 7
Upstream Drivers
Install the relevant packages
# yum groups install Infiniband\ Support
# yum install opensm rdma infiniband-diags
Enable subnet manager on all nodes
# systemctl enable opensm --now
NOTE: opensm only needs to run on a single node in the fabric. I'm enabling it on all nodes for redundancy. opensm will detect that other instances are running in the fabric and shut itself down.
At this point, you should have a working infiniband fabric. However, you can create TCP/IP interfaces over the Infiniband network (IPoIB) by creating configuration files in /etc/sysconfig/network-scripts similar to this:
DEVICE=ib0
TYPE=Infiniband
BOOTPROTO=static
BROADCAST=192.168.0.255
IPADDR=192.168.0.1
NETMASK=255.255.255.0
NETWORK=192.168.0.0
MTU=65520
ONBOOT=yes
Mellanox (MLNX OFED)
Download and install the OFED stack from MLNX:
https://www.mellanox.com/products/infiniband-drivers/linux/mlnx_ofed
The downloaded package should come with instructions and an installation script. Please see the README there for details.
After installed, enable the opensmd (same as opensm in upstream instructions) on all nodes:
# systemctl enable opensmd --now
You can use the same IPoIB configuration from the upstream instructions to configure TCP/IP on top of these devices as well.
Reviewed 2020/12/01 - DGT