Skip to main content

RDMA with QLogic FastLinQ 41xxx and 45xxx (qedentv) on ESXi

So, all the infos I'm giving here can be found in VMware KBs or on the HPE support site, it just took me literal days to find them. My problem was to get RoCEv2 running with the qedentv driver on ESXi, the driver alone doesn't support RDMA. However, the driver comes in a package of 4 drivers or modules where qedrntv does support RDMA. I suspect the "e" in qedentv stands for ethernet and the "r" in qedrntv stands for RDMA? Either way, the solution sounds quite simple, swap out qedentv with qedrntv as the active driver and done. Turns out its not that easy or I wouldn't be writing this...

As far as I understand it now, the actual driver is edentv and the other files are modules for the driver which can be loaded to support more features. First off, I would suggest installing the driver package that matches your card's firmware version. You can usually find the firmware version in the BIOS or in the status/hardware section of your RMM WebUI. To find and download the driver open the VMware Compatibility Guide (I/O device search), now you could search for the driver name or card name. The more precise option is to search for the VID, DID, SVID and SSID. You can get these with the command

vmkchdev -l | grep vmnic0

where vmnic0 is one of the vmnics of the QLogic card. The format is VID:DID SVID:SSID

/images/rdma_with_qlogic/vmkchdev.png

So in this example it is

VID: 1077
DID: 8070
SVID: 1590
SSID: 021d

After you click search, click on the model and choose your ESXi version. Check the firmware version you got ealier and download the matching driver package. Install the package e.g. via VMware Lifecycle Manager or CLI.

After installing/updating the driver we can activate the RDMA module with a simple command

esxcfg-module -s 'enable_roce=1' qedentv

then restart the ESXi host. After the restart you should the see "RDMA adapters" tab on the left side.

/images/rdma_with_qlogic/rdma_adapters.png

Job done and time for a coffee :)

Reference