NFSoRDMA incorrect mount option

[Elliott]

Hello. I'm trying to mount a Dell Isilon server using NFSoRDMA and it fails with an error about invalid argument. Unfortunately there's no more detail about which argument is bad. If I remove the 'rdma' option then it works fine.

Code: Select all

# mount -o rdma,vers=3 10.49.54.136:/ifs/data/test /mnt/isilonrdma/ -vvv
mount.nfs: timeout set for Wed Jun  1 16:25:48 2022
mount.nfs: trying text-based options 'rdma,vers=3,addr=10.49.54.136'
mount.nfs: mount(2): Invalid argument
mount.nfs: an incorrect mount option was specified

I'm running Centos 8.5.2111 with a Mellanox ConnectX-6 Dx 100Gb NIC and the latest OFED driver. I verified RDMA connectivity is working with 'ucmatose'. Is there some way to get more information about this error? I didn't see anything in /var/log/messages or dmesg. I'm using v3 because that's the only version supported by the Isilon server.

[BShT]

# mount -o proto=rdma,port=20049 host:/export /mnt

[Elliott]

That is the correct port number according to Dell but I get the same error that way.

[TrevorH]

I'm running Centos 8.5.2111

Which, unfortunately, is dead and will never receive any more updates. You should look to migrate to one of the alternatives ASAP. It may not help with this problem or even have anything to do with it but CentOS Linux 8 stopped receiving any updates at the end of 2021 so is getting more and more out of date. RHEL and all the clones are now on 8.6 so you are one full point release behind.

For the problem itself, assumging that there is very little other traffic going from this machine to the NFSoRDMA target, I would fire up tcpdump/wireshark and capture some traffic so you can see what it's trying to reach on what ports and what replies (if any) it's receiving.

[jlehtone]

I have a faint recollection that one had to change some config (in CentOS 7) before the NFSoRDMA was enabled.
Also a recollection that there was a bug in RDMA that did corrupt NFS in CentOS 7.

[Elliott]

tcpdump shows nothing at all. The command fails almost immediately, as shown in strace:

Code: Select all

# strace mount -vvv -o rdma,vers=3,port=20049 10.49.54.136:/ifs/data/test /mnt/isilonrdma/
execve("/bin/mount", ["mount", "-vvv", "-o", "rdma,vers=3,port=20049", "10.49.54.136:/ifs/data/test", "/mnt/isilonrdma/"], 0x7ffd5e071878 /* 25 vars */) = 0
brk(NULL)                               = 0x557e8abbc000
arch_prctl(0x3001 /* ARCH_??? */, 0x7ffd2ec64ed0) = -1 EINVAL (Invalid argument)

Trevor, I'm curious why you recommend those other distros instead of CentOS Stream?

[jlehtone]

The RDMA traffic is not TCP, is it? Not typical Ethernet traffic at all, which the tcpdump presumably listens.

Trevor, I'm curious why you recommend those other distros instead of CentOS Stream?

If you have had CentOS Linux, then you have had system that is bug-for-bug compatible with RHEL. "Those other distros" are that too. That would be a minimal change in practice to your current OS -- just different name and logo.

CentOS Stream is not bug-for-bug compatible with RHEL. It is a development branch, a "trunk", a preview from "upstream" of what RHEL might get in next point update. That (from CentOS Linux to CentOS Stream) is in practice much larger leap.

It is of course up to you what kind of distro you want to use, but please switch to something that is not EOL, like the 8.5.

[Elliott]

I believe the initial mounting does happen over TCP, and RDMA is used only for the data transfer. So I would expect to see some activity with tcpdump here. Since this is just a test machine, I upgraded the kernel to 5.18 and that fixed it! I'm surprised, as I thought this was a problem with nfs-utils. Now I'll have to think about the implications of upgrading kernels on other machines in prod.

Thanks for that explanation on stream. It sounds like Rocky is becoming more popular in the M&E industry so I'll consider going that direction.

[TrevorH]

CentOS Stream is now "ahead" of RHEL so it gets patches first in a lot of cases. And those cases seem to often be broken and either yum update fails because of repo problems (missing prereqs or wrong version/release numbers) or worse, the patch installs and is then broken and a few days/weeks later there is a subsequent update to the same component that fixes the brokeness.

In addition and much worse, security updates for Stream lag behind RHEL by a considerable margin sometimes - there was a kernel update in January that hit RHEL and the clones were all patched within 2 days and yet Stream had to wait 3 weeks for a new kernel to appear that fixed it. A similar story happened with an Apache httpd update that was out for RHEL and fixed several CVEs, again available for the other clones within days and Stream users had to wait 71 days for it.

It's not a distro that I would use in production. It might be great if you are one of the 1% of users that needs access to what's coming in the next point release of RHEL so you can have your packages available to your users on the day that the new RHEL point release drops. For the rest of us, it's just not good enough.

[Elliott]

Kernel upgrade will not be feasible in this situation. Is there any way to get additional logging to help identify the source of this problem? The "invalid argument" doesn't make much sense to me, since NFSoRDMA should be supported all the way back to el6.