This is an automated archive made by the Lemmit Bot.

The original was posted on /r/homelab by /u/ConstructionSafe2814 on 2023-08-03 22:07:03.

OK so I have a c7000 enclosure in my home lab with a QDR BladeSystem Infiniband 40Gbps switch attached to it. The (BL460c gen8) blades have a 544M mezzanine QDR Mellanox ConnectX4 card in them. Then I used IPoIB to connect 4 hosts (mtu 64k)Yay so far so good

I put a 3.7GB ISO file on tmpfs (RAM) and exported it over NFS. Then I mounted the drive and cp’d it to the NFS client, also to RAM to avoid HDD bottlenecks.

I am “only” able to reach 1.2GB/s file transfer ratio. The bottleneck sits on the client side and is a single core saturated in kernel space. It’s always the last core of one of the 2 CPU’s installed in the blade.

I might try NFS over UDP? That might be friendlier to the CPU. But I noticed it’s NFSv4.2. Not sure if it can go over UDP.

IPoIB might not be ideal for performance. If I choose a higher clocked CPU, I will squeeze out a couple of 100MB/s or so. But how do I get 40Gbps? I’m roughly at one fourth/fifth now.

Would mounting NFS over RDMA be better? Or what other use cases transfer data from one host to another much faster?