I’ve just made a breakthrough in my research into setting up an infiniband fabric at home. I started out at about 135MB/sec. made slow progress up to 200-220MB/sec, but this evening broke the 900MB/sec barrier. Almost 1 GIGABYTE per second. That’s well over a CD every second! Here’s the ATTO benchmark output…
In previous posts, I’d identified several bottlenecks, and as I got around each one, I slowly got the speed up. However, everything seemed to max out at 200-220MB/sec. I’d identified PCI-express slots as one potential problem but when I set up a ramdisk on the linux box and exported that as an SRP target, I was still only getting 200MB/sec. Then I thought – could it be my i7 Win7 box? Surely not. That’s sooo much more powerful than the athlon64 x2 3GHz box that I’d linux installed on. But for a laugh I had a look at the PCI express slot that I’d inserted the Mellanox Infiniband card into. Even though it’s a 16x sized slot, my jaw dropped when I saw PCIEx4 written on the motherboard beside it. But the Mellanox card is a PCIEx8 card! So I moved the card into the spare PCIEx16 slot, and re-ran my tests, only to smash all previous records, achieving almost 950MB/sec for some packet sizes.
Just to put this into context:
- 3GHz Athlon 64 dual core, Asus A8N-SLI Premium, 4Gig ram, Mellanox MHEA28-XTC 10Gbps Infiniband card.
- Running Ubuntu 11.10, 2Gig Ramdisk set up as an SRP target
- 3.20GHz Intel i7 quad core, Gigabyte GA-X58-USB3, 12Gig ram, Mellanox MHEA28-XTC 10Gbps Infiniband card
- Running Windows 7 Home Premium, OFED infiniband drivers package, SRP Initiator. ATTO Disk Benchmark utility.
Both machines are connected directly together, no switch was needed.
Consumer grade hardware for the most part, with the magical Mellanox ingredient that pushes this beyond what I previously though possible.
Also, it’s worth noting that this is a stock Ubuntu 11.10 install, a custom kernel was NOT required. I’m working on a HOWTO to get the Infiniband fabric up and SRP targets configured, I’ll be posting that soon. I reckon it’ll be possible to configure an Ubuntu box up to the 900MB/sec speeds in under 15 minutes.
Here’s the HDTune benchmark of the same setup. Not quite 900MB/sec, but not far off….
“You can learn from my mistakes” (putting the infiniband card in the wrong slot) :)
More to come soon…..
Here’s a shot of the Linux Box. It’s changed a bit since this photo was taken, there’s now a raid card in there where the graphics card is shown, but it give you and idea of the “consumer grade” hardware I’m doing this with….