I’ve just made a breakthrough in my research into setting up an infiniband fabric at home. I started out at about 135MB/sec. made slow progress up to 200-220MB/sec, but this evening broke the 900MB/sec barrier. Almost 1 GIGABYTE per second. That’s well over a CD every second! Here’s the ATTO benchmark output…
In previous posts, I’d identified several bottlenecks, and as I got around each one, I slowly got the speed up. However, everything seemed to max out at 200-220MB/sec. I’d identified PCI-express slots as one potential problem but when I set up a ramdisk on the linux box and exported that as an SRP target, I was still only getting 200MB/sec. Then I thought – could it be my i7 Win7 box? Surely not. That’s sooo much more powerful than the athlon64 x2 3GHz box that I’d linux installed on. But for a laugh I had a look at the PCI express slot that I’d inserted the Mellanox Infiniband card into. Even though it’s a 16x sized slot, my jaw dropped when I saw PCIEx4 written on the motherboard beside it. But the Mellanox card is a PCIEx8 card! So I moved the card into the spare PCIEx16 slot, and re-ran my tests, only to smash all previous records, achieving almost 950MB/sec for some packet sizes.
Just to put this into context:
Machine A:
- 3GHz Athlon 64 dual core, Asus A8N-SLI Premium, 4Gig ram, Mellanox MHEA28-XTC 10Gbps Infiniband card.
- Running Ubuntu 11.10, 2Gig Ramdisk set up as an SRP target
Machine B:
- 3.20GHz Intel i7 quad core, Gigabyte GA-X58-USB3, 12Gig ram, Mellanox MHEA28-XTC 10Gbps Infiniband card
- Running Windows 7 Home Premium, OFED infiniband drivers package, SRP Initiator. ATTO Disk Benchmark utility.
Both machines are connected directly together, no switch was needed.
Consumer grade hardware for the most part, with the magical Mellanox ingredient that pushes this beyond what I previously though possible.
Also, it’s worth noting that this is a stock Ubuntu 11.10 install, a custom kernel was NOT required. I’m working on a HOWTO to get the Infiniband fabric up and SRP targets configured, I’ll be posting that soon. I reckon it’ll be possible to configure an Ubuntu box up to the 900MB/sec speeds in under 15 minutes.
Here’s the HDTune benchmark of the same setup. Not quite 900MB/sec, but not far off….
“You can learn from my mistakes” (putting the infiniband card in the wrong slot) :)
More to come soon…..
Additional Notes:
Here’s a shot of the Linux Box. It’s changed a bit since this photo was taken, there’s now a raid card in there where the graphics card is shown, but it give you and idea of the “consumer grade” hardware I’m doing this with….
Tags: backup, Bandwidth, HPC, Infiniband, RAID, throughput
[...] the IPoIB bottleneck is removed, and it’s possible to get the throughput up past 700MB/sec. Using RAMDisk I was able to pull over 900MB/sec from the SRP target to the Windows initiator. Care needs to be taken in the selection of PCI-express slots, though, and you need to understand [...]
[...] gone to all the trouble of getting a 900MB/sec SRP target working, I was interested to go back and see how a samba mount would perform now that I’d [...]
Excellents posts about infinband, thank you for that!
Hope to read more from you about this subject.
Regards from the Netherlands,
IJzerboot
David,
I found this information VERY useful, in particular with the information regarding the costs of an infiniband setup as well as real-world transfer rates. This is a huge help indeed.
I’m hitting a little bit of a wall, however. While I understand that I’ll need to rely on the SRP protocol for a infiniband links, here’s my problem:
I’ve planned to set up in a second system (at least a quad-core xeon) a RAMDISK of at least 120 GB (hopefully DDR3, triple-channel if possible). I know that’ll be quite costly, but I wish to use the RAMDISK for booting into a Windows 7 install. Sounds nuts eh? I’ll use the server for all my storage while I plan on having my regular system diskless; it’ll only contain one (or two) graphics card(s) and at least one infiniband adapter. I plan on using the famous iPXE for this (iSCSI or AoE is still something I’m debating over).
I need your input on this. Can this be done? I’d love it if I could somehow bootstrap Windows directly in a RAMDISK, it’d make performance lightening fast.
Finally, I know it sounds stupid but I’d appreciate a reply. Thanks!
Blitz
Why not use one of these: http://www.overclockers.co.uk/showproduct.php?prodid=HD-085-OC
They give about 1Gb/sec read/write, and are probably a helluva lot cheaper than putting 120GB of RAM into a machine with an infiniband setup. Even if you have 10 machines, it would probably be cheaper to stick a 120GB Revodrive in all of them
daveh,
quite honestly i find flash spiteful. limited write cycles, plus if i have an os on one then wear happens much faster. admittedly, ssd is low on power and connects directly with the pci-e interface, but in the end flash is still flash.
in the case of a ramdrive, yes it’s much more expensive and it’s a lot more power-hungry, but the performance ratings are considerably better. hell, with a qdr 4x infiniband connection i could potentially reach the ram’s bottleneck, or close to it.
also, ddr4 is about to hit the shelves. imagine a ramdrive based on ddr4 tech. it’d be a huge speed jump.
ssd is a great idea for storage solutions. for constantly changing data, flash-based ssd is epic fail (at least to me anyway).
also, IT CAN BE DONE!! that alone is reason enough for me, doing things that people can’t be bothered to do.
blitz
correction to my last post, i just checked ram transfer rates and for ddr3 they’re HUGE, starting with 6.4 GB/s (approximately, more info here http://www.quepublishing.com/articles/article.aspx?p=1416688&seqNum=2 ). so yeah, even a 12x edr infiniband would bottleneck with a server providing triple-channel 1333 ddr3 ram……. xO i’m going bonkers just thinking of the numbers (of course real world scenarios are always different)
blitz