Here’s some more ramblings on 10G Networking on the Cheap.
Having gone to all the trouble of getting a 900MB/sec SRP target working, I was interested to go back and see how a samba mount would perform now that I’d sorted out all the cards into their optimum PCI-express slots.
So, I did the same setup as the 900MB/sec SRP test, Ramdisk to Ramdisk. This time, I set up a 2Gig ramdisk as a samba mount, and copied large files to and from that ramdisk from a 4G Ramdisk on the Windows side. ATTO disk benchmark doesn’t test network drives, so I had to make do with copying large files and keeping an eye on the throughput with dstat.
I initially measured the IPoIB throughput using netperf, than this gave me about 6Gbits.
daveh@raid:~$ netperf -H 10.4.12.2 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.4.12.2 (10.4.12.2) port 0 AF_INET : demo Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec
8192 16384 16384 10.00 6022.50
Based on that number, I was not expecting anywhere close to the 900MB/sec throughput I was getting with SRP. Anyway, just to compare samba throughput with SRP, I pressed on.
While copying a 1 GB video file to the ramdisk on the linux box (write)
usr sys idl wai hiq siq| read writ| recv send 3 57 31 0 0 9| 0 0 | 191M 565k 3 61 28 0 0 8| 0 0 | 188M 556k 4 56 31 0 0 9|4096B 40k| 192M 568k 4 59 32 0 0 6| 0 0 | 163M 485k 5 56 29 0 0 9| 0 0 | 186M 6419k 3 61 29 0 0 8| 0 0 | 187M 6519k 2 64 25 0 0 9| 0 0 | 183M 543k
While copying a 1 GB video file from the ramdisk on the linux box (read) usr sys idl wai hiq siq| read writ| recv send 3 40 54 0 0 4| 0 4096B| 863k 251M 6 42 47 1 0 5| 0 52k| 910k 263M 5 44 48 0 0 5| 0 0 | 816k 233M 5 42 46 0 0 6| 0 0 | 812k 231M 4 45 48 0 0 4| 0 0 | 826k 236M 3 44 49 0 0 5| 0 0 | 819k 234M 4 44 48 0 0 5| 0 0 | 832k 236M
I know it’s not very scientific, and I didn’t spend any time tuning samba. But the 900MB/sec throughput is practically out-of-the-box for SRP, so why not test samba at it’s out-of-the-box settings. 
So, even with netperf showing a 6Gbit throughput, I was only seeing around 190-235MB/sec throughput over samba (at 40-60% CPU usage). So I think I’ll stick with my SRP target setup rather than going back to samba. With SRP, I know that whatever disk subsystem I put into the linux box, it probably won’t be able to keep up with the 900MB/sec that the link is capable of. Unless, that is, I buy a very expensive RAID card and a bunch more disks. And with the way disk prices have gone over the last few months with the floods in Thailand, it’ll be a while until I even think about doing that.
Tags: backup, Bandwidth, Infiniband, RAID, srp, Storage, throughput
Have you tried to boot from SRP yet?
I just began this same process myself–buying the same hardware and then came across your webpage, which has been a great source of information!
[...] 2. http://davidhunt.ie/wp/?p=2410 [...]
Hi thanks for documenting your experience with this, it has inspired me to give it a go. Ebay is not being kind on prices for the Mellanox MHEA28 cards, however there are some cheap Qlogic QLE6140 aka QLE7140 or PE-880 / HT-465 cards around. Do you have any knowledge of this and direct cable connection between two of the cards?
A google search show one ancient post that says they cannot be used with a direct cable connection & they require a switch, does not offer any details on why they won’t work with direct connections though..
John,
Holy Cow, those Ebay prices aren’t great now, are they? I guess all the cheap ones got snapped up!
On the QLogic models, I’m afraid I haven’t tried them. At the time I built my fabric, there were mixed reports on whether the Mellanox adapters would work or not (direct), so I but the bullet and bought them. There was a lot of work in getting it to work at the time, but it was worth it. Maybe it’ll be the same for the QLogic’s?
Dave.