Here’s some more ramblings on 10G Networking on the Cheap.
Having gone to all the trouble of getting a 900MB/sec SRP target working, I was interested to go back and see how a samba mount would perform now that I’d sorted out all the cards into their optimum PCI-express slots.
So, I did the same setup as the 900MB/sec SRP test, Ramdisk to Ramdisk. This time, I set up a 2Gig ramdisk as a samba mount, and copied large files to and from that ramdisk from a 4G Ramdisk on the Windows side. ATTO disk benchmark doesn’t test network drives, so I had to make do with copying large files and keeping an eye on the throughput with dstat.
I initially measured the IPoIB throughput using netperf, than this gave me about 6Gbits.
daveh@raid:~$ netperf -H 10.4.12.2 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.4.12.2 (10.4.12.2) port 0 AF_INET : demo Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec
8192 16384 16384 10.00 6022.50
Based on that number, I was not expecting anywhere close to the 900MB/sec throughput I was getting with SRP. Anyway, just to compare samba throughput with SRP, I pressed on.
While copying a 1 GB video file to the ramdisk on the linux box (write)
usr sys idl wai hiq siq| read writ| recv send 3 57 31 0 0 9| 0 0 | 191M 565k 3 61 28 0 0 8| 0 0 | 188M 556k 4 56 31 0 0 9|4096B 40k| 192M 568k 4 59 32 0 0 6| 0 0 | 163M 485k 5 56 29 0 0 9| 0 0 | 186M 6419k 3 61 29 0 0 8| 0 0 | 187M 6519k 2 64 25 0 0 9| 0 0 | 183M 543k
While copying a 1 GB video file from the ramdisk on the linux box (read) usr sys idl wai hiq siq| read writ| recv send 3 40 54 0 0 4| 0 4096B| 863k 251M 6 42 47 1 0 5| 0 52k| 910k 263M 5 44 48 0 0 5| 0 0 | 816k 233M 5 42 46 0 0 6| 0 0 | 812k 231M 4 45 48 0 0 4| 0 0 | 826k 236M 3 44 49 0 0 5| 0 0 | 819k 234M 4 44 48 0 0 5| 0 0 | 832k 236M
I know it’s not very scientific, and I didn’t spend any time tuning samba. But the 900MB/sec throughput is practically out-of-the-box for SRP, so why not test samba at it’s out-of-the-box settings. 
So, even with netperf showing a 6Gbit throughput, I was only seeing around 190-235MB/sec throughput over samba (at 40-60% CPU usage). So I think I’ll stick with my SRP target setup rather than going back to samba. With SRP, I know that whatever disk subsystem I put into the linux box, it probably won’t be able to keep up with the 900MB/sec that the link is capable of. Unless, that is, I buy a very expensive RAID card and a bunch more disks. And with the way disk prices have gone over the last few months with the floods in Thailand, it’ll be a while until I even think about doing that.
