Continuing on my research into a high-bandwidth infiniband fabric at home, mainly with a view to fast backups of the 2000Gigs of photos I now have, I was now on the lookout on eBay for a good-value hardware RAID controller card. After a couple of weeks of looking at cards and prices, I came across an auction for a Dell PERC 5/i 8-port card. Bidding was pretty low, probably due to the comment on the auction that the seller could not guarantee that the card was in fully working condition. Turned out later to be in great working order.
I ended up getting it for €31 including shipping to Ireland from Finland. Most of these cards go for three times that price. I was just lucky that the auction ended at 3am, and I dont think that many people bothered to stay up to keep an eye on it.
About a week later, the card arrived, along with the seperate order of a couple of SFF8484 to 4-way SATA cables, I was ready to go.
I’d already freed up one of the PCI-e 16x slots in by Ubuntu box, so the Perc 5/i slotted right in. I hooked up the 5 x 1TB drives that I was configuring in a RAID5 volume, and booted up.
Upon boot, the raid card was detected, and I hit CTRL-R to enter the setup. It was fairly straightforward, I selected the 5 drives that I’d connected up to it, and initialised a new RAID5 volume.
Anyway, the drive then appeared to Linux as a 4TB volume, which I used parted to partition into one big volume.
There were a couple of problems once I’d installed the drive, read speed and write speed. I was getting 200MB/sec reads and 50MB/sec writes. I did a bit of googling and found an excellent thread on the Dell PERC 5/i controller (ref1), so I flashed the firmware up to the very latest from LSI. I initially did not have a battery, but the latest firmware allowed me to force Write-Back and Adaptive Read-Ahead (not safe settings without battery, but it’d be ok till the battery arrived, another eBay purchase), which should improve the throughput. Also, I found that when you do a quick init of the aray, it says that it completes very quickly, but it’ll go ahead and do a full background init afterwards thereby confusing my throughput results. I found that the best thing to do was to do a full init in the RAID configuration utility and let it sit there for 2.5 hours to complete the full init of 4TB (5x1TB). That way I’d know that the initialisation process would not be skewing my measurements later. Before I did the full init, though, I was getting 170MB read/write (across the infiniband fabric) even when the background init was in progress, so the firmware upgrade was definitely a step in the right direction. I was looking forward to full speed measurements in a couple of hours.
Once the init finished, I was disappointed to find that I was only getting just under 200MB/sec read/write.
So, I tore down the array again and went back to a 5 disk RAID0 array, for max speed. This gave me the following:
This was only marginally better than the RAID5 readings, and worse read speed than using software RAID and motherboard SATA connectors. So to eliminate my Linux box motherboard and CPU, I took the RAID controller out of the Linux box and interted it into my i7 desktop (firstname.lastname@example.orgGHz 12G RAM), leaving the drives in the linux box, and running the SATA cables between the two machines. The throughput from the raid card was quite different.
So, now I was getting 300-350MB/sec writes, and up to 600MB/sec reads, with 500MB/sec reads quite common above block size of 64K. That proved that my linux box motherboard is now the bottleneck. Maybe the fact that I’ve two PCIe-8x cards plugged into the two PCIe-16x slots intended for graphics cards is a problem?
At least now I knew what the Perc 5i RAiD controller was capable of, I just had to get the machine that it was hosted in up to spec to be able to drive it at full speed. Maybe then we’ll get more out of the infiniband link between the two machines. I’ll look at re-organising the cards in the machine, maybe look at getting a new motherboard with faster PCI-express lanes.
I’m rapidly coming to the conclusion that trying to re-cycle 5-year-old components to get uber-speed throughput is a lot more difficult than it seems at first. Might be something to do with the hardware not being up to the task when pushed to the limit. The motherboard I’m using in the Linux box with the RAID controller is an Asus A8N-LSI-Premium, which was a top-end consumer-grade motherboard in 2005. With two PCI-e 16x slots, each theoretically capable of 2GB/sec (PCI-e 1.0 8 lanes) in SLI mode, yet I’m only able to get 200-300MB/sec through them? What’s with that? Surely there’s gotta be a helluva lot more data going through those slots when there’s two big fat graphics cards in them.
Stay posted for more updates soon….
Update: Just to help narrow down the bottleneck, I managed to get a 1Gigabyte ramdisk set up as an SRP target. Here’s the throughput across the fabic:
So, The infiniband card in the PCI-express socket with 8 lanes assigned is only getting 200MB/sec through it from RAM. CPU was about 25% busy through the whole test.