In a previous blog article, I covered how to get the infiniband fabric up on Ubuntu 10.10 (Maverick Meerkat). This time I’ll cover IPoIB on Ubuntu 11.10 (Oneiric Ocelot), and also add in the information on how to also get the SRP target configured and running. This guide does not require you to rebuild the linux kernel, and works from the stock install of Ubuntu 11.10. So you should be up-and running with an SRP target in about 15 minutes, all going well.
As before, I’m using an Ubuntu box for the disk subsystem, and a Windows 7 as a client (initiator) machine.
Here’s a few acronyms:
SCSI – we should know what this one is (Small Computer Systems Interface)
iSCSI – SCSI protocol over a network. (Internet SCSI).
SCST – SCSI Target subsystem for linux with target drivers for iSCSI, Fibre Channel, SRP, SAS, FCoE, etc. We’re most interested in the SRP target.
RDMA – Remote Direct Memory Access – fast way of copying chunks of memory from one machine to another.
SRP – SCSI RDMA Protocol – wraps it all up using RDMA over SCSI protocol.
Stage 1 – Setting up IPoIB
This section shows how to get the basic fabric up and running with TCP/IP over Infiniband (IPoIB). Once you’ve done this, you’ll be able to send pings between the machines, ssh from one to the other across the fabric, etc.
For the windows 7 machine, it’s a simple case of installing the OFED drivers from openfbrics.org. While installing the windows drivers, make sure you select the SRP options, as later in this guide we’ll be setting up the RAID box as a drive on the Windows box using SRP (iSCSI RDMA Protocol). So, install the OFED driver package.
To enable improved throughput, enable connected mode on the link by updating the “Connected Mode” and “Connected Mode Payload Mtu size” in the properties of the interface. Open up “Device Manager” and find your Infiniband adapter:
Right clock on the adapter and bring up the properties dialog:
So, on to the linux box. I started with a fresh install of Ubuntu 11.10.
Firstly Install Ubuntu. Update all the packages to the latest using
sudo apt-get update sudo apt-get upgrade
Everything below is done as root, this avoids having to type ‘sudo’ before every command, so I just call “sudo bash”.
Note: No need to edit the udev rules any more as per the Ubuntu 10.10 HOWTO, as they are correctly set by the kernel 3.0.0 and newer.
Edit /etc/modules and add the following modules:
ib_sa ib_cm ib_umad ib_addr ib_uverbs ib_ipoib ib_ipath ib_qib
Next,
apt-get install opensm
This will install the subnet manager and all the relevant dependencies, libibverbs, etc.
Then add the relevant entries for the interface into /etc/network/interfaces file:
auto ib0 iface ib0 inet static address 192.168.1.1 netmask 255.255.255.0
Then reboot. This will create the relevant infiniband entries in /sys, load the ipoib modules, and bring up the infiniband port with an ip address. You should now have a functioning infiniband port on your Ubuntu machine, and you should be able to ping the remote machine.
The next thing is to enable connected mode for the infiniband connection. I found that this increased the tcp/ip netperf benchmarks from 3gbps tp 7gbps.
root@raid:~# echo connected >`find /sys -name mode | grep ib0` root@raid:~# echo 65520 >`find /sys -name mtu | grep ib0`
To make this happen when the ib0 interface is brought up, modify the /etc/network/interfaces file as follows:
auto ib0 iface ib0 inet static address 10.4.12.1 netmask 255.255.255.0 up echo connected >`find /sys -name mode | grep ib0` up echo 65520 >`find /sys -name mtu | grep ib0`
Stage 2 – Setting up SRP Target
The next step, should you wish to take it, is to set up the SRP targets. This is much better option than using samba shares, as it gives you a massive boost in throughput, and uses much less CPU. For example, on my setup using IPoIB and samba shares, I was able to achieve no more than 125MB/sec. Using a ramdisk set up as an SRP target, I was able to achieve 900MB/sec between my two machines.
So, first get a few packages:
apt-get install libmthca1 apt-get install iscsitarget apt-get install open-iscsi apt-get install lsscsi apt-get install scsitools
change /etc/default/iscsitarget to:
ISCSITARGET_ENABLE=true
We now need to get scstadmin. I typically pull the latest version from subversion, so we first need to get svn. apt-get install subversion
There’s a choice to me made now. According to the guide at http://iscsi-scst.sourceforge.net/iscsi-scst-howto.txt , you get a slight performance increase by patching and re-compiling the kernel. That’s a lot of work, so I skipped that step for this guide. I went for the easier option, which is just to run the few commands below. I’m not sure what the difference in performance is, but I can still pull 900MB/sec across the fabric from a ramdisk set up as a an SRP target. And not having to rebuild the kernel makes this a 15 minute procedure rather than a 3 hour one. 
cd ~ svn co https://scst.svn.sourceforge.net/svnroot/scst/trunk scst cd scst make scst scst_install make iscsi iscsi_install make scstadm scstadm_install make srpt srpt_install
N.B.: You’ll need to rebuild these each time Ubuntu upgrades the kernel, as the modules get out of sync. Just re-run the make’s again.
Lets now use scstadmin to create an SRP target:
First we modprobe a couple of kernel modules. That will make the relevant drivers and targets available to scstadmin. We should then be able to see thse drivers/targets with a few scstadmin ‘list’ commands:
modprobe scst_vdisk modprobe ib_srptroot@raid:~# scstadmin -list_handler Collecting current configuration: done. Handler ------------- vdisk_fileio vdisk_blockio vdisk_nullio vcdrom All done. root@raid:~# scstadmin -list_driver Collecting current configuration: done. Driver ------- ib_srpt All done. root@raid:~# scstadmin -list_device Collecting current configuration: done. Handler Device ------------------------ vdisk_nullio - vdisk_fileio - vdisk_blockio DISK01 vcdrom - All done. root@raid:~#So, if you can see those handlers, targets, and drivers, your good to go with the next step.
Summary:
The following command list is the quick list to use (and modify) if you’ve done this kind of thing before. The detail section below gives a bit of explanation on each of the lines.
scstadmin -disable_target ib_srpt_target_0 -driver ib_srpt scstadmin -clear_config -force scstadmin -open_dev DISK01 -handler vdisk_blockio -attributes filename=/dev/sda scstadmin -set_dev_attr DISK01 -attributes t10_dev_id=0x2346,threads_num=4 scstadmin -add_group HOST01 -driver ib_srpt -target ib_srpt_target_0 scstadmin -add_lun 0 -driver ib_srpt -target ib_srpt_target_0 -group HOST01 -device DISK01 -attributes read_only=0 scstadmin -add_init 0x0002c9020021f9fc0002c902002200bc -driver ib_srpt -target ib_srpt_target_0 -group HOST01 scstadmin -enable_target ib_srpt_target_0 -driver ib_srpt scstadmin -write_config /etc/scst.conf
Detail:
Clear the current config (only if that’s ok, and you don’t have any other config that you want to keep. I do this because I’m starting with a clean slate).
scstadmin -clear_config -force
Create DISK01, assigning it to a partition (/dev/sdg1, /dev/md0p1, etc., etc.). I’m using a disk partition in this example.
scstadmin -open_dev DISK01 -handler vdisk_blockio -attributes filename=/dev/sdg1
Now set the drive attributes.
scstadmin -set_dev_attr DISK01 -attributes t10_dev_id=0x2345
Now add a group
scstadmin -add_group HOST01 -driver ib_srpt -target ib_srpt_target_0
Add a LUN to the group, assigning it to DISK01
scstadmin -add_lun 0 -driver ib_srpt -target ib_srpt_target_0 -group HOST01 -device DISK01 -attributes read_only=0
Add an initiator to the group, allowing the initiator to connect to our new target. I got this from watching /var/log/messages while I was disabling and enabling the Infiniband SRP miniport in device manager on the Win7 box. This caused the SRP miniport to attempt to connect to the Ubuntu target, and that attempt is shown in the /var/log/messages file along with the initiator ID.
scstadmin -add_init 0x0002c9020021f9fc0002c902002200bc -driver ib_srpt -target ib_srpt_target_0 -group HOST01
Finally enable the target
scstadmin -enable_target ib_srpt_target_0 -driver ib_srpt
And write the config.
scstadmin -write_config /etc/scst.conf
At this point you should see a new drive appear on the Win7 box, and asking you to format it. If not, you could try disabling and enabling the SRP miniport driver again, and take a look at /var/log/messages to see what’s happening.
Once the new drive appears on the Windows host, you should be able to format it and start using it. If you don’t see it immediately, have a look in “Disk Management”.
I found that the //etc/init.d/scst script was not getting called at boot, so I added a softlink in /etc/rcS.d
cd /etc/rcS.d ln -s ../init.d/scst S26scst
This is so it will start after scsitools, opensm and open-iscsi. Using the /etc/init.d/scst script also resolved a problem I’ve had for a long time, in that I used to have some modules in /etc/modules, so the SRP miniport would attempt to connect too early in the linux boot sequence, and fail, so that I had to go into the device manager and disable/enable to get it working. Now it pops up perfectly every boot of the linux box, and I don’t have to go into device manager on the Win7 box any more.
I’d suggest you run a benchmark on it also just to see what kind of speed you’re getting out if it in real-world usage. There’s a very handy (free) benchmarking took available from attotech.com. Oh, and please do drop a comment below if you do get decent speeds. I’m always interested to hear what people are getting. Oh, and also, comment if you find this guide useful!
References:
SCST – http://scst.sourceforge.net/
Installing iSCSI-SCST – http://iscsi-scst.sourceforge.net/iscsi-scst-howto.txt
Performance Testing.
If you have plenty of memory in your linux machine, you might like to do a test of a ramdisk SRP target over the fabric.
There’s a ramdisk set up by default in Ubuntu at /dev/ram0, but it’s quite small, at 64K. I like to bump it up to 1Gig or 2Gig, as I have 4Gig in my linux box and it does need all that for normal operation. Adding an extra parameter to the kernel line in /boot/grub/grub.cfg does the trick nicely.
ramdisk_size=1024000 for a 1Gig ramdisk at /dev/ram0
ramdisk_size=2048000 for a 2Gig ramdisk at /dev/ram0
so the full line would look like:
linux /boot/vmlinuz-3.0.0-12-generic root=UUID=39331d95-e9ba-48a5-9dd9-09978e0503c4 ro quiet splash vt.handoff=7 ramdisk_size=1024000
Then using scstadmin, we add another disk to the script above, but we use vdisk_fileio rather than vdisk_blockio.
scstadmin -open_dev DISK02 -handler vdisk_fileio -attributes filename=/dev/ram0
Then when you open disk manager in windows, you should see a disk requesting a new MBR, which you go ahead and add, then format and run a few benchmarks on it.
