Infiniband cluster Part 1

Overview and installing the operating system


To facilitate testing of our software and so that support can model possible customer environments I was tasked with setting up a cluster with infiniband. I was excited as the system was a Dell PowerEdge blade cluster (our 1st blade system) and we have never used or deployed infiniband (this would be our 1st infiniband system), in fact other than a basic “we have heard of this and know (very) roughly what it is” no-one in the group had any real knowledge of it. After a few hours of searching the deepest recesses of the Internet I had a limited understanding of how to setup the system. I know how irritating it is to have to read the crud in order to find that wee gem of info so from here on I’ll mark the paragraphs of the article so those of you wanting to “just get it done” can skip them.

System Info:

  • 8 Nodes
  • 3 Vlans – 1 admin, 2 LAN, 3 Cluster Private
  • RHEL 5.3 (CentOS 5.3 should work fine as well)
  • Class 2 cluster

System Specs:

  • PE M1000e Blade Enclosure
  • InfiniBand Mellanox  DDR M2401G Switch
  • M6220 Switch 20 ports
  • 8 PowerEdge M610 Blades
    • 2x Intel Xeon E5530 (2.4Ghz, 8Mb, 5.86GT/s QPI, Turbo, HT, 1066Mhz)
    • 12GB Ram
    • 73GB SAS 15K 2.5″
    • Infiniband Mellanox QDR I/O Card



CPU/RAM: As you can see from the spec this is no mean system, each node is dual CPU quad core nehalem. 12GB (6x2GB) of RAM per node is due to the nehalem architecture, we use 3 channels in 2 banks as using all 3 banks would cause the speed to drop to 800Mhz ( good article here: Nehalem Memory Configuration ).

Cluster Type: I decided to setup a class 2 cluster as it removes a lot of the effort required for me to admin the system, it can authenticate users directly against the AD and the users can access their network shares from any node.

VLAN: As I was going with a class 2 cluster I figured that the best option was to setup the network with vlans. 1 for CMC traffic, 1 for Private cluster traffic (the cluster uses it’s internal connectivity to communicate) and 1 to allow the nodes to talk to the main LAN. This allowed me to setup each of the nodes from my primary RIS server instead of setting up the head node and installing tftp on it and then deploying the each of the nodes from there. This also allows me to change the cluster into a class 1 simply by disabling the 2nd VLAN on all but the head node.

Install the OS:

The following package selection has NO GUI as my blades are headless.

  • @base
  • @server-cfg
  • @development-libs
  • @development-tools
  • @admin-tools
  • @system-tools
  • @legacy-software-development
  • @legacy-software-support
  • @editors
  • @openfabrics-enterprise-distribution
  • opensm

The last 2 in the list are the most important, The rest is pretty much dependant on your needs. The last part in the series will contain a bash script for configiration & a kickstart file.

Update the system with yum

yum update

UPDATE: There is a GOTCHA here, I found updating to 5.4 from 5.3 the updater got stuck in a loop wanting to update “openib-perftest” to “perftest” and vice versa, in 5.3 it was “openib-perftest” and in 5.4 it was renamed to “perftest”. So once it has been upgraded once you may want to add it to the exclude list for yum.

Series Links

Leave a Reply

Your email address will not be published. Required fields are marked *


This blog is kept spam free by WP-SpamFree.