So, this will be a multi-part series documenting my adventures in deploying OpenStack Mitaka on Ubuntu 14.04 leveraging EMC’s ScaleIO storage (along with some iSCSI sitting on ZFS).  This post is meant to document how I’ve set up my environment, and any issues i’ve encountered.

The environment:

I have 6 total servers in the environment all connected via Mellanox 20Gb/s Infiniband and 1Gb/s IP.

• Server #1 – this is an older Dell Optiplex with 4 CPU cores and 16GB of RAM.  I currently have Docker installed running a handful of containers supporting this site, as well as some internal apps.  Additionally, this server is set up as the ScaleIO gateway.
• Server #2 – this is a 2U Supermicro box with 12 CPU cores and 64GB of RAM.  Currently this server is tasked to run the bulk of the supporting OpenStack services and databases.  It also contains ~10TB of useable capacity on ZFS on Linux that is exported via NFS, and eventually via iSCSI.
• Servers #3 through 6 – these are my Nova compute nodes, as well as my ScaleIO SDS nodes.  Each contains 8 CPU cores and 32GB of RAM.

Each of these servers contains an identical base build at this point.

The Setup:

When installing Ubuntu, the only server role i’ve given them is OpenSSH server.  Once the base install is complete, I’ve installed the opensm package in order to bring up the Subnet Manager in the IB network, and I’ve loaded the appropriate modules for my environment:

$ sudo apt-get install opensm

Once the OpenSM package is installed, I load the appropriate modules:

$ sudo modprobe mlx4_ib
$ sudo modprobe ib_umad
$ sudo modprobe ib_ipoib

I also add these modules to the existing /etc/modules file so it looks like this:

$ cat /etc/modules
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
# Parameters can be specified after the module name.

lp
rtc
mlx4_ib
ib_umad
ib_ipoib

Now that I have my modules loaded, I can set up my network interfaces for both my OpenStack provider network, and my IB storage network.  I edit the /etc/network/interfaces file to this:

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
# The loopback network interface
auto lo
iface lo inet loopback

# The Management network interface
auto eth0
iface eth0 inet static
     address 10.0.0.10/24
     gateway 10.0.0.1
     dns-nameservers 10.0.0.2
     dns-search battledome.lcl

# The Provider network Interface
auto eth1
iface eth1 inet manual
     up ip link set dev $IFACE up
     down ip link set dev $IFACE down

# The Infiniband Network
     auto ib0
     iface ib0 inet static
     address 10.1.1.10/24
     post-up echo connected > /sys/class/net/ib0/mode
     post-up ifconfig ib0 mtu 65520

Once the /etc/network/interfaces file is ready, simply bring up the eth1 and ib0 interfaces to ensure they’re operational.

$ sudo ifup eth1 && sudo ifup ib0

Next we want to update our packages using apt-get.  DO NOT upgrade the Kernel just yet.

$ sudo apt-get update
$sudo apt-get upgrade

Now, we want to update the Kernel to the latest SUPPORTED Kernel for ScaleIO.  as of this post, the latest SUPPORTED for Ubuntu 14.04 is 4.2.0-30.  If you blindly do a apt-get dist-upgrade, you will get a newer kernel than what is supported.  This is BAD, the ScaleIO Kernel module for the SDC’s will not load.  To get around this, we will install the 4.2.0-30 kernel, then ensure that apt-get HOLDS at that level.

nbsp;sudo apt-get install linux-headers-4.2.0-30-generic linux-headers-4.2.0-30 linux-image-4.2.0-30-generic linux-image-extra-4.2.0-30-generic 
Once the kernel install completes, we will set apt-get to HOLD at this level
$ sudo apt-mark hold install linux-headers-4.2.0-30-generic linux-headers-4.2.0-30 linux-image-4.2.0-30-generic linux-image-extra-4.2.0-30-generic 
REBOOT
 **NOTE:  There doesn’t appear to be a similar way to hold packages with the apt command.  I know there’s a way to hold packages with dpkg, but I havent tested this with apt.
Installing components necessary for ScaleIO
There’s not a lot in terms of prerequisites fro ScaleIO on Ubuntu.  we need to install Java 8, numactl, libaio, and bash-completion.
$ sudo add-apt-repository ppa:webupd8team/java
$ sudo apt-get update
$ sudo apt-get install oracle-java8-installer
$ sudo apt-get install numactl libaio1 bash-comletion 
Do this on each MDM, SDS, and SDC node you plan to use in your environment.
Additionally, it appears that you need to enable the root account and enable it to login via SSH for the Gateway installer to work.  I don’t believe this is a requirement for a manual install.
$ sudo passwd root
Allow root to login via ssh by modifying the /etc/ssh/sshd_config file with the following parameter:
PermitRootLogin yes
Restart the ssh service.
$ sudo service ssh restart
Repeat this fro each nod you will be using as an MDM, SDS, or SDC in your environment.
You are now ready to begin deploying ScaleIO.  I will cover that in Part 2.

 

Share this:

				
				X
			

				
				Facebook
			

				
				LinkedIn
			

				
				Email