Continuous testing environment

The below screencast is a quick run through of my continuous testing environment.

Creating a cloudstack plugin

Hi Guys,

Its been awhile since my last post. I've been playing about with Jenkins a lot in order to setup a continuous testing environment, but that's going to be saved for another post.

I began making progress on the API commands I need to make. I created an external class for connecting to LDAP just to get some JDNI experience before diving straight into the Cloudstack codebase of over 4000 lines!

After making this external class I started wondering how to integrate it in with Cloudstack. Turns out you can easily do this as a plugin.

I have successfully made a sample plugin and its now just a matter of changing it about a bit to suit my needs. If you wish to view my notes on this I have uploaded them to http://ianduffy.ie/cloudstack/CreatingAPlugin.pdf

Configuring cloudstack to authenticate against LDAP

Cloudstack currently has some basic implementation for authenticating against LDAP. In this post I will detail how I configured it to authenticate against the OpenLDAP server I setup in a past post.

Login as an admin, navigate to the accounts page and create a new user. The domain must match your LDAP domain and the username must match the username on LDAP. The password can be anything as it is ignored.

Go into global settings and enable API access by setting integration.api.port. Finally navigate to http://ip-of-your-manager:8096/client/api?command=ldapConfig and give it the following parameters like the following: host: ldap.clouddev.lan
searchBase: ou=users,dc=clouddev,dc=lan
queryfilter: (&(uid=%u))
binddn: CN=Manager,DC=clouddev,DC=lan
bindpass: PASSWORD
port: 389

 an example url for this would be as follows:

http://ip-of-your-manager:8096/client/api?command=ldapConfig&hostname=ldap.clouddev.lan&searchbase=OU%3DUsers,DC%3Dclouddev,DC%3Dlan&queryfilter=%28%26%28uid%3D%25u%29%29&binddn=CN%3DManager,DC%3Dclouddev,DC%3Dlan&bindpass=PASSWORD&port=389&response=json


Note: In versions 4.2.0> there is a UI feature under global settings for configuring authentication against an LDAP server. However when I used it I found it ran HTML encoding on my query filter. Opened a bug report for this: https://issues.apache.org/jira/browse/CLOUDSTACK-3044

Git command line enhancements

Enhancing your bash prompt for git:

The standard bash prompt normally shows your username, the hostname and the folder you are in. This is handy, but wouldn't it be great if it told you the branch you were on and the state it was in?

Lets go from this:

to this:

Here you can see the "master" denotes the branch I'm in and the asterix lets me know that I've made some modifications.  I also have the full path of where I am displayed. This removes the need for me to use commands like pwd, git branch, and git status a lot.

This is done by modifying your .bashrc file (located within your home folder).

To produce the above I added the following to ~/.bashrc:

 if tput setaf 1 &> /dev/null; then
 tput sgr0
 if [[ $(tput colors) -ge 256 ]] 2>/dev/null; then
  MAGENTA=$(tput setaf 9)
  ORANGE=$(tput setaf 172)
  GREEN=$(tput setaf 190)
  PURPLE=$(tput setaf 141)
  WHITE=$(tput setaf 7)
 else
  MAGENTA=$(tput setaf 5)
  ORANGE=$(tput setaf 4)
  GREEN=$(tput setaf 2)
  PURPLE=$(tput setaf 1)
  WHITE=$(tput setaf 7)
 fi
 BOLD=$(tput bold)
 RESET=$(tput sgr0)
else
 MAGENTA="\033[1;31m"
 ORANGE="\033[1;33m"
 GREEN="\033[1;32m"
 PURPLE="\033[1;35m"
 WHITE="\033[1;37m"
 BOLD=""
 RESET="\033[m"
fi

export MAGENTA
export ORANGE
export GREEN
export PURPLE
export WHITE
export BOLD
export RESET

function parse_git_dirty() {
 [[ $(git status 2> /dev/null | tail -n1) != *"working directory clean"* ]] && echo "*"
}

function parse_git_branch() {
 git branch --no-color 2> /dev/null | sed -e '/^[^*]/d' -e "s/* \(.*\)/\1$(parse_git_dirty)/"
}

export PS1="\[${BOLD}${MAGENTA}\]\u \[$WHITE\]at \[$ORANGE\]\h \[$WHITE\]in \[$GREEN\]\w\[$WHITE\]\$([[ -n \$(git branch 2> /dev/null) ]] && echo \" on \")\[$PURPLE\]\$(parse_git_branch)\[$WHITE\]\n\$ \[$RESET\]"
export PS2="\[$ORANGE\]→ \[$RESET\]"

Prettifying git and creating some aliases:

Using our git configuration file it is possible to add aliases for certain commands. Along with this it is possible to set colors to denote different outputs. Below is an extract of some of the things within my ~/.gitconfig

[alias]
 # View the SHA, description, and history graph of the latest 20 commits
 l = log --pretty=oneline -n 20 --graph
 # View the current working tree status using the short format
 s = status -s
 # Show the diff between the latest commit and the current state
 d = !"git diff-index --quiet HEAD -- || clear; git diff --patch-with-stat"
 # `git di $number` shows the diff between the state `$number` revisions ago and the current state
 di = !"d() { git diff --patch-with-stat HEAD~$1; }; git diff-index --quiet HEAD -- || clear; d"
 # Pull in remote changes for the current repository and all its submodules
 p = !"git pull; git submodule foreach git pull origin master"
 # Clone a repository including all submodules
 c = clone --recursive
 # Commit all changes
 ca = !git add -A && git commit -av
 # Switch to a branch, creating it if necessary
 go = checkout -B
 # Show verbose output about tags, branches or remotes
 tags = tag -l
 branches = branch -a
 remotes = remote -v
 # Credit an author on the latest commit
 credit = "!f() { git commit --amend --author \"$1 <$2>\" -C HEAD; }; f"
 # Interactive rebase with the given number of latest commits
 reb = "!r() { git rebase -i HEAD~$1; }; r"

[color]
 # Use colors in Git commands that are capable of colored output when outputting to the terminal
 ui = auto
[color "branch"]
 current = yellow reverse
 local = yellow
 remote = green
[color "diff"]
 meta = yellow bold
 frag = magenta bold
 old = red bold
 new = green bold
[color "status"]
 added = yellow
 changed = green
 untracked = cyan

Credits to Mathias Bynens dotfiles project for the above code extracts.

Setting up an OpenLDAP server

As mentioned in an earlier post I will require an LDAP server for testing my project. I will be using OpenLDAP on Centos 6.4.

Lets kick things off by installing some packages:

 $ sudo yum install openldap openldap-clients openldap-servers

Generate a password for your Manager/Administrator user:

 $ slappasswd

This will generate a SHA hash something like this:

{SSHA}q6sOQ5FGWkU6YE5H+awaGZj8UKpLVkBH

This needs to be inserted into the servers configuration file so note it down.

Open up /etc/openldap/slapd.d/cn\=config/olcDatabase\={2}bdb.ldif and modify it so it looks like the following:

 ...................................
olcReadOnly: FALSE
olcRootDN: cn=Manager,dc=my-domain,dc=com
olcRootPW: {SSHA}q6sOQ5FGWkU6YE5H+awaGZj8UKpLVkBH
olcSyncUseSubentry: FALSE
olcMonitoring: TRUE
...................................

Next we need to configure our domain, you can do this by using the replace(replace dc=my-domain,dc=com) feature in your editor or you can use sed like I have done below:

 $ sed -i -e 's/dc=my-domain,dc=com/dc=clouddev,dc=lan/g' /etc/openldap/slapd.d/cn\=config/olcDatabase\={2}bdb.ldif
 $ sed -i -e 's/dc=my-domain,dc=com/dc=clouddev,dc=lan/g' /etc/openldap/slapd.d/cn\=config/olcDatabase\={1}monitor.ldif

Auto start OpenLDAP:

 $ chkconfig slapd start
 $ service slapd start

Populate it:

For this example I will only populate the root directory and will configure the rest later using phpldapadmin. If you wish you can create users/groups using ldapadd and ldif files.

 $ echo -e "dn: dc=clouddev,dc=lan\nobjectClass: dcObject\nobjectClass: organization\ndc: clouddev\no : clouddev" > /tmp/base.ldif
 $ ldapadd -f /tmp/base.ldif -D cn=Manager,dc=clouddev,dc=lan -w password

Configure iptables:

Since I'm just doing this for a development environment I just turned of iptables completely:

 $ iptables --flush
 $ service iptables stop
 $ chkconfig iptables off

If you wish to configure them simply insert:

-A INPUT -p tcp --dport 389 -j ACCEPT

into /etc/sysconfig/iptables

Finally test that your server is up and working by querying it:

 $ ldapsearch -h localhost -b dc=clouddev,dc=lan -xxx

I didn't wish to spend time doing ldif dumps and applying them with ldapadd/ldapdelete/ldapmodify so I opted to install phpldapadmin which supplies a web based interface for managing OpenLDAP.

Start by setting up the EPEL repos on CentOS:

 $ wget http://ftp.riken.jp/Linux/fedora/epel/RPM-GPG-KEY-EPEL-6
 $ rpm --import RPM-GPG-KEY-EPEL-6 
 $ wget http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
 $ rpm -ivh epel-release-6-8.noarch.rpm 

Continue on to install phpldapadmin:

 $ yum --enablerepo=epel install phpldapadmin

Configure the access you require to it by modifying the allow access from line in /etc/httpd/config.d/phpldapadmin.conf

Next we need to change an option in phpldapadmin's configuration file to use a dn for login instead of a uid.

 $ sed -i -e "s/$servers->setValue('login','attr','uid');/\/\/$servers->setValue('login','attr','uid');/g" -e "s/\/\/$servers->setValue('login','attr','dn');/$servers->setValue('login','attr','dn');/g" /etc/phpldapadmin/config.php

Finally restart the httpd:

 $ service httpd restart

browse to http://server-address/ldapadmin and login with
username: cn=Manager,dc=clouddev,dc=lan
password: password

and create your wanted OUs, users, groups, etc.

Setting up a Cloudstack Development environment

This process is detailed for Windows, OSX, Ubuntu and CentOS on the Cloudstack website. However I hit some issues when following them. Below I have documented my setup for future reference purposes.

My development environment is running on CentOS 6.4 which is installed within virtualbox. It has a host-only interface for communication with other VMs and the host, it has a NAT interface for getting internet access.

So lets start of by installing some required tools and dependencies:

 $ sudo yum install git java-1.6.0-openjdk java-1.6.0-openjdk-devel mysql mysql-server tomcat6 mkisofs gcc python python-setuptools openssh-clients wget

Add mysql server to auto start:

 $ sudo chkconfig mysqld start
 $ sudo service mysqld start

Install Maven:

 $ cd /usr/local/
 $ sudo wget http://www.us.apache.org/dist/maven/maven-3/3.0.4/binaries/apache-maven-3.0.4-bin.tar.gz
 $ sudo tar -zxvf apache-maven-3.0.4-bin.tar.gz

Set environmental variables:

 $ echo export CATALINA_BASE=/usr/share/tomcat6 >> ~/.bashrc
 $ echo export CATALINA_HOME=/usr/share/tomcat6 >> ~/.bashrc
 $ echo export M2_HOME=/usr/local/apache-maven-3.0.4
 $ echo export PATH=${M2_HOME}/bin:${PATH} >> ~/.bashrc

Set maven options:

 $ echo export MAVEN_OPTS="-XX:MaxPermSize=256m -Xmx1g" >> ~/.bashrc

Fix permissions:

 $ chown -R <user>:<group> $CATALINA_HOME

Generate ssh keys:

 $ ssh-keygen

Install python modules:

 $ sudo easy_install pip
 $ sudo pip install requests
 $ sudo pip install mysql-connector-python

Grab the code:

 $ mkdir ~/projects
 $ cd ~/projects
 $ git clone https://git-wip-us.apache.org/repos/asf/cloudstack.git

Build it:

 $ cd ~/projects/cloudstack
 $ mvn -P developer,systemvm clean install

Deploy the devcloud database:

 $ mvn -P developer -pl developer,tools/devcloud -Ddeploydb

If you have a root mysql password set just copy utils/conf/db.properties to utils/config/db.properties.override and insert it.

Run the management server:

 $ mvn -pl :cloud-client-ui jetty:run

Browse to http://localhost:8080/client and you should be able to login with the default credentials of admin and password.

Bring up devcloud:

If you haven't already done so create a host-only adapter on virtualbox and  disable its DHCP server.

Download the devcloud virtual appliance, import it and check that PAE/NX mode is enabled. Launch it and check that you can ping it on 192.168.56.10 on from your development box.

Deploy the devcloud configuration:

Make sure your management server and devcloud is running and execute the following from inside your Cloudstack source folder:

 $ mvn -P developer -pl tools/devcloud -Ddeploysvr

This will deploy the settings defined in tools/devcloud/devcloud.cfg. If you open the management ui and navigate over to the System VMs you should see them starting to come up.

I found having the http_proxy environmental variable set broke this from executing successfully. As a work around I just unset it before executing, export http_proxy="".

For more information see: https://cwiki.apache.org/CLOUDSTACK/devcloud.html

Community Bonding Period

Within the Google Summer of Code program we are currently within the "Community Bonding" period. Within this period it is expected that we research documentation surrounding our project, get involved with the opensource community that has been working on the project and work with our mentors. 

So far everything has been amazing. As mentioned in the introduction post I am working with the Apache Software Foundation with their Cloudstack Project. Sebastien, Coordinator for the GSoC program within Cloudstack has been of great help and has given guidelines and tasks towards getting started.

Our first task was pretty simple, It was just a case of introducing ourself, registering on JIRA, review board, and their confluence based wiki. Along with this we had to convert our proposals to docbook format and submit them to the git repository by uploading patch to their review board. The support supplied by the community for this was fantastic, they had written documentation for creating a patch on their site http://cloudstack.apache.org/develop/non-committer.html. Along with this Sebastien create an intro to git screencast for us which can be viewed on youtube http://www.youtube.com/watch?v=3c5JIW4onGk.

For me my experience with the GSoC program will probably differ from most others. Paddy Power's platform team are facilitating me within their offices. Working with them so far has been an great experience. It is supplying me with exposure to how things are done within industry and it is filling in a lot of the lose ends college theory fails to cover. I will be following their producers for project deliverance, this should help me to develop a good understanding of agile software development. 

My personal research so far has been going OK. I setup a development environment and used their devcloud appliance to supply a host following community written instructions: http://cloudstack.apache.org/develop/environment.html. There were few things that didn't execute exactly like the instructions stated but I eventually got around it and got Cloudstack build and running successfully. I plan to do post about setting up the developer environment and issues I hit at a later date. As my project require an LDAP server I also did an installation of OpenLDAP and linked it up with Cloudstack. Again I will do a post at a later date detailing the setup of this. 

I am currently looking at creating unit and acceptance tests. I have been playing about a good bit with Selenium in an attempt to create automated acceptance tests for my additions to the project. I will follow up on my progress with this over the next week hopefully.

About this blog

Hi welcome to my blog,

My name is Ian Duffy and I'm a computing student attending Dublin City University in Ireland. I have interests in virtualization, automation, information systems, networking and web development.

I am currently undertaking a project set out by the Apache Software Foundation within Google's Summer of code program. I am working on a project called Cloudstack, this is cloud computing software for creating, managing and deploying infrastructure cloud services. Specifically, my project aims to improve LDAP user provisioning. LDAP is simply a directory service that holds user profile information, it will be used to provide single-sign-on for provisioned users. That is, if a user exists within the LDAP database and they have been provisioned on Cloudstack then they will be able to login to Cloudstack with their LDAP username/password. A full project proposal can be seen at http://ianduffy.ie/cloudstack-ldap.pdf

I hope to use this blog for reflections and progress reports. If you have any questions, advice, feedback or criticism please leave a comment or email me on ian@ianduffy.ie