Jenkins Configuration

As I mentioned in an earlier post I am doing the work required for Google Summer of Code within the Paddy Power offices. Due to this I am following their standards for delivering a project.

A big part of this is following a behavior driven development approach. This puts a great emphases on continuous testing and for this reason a build pipeline as developed within Jenkins.

The layout of the pipeline is as follows:

• Pull in non-oss dependencies
• Pull in the cloudstack branch I've been working on, move in the non-oss dependencies and build it executing unit tests
• Run static analysis on the code using sonar source
• Create a new Cloudstack Database, Start the simulator and run the pre-integration configuration
• Launch the integration tests
• Stop the simulator
• Build a RPM binary of the source
• Push the binary out onto a local repository

This setup was showcased in a screencast I did awhile back, if you are interested: http://imduffy15.blogspot.ie/2013/06/continuous-testing-environment.html

This post is going to go through the process of creating such a setup. It is assumed you have a clean version of jenkins installed. The configuration of sonar will be detailed in a separate post.

Plugins required:

• Install the following plugins from Manage Jenkins -> Plugin Manager
• Build pipeline
• Jenkins Build timeout
• Jenkins clone workspace SCM
• Jenkins GIT
• Jenkins Parameterized Trigger
• Jenkins Sonar

After installing these plugins you will need to do a small amount of configuration. Navigate to Manage Jenkins -> Configure System

• Configure JDK - Under the JDK heading set your java_home path
• Configure git - Under the git heading fill out the options as necessary
• Configure maven - Under the maven heading set the path to your Maven 3.0.5 Home
• Configure sonar - Set its URL, account login, account password, database url(matches the same url in your sonar config), database login, database pasword
• Job Configuration

Job Configuration

To create a new job simply git "new job" on the sidebar. For most of these jobs I will be using a build a free-style project, unless stated otherwise.

Update non-OSS libs:

• Configure git to pull git@github.com:vogxn/cloudstack-nonoss.git this is a github repository that contains the non-oss dependencies required for the non-oss build of cloudstack
• Set a Poll SCM of @hourly
• Set the build to abort if it's stuck. For this I used Elastic with the default value of 150%
• Create a post build action of "Trigger parameterized build on other projects". Set projects to build to "build-master-nonoss", set trigger when build is stable, finally check trigger build without parameters.
• Save the job

Build non-OSS components:

• Description > Build non-OSS components
• Discard old builds > Log Rotation
• Days to keep builds > 5
• Max # of builds to keep > 5
• This build is paramaterized > true
• String Parameter
• NAME > MAJOR_VERSION_NUMBER
• DEFAULT VALUE > 4.2
• String Parameter
• NAME > PROJECT_REPOSITORY
• DEFAULT VALUE > https://git-wip-us.apache.org/repos/asf/cloudstack.git
• String parameter
• NAME > GROUP_ID
• DEFAULT VALUE  > org/apache/cloudstack
• Git Repository
• Url> $PROJECT_REPOSITORY
• Branch Specifier > master
• Build
• Invoke Maven 3
• Maven version > Maven
• Root pom > pom.xml
• Goals and options > clean install -P developer,systemvm -D simulator -D nonoss
• Execute shell
• git checkout -b build-$MAJOR_VERSION_NUMBER.$BUILD_NUMBER
• git commit -a -m 'Build $MAJOR_VERSION_NUMBER.$BUILD_NUMBER
• Execute Shell

echo "Getting nonoss patches"
LIBS=$JENKINS_HOME/jobs/mgmt-update-nonoss-libs/workspace

#Remove old jars
rm -fr deps/*.jar deps/XenServerJava deps/awsapilib deps/*.mar

#Replace with latest from https://github.com/vogxn/cloudstack-nonoss.git cloned into mgmt-update-nonoss-libs
cp -r $LIBS/*.jar $LIBS/XenServerJava/ $LIBS/awsapi-lib/ $LIBS/*.mar deps

#install the non-oss jars into .m2
cd deps
bash -x install-non-oss.sh

#now build the non-oss profile
cd $WORKSPACE
mkdir -p /var/lib/jenkins/jobs/CodeCommit/workspace/services/console-proxy/server/dist

mvn clean install -P developer,systemvm -D simulator -D nonoss

• Post build actions
• Archive for cloud workspace scm
• Files to include in cloned workspace > **
• Criteria for build to be archieved > Most recent completed build
• Archive method > Gzipped
• Trigger paramterized build on other projects
• Projects to build > static-analysis
• Trigger when build is > Stable

Static Analysis:

• Discard old builds > Log Rotation
• Days to keep builds > 5
• Max # of builds to keep > 5
• Source Code Management
• Clone workspace
• Parent project > build-master-nonoss
• Criteria for parent build > Most Recent Completed Build
• Post build actions
• Sonar
• Root pom > pom.xml
• MAVEN_OPTS > -Dsonar.profile=Cloudstack
• Trigger parameterized build on other projects
• Projects to build > start-simulator
• Trigger when build is > Stable

Start simulator:

• Discard old builds > Log Rotation
• Days to keep builds > 5
• Max # of builds to keep > 5
• Build
• Execute shell

mvn -Pdeveloper -pl developer -Ddeploydb
mvn -Pdeveloper -pl developer -Ddeploydb-simulator
export OLD_BUILD_ID=$BUILD_ID
export BUILD_ID=dontKillMe
daemonize -c . -o log.txt /opt/apache-maven-3.0.5/bin/mvn -pl client jetty:run
export BUILD_ID=$OLD_BUILD_ID
while ! nc -vz localhost 8096; do sleep 10; done
export http_proxy=""
nosetests -v --with-marvin --marvin-config=setup/dev/advanced.cfg -w /tmp

• Trigger paramaterized build on other projects
• Projects to build > integration-tests.
• Trigger when build is > stable

Integration tests:

Use a configuration matrix for this

• Discard old builds > Log Rotation
• Days to keep builds
• Max # of builds to keep
• Configuration Matrix
• User defined axis
• Name: suite
• Values
• test_affinity_groups
• test_deploy_vm
• test_deploy_vm_with_userdata
• test_disk_offerings
• test_global_settings
• test_guest_vlan_range
• test_internal_lb
• test_network_acl
• test_nic
• test_non_contigiousvlan
• test_portable_publicip
• test_privategw_acl
• test_public_ip_range
• test_pvlan
• test_regions
• test_resource_detail
• Execute touchstone builds first
• Filter > suite="test_vm_life_cycle"
• Required result > "Stable"
• Build environment
• Abort the build if its stuck > true
• Absolute
• Timeout minutes > 1440
• Build
• Execute shell

nosetests --with-xunit --xunit-file=$suite.xml --with-marvin --marvin-config=$WORKSPACE/../../setup/dev/advanced.cfg $WORKSPACE/../../test/integration/smoke/$suite.py --load -a tags=advanced

• Post Build Actions 
• Trigger paramaterized build on other projects
• Projects to build > stop simulator
• Trigger when build is > Complete

Stop Simulator:

• Discard old builds > Log Rotation
• Days to keep builds > 5
• Max # of builds to keep > 5
• Build
• Execute shell
• mvn -pl :cloud-client-ui jetty:stop
• Post build actions
• Trigger paramaterized build on other projects
• Projects to build > package-rhel63-4.2-nonoss,
• Trigger when build is > Stable

Package Binary:

• Discard old builds > Log Rotation
• Days to keep builds > 5
• Max # of builds to keep > 5
• Source code management
• Clone workspace 
• Parent project > build-master-nonoss
• Criteria for parent build > Most recent completed build
• Build
• Execute shell

set -x
PACKAGE_VERSION=4.2.0

cd scripts/vm/hypervisor/xenserver/

if [ ! vhd-util ]; then
    wget http://download.cloud.com.s3.amazonaws.com/tools/vhd-util
fi

cd $WORKSPACE/packaging/centos63/
bash -x package.sh

Progress so far

Google summer of code has been in the coding period for 17 days! So its time for a progress review. I'll admit and say there were times where I was thinking the task might of been too much and way beyond my skill set, but things are getting there. You can judge for yourself and let me know in the comments.

In terms of coding I have been working on cleaning up the current LDAP implementation and extending its API features. I'll use the word cleaning here loosely, because if I'm honest it was really more of a re-write. When I reviewed the code for the current LDAP implementation I didn't think it was in a great position to be extended. There was duplicate code, no central manager and no data access objects for LDAP interaction. Extending on such a foundation would of caused major headaches and possible performance issues in the future.

I started of by creating a plugin to handle connections to ldap and creating objects to model the data it returned. This involved creating the following main(I have excluded VOs and DAOs from this list to highlight functionality) classes:

• LdapManager - Manages all connections with LDAP.
• LdapConfiguration - Supplies all configuration from the cloudstack database.
• LdapUserManager - Handles any interaction with LDAP user information e.g. search for an ldap user.
• LdapUtils - Supplies static helpers e.g. Escape search queries, escapse DNs, get attributes from search queries etc.
• LdapContextFactory - Manages the creation of contexts
• LdapAuthenticator - Supplies an authentication system to Cloudstack

So from this I had a solid foundation to start creating API commands I went ahead and created the following:

• Add Configuration (Support for multi-able LDAP servers added)

• Delete Configuration

• ListLdapUsers - All or via a search on the specified username attribute

• ListLdapConfiguration

Along with this I updated the UI components that currently exist for the configuration of LDAP

I believe the above work puts me on a nice foundation to begin introducing UI features to enable easy user provisioning. Of course there will be some additions to the LDAP API and modifying of other API commands but this should be manageable. I plan to start with this come the second term of the coding period, July 29th (Knowing me it'll probably happen earlier). Until then I'm writing unit tests for the above code and cleaning up any little issues I come across.

The lead Cloudstack mentor, Sebastian will be in Dublin next week on July 10th giving a talk about the Cloudstack API. I have to put together a 5 minute follow up talk about my experience so far and a brief description on how to get involved with the community. If you'd like to come along grab a ticket over here: https://tito.io/tcube/cloudstack-clients-and-wrappers the event is Paddy Power sponsored and you can grab yourself a free pair of underwear just like the ones in the opening photo of this blog posted!

Continuous testing environment

The below screencast is a quick run through of my continuous testing environment.

Creating a cloudstack plugin

Hi Guys,

Its been awhile since my last post. I've been playing about with Jenkins a lot in order to setup a continuous testing environment, but that's going to be saved for another post.

I began making progress on the API commands I need to make. I created an external class for connecting to LDAP just to get some JDNI experience before diving straight into the Cloudstack codebase of over 4000 lines!

After making this external class I started wondering how to integrate it in with Cloudstack. Turns out you can easily do this as a plugin.

I have successfully made a sample plugin and its now just a matter of changing it about a bit to suit my needs. If you wish to view my notes on this I have uploaded them to http://ianduffy.ie/cloudstack/CreatingAPlugin.pdf

Configuring cloudstack to authenticate against LDAP

Cloudstack currently has some basic implementation for authenticating against LDAP. In this post I will detail how I configured it to authenticate against the OpenLDAP server I setup in a past post.

Login as an admin, navigate to the accounts page and create a new user. The domain must match your LDAP domain and the username must match the username on LDAP. The password can be anything as it is ignored.

Go into global settings and enable API access by setting integration.api.port. Finally navigate to http://ip-of-your-manager:8096/client/api?command=ldapConfig and give it the following parameters like the following: host: ldap.clouddev.lan
searchBase: ou=users,dc=clouddev,dc=lan
queryfilter: (&(uid=%u))
binddn: CN=Manager,DC=clouddev,DC=lan
bindpass: PASSWORD
port: 389

 an example url for this would be as follows:

http://ip-of-your-manager:8096/client/api?command=ldapConfig&hostname=ldap.clouddev.lan&searchbase=OU%3DUsers,DC%3Dclouddev,DC%3Dlan&queryfilter=%28%26%28uid%3D%25u%29%29&binddn=CN%3DManager,DC%3Dclouddev,DC%3Dlan&bindpass=PASSWORD&port=389&response=json


Note: In versions 4.2.0> there is a UI feature under global settings for configuring authentication against an LDAP server. However when I used it I found it ran HTML encoding on my query filter. Opened a bug report for this: https://issues.apache.org/jira/browse/CLOUDSTACK-3044

Git command line enhancements

Enhancing your bash prompt for git:

The standard bash prompt normally shows your username, the hostname and the folder you are in. This is handy, but wouldn't it be great if it told you the branch you were on and the state it was in?

Lets go from this:

to this:

Here you can see the "master" denotes the branch I'm in and the asterix lets me know that I've made some modifications.  I also have the full path of where I am displayed. This removes the need for me to use commands like pwd, git branch, and git status a lot.

This is done by modifying your .bashrc file (located within your home folder).

To produce the above I added the following to ~/.bashrc:

 if tput setaf 1 &> /dev/null; then
 tput sgr0
 if [[ $(tput colors) -ge 256 ]] 2>/dev/null; then
  MAGENTA=$(tput setaf 9)
  ORANGE=$(tput setaf 172)
  GREEN=$(tput setaf 190)
  PURPLE=$(tput setaf 141)
  WHITE=$(tput setaf 7)
 else
  MAGENTA=$(tput setaf 5)
  ORANGE=$(tput setaf 4)
  GREEN=$(tput setaf 2)
  PURPLE=$(tput setaf 1)
  WHITE=$(tput setaf 7)
 fi
 BOLD=$(tput bold)
 RESET=$(tput sgr0)
else
 MAGENTA="\033[1;31m"
 ORANGE="\033[1;33m"
 GREEN="\033[1;32m"
 PURPLE="\033[1;35m"
 WHITE="\033[1;37m"
 BOLD=""
 RESET="\033[m"
fi

export MAGENTA
export ORANGE
export GREEN
export PURPLE
export WHITE
export BOLD
export RESET

function parse_git_dirty() {
 [[ $(git status 2> /dev/null | tail -n1) != *"working directory clean"* ]] && echo "*"
}

function parse_git_branch() {
 git branch --no-color 2> /dev/null | sed -e '/^[^*]/d' -e "s/* \(.*\)/\1$(parse_git_dirty)/"
}

export PS1="\[${BOLD}${MAGENTA}\]\u \[$WHITE\]at \[$ORANGE\]\h \[$WHITE\]in \[$GREEN\]\w\[$WHITE\]\$([[ -n \$(git branch 2> /dev/null) ]] && echo \" on \")\[$PURPLE\]\$(parse_git_branch)\[$WHITE\]\n\$ \[$RESET\]"
export PS2="\[$ORANGE\]→ \[$RESET\]"

Prettifying git and creating some aliases:

Using our git configuration file it is possible to add aliases for certain commands. Along with this it is possible to set colors to denote different outputs. Below is an extract of some of the things within my ~/.gitconfig

[alias]
 # View the SHA, description, and history graph of the latest 20 commits
 l = log --pretty=oneline -n 20 --graph
 # View the current working tree status using the short format
 s = status -s
 # Show the diff between the latest commit and the current state
 d = !"git diff-index --quiet HEAD -- || clear; git diff --patch-with-stat"
 # `git di $number` shows the diff between the state `$number` revisions ago and the current state
 di = !"d() { git diff --patch-with-stat HEAD~$1; }; git diff-index --quiet HEAD -- || clear; d"
 # Pull in remote changes for the current repository and all its submodules
 p = !"git pull; git submodule foreach git pull origin master"
 # Clone a repository including all submodules
 c = clone --recursive
 # Commit all changes
 ca = !git add -A && git commit -av
 # Switch to a branch, creating it if necessary
 go = checkout -B
 # Show verbose output about tags, branches or remotes
 tags = tag -l
 branches = branch -a
 remotes = remote -v
 # Credit an author on the latest commit
 credit = "!f() { git commit --amend --author \"$1 <$2>\" -C HEAD; }; f"
 # Interactive rebase with the given number of latest commits
 reb = "!r() { git rebase -i HEAD~$1; }; r"

[color]
 # Use colors in Git commands that are capable of colored output when outputting to the terminal
 ui = auto
[color "branch"]
 current = yellow reverse
 local = yellow
 remote = green
[color "diff"]
 meta = yellow bold
 frag = magenta bold
 old = red bold
 new = green bold
[color "status"]
 added = yellow
 changed = green
 untracked = cyan

Credits to Mathias Bynens dotfiles project for the above code extracts.

Setting up an OpenLDAP server

As mentioned in an earlier post I will require an LDAP server for testing my project. I will be using OpenLDAP on Centos 6.4.

Lets kick things off by installing some packages:

 $ sudo yum install openldap openldap-clients openldap-servers

Generate a password for your Manager/Administrator user:

 $ slappasswd

This will generate a SHA hash something like this:

{SSHA}q6sOQ5FGWkU6YE5H+awaGZj8UKpLVkBH

This needs to be inserted into the servers configuration file so note it down.

Open up /etc/openldap/slapd.d/cn\=config/olcDatabase\={2}bdb.ldif and modify it so it looks like the following:

 ...................................
olcReadOnly: FALSE
olcRootDN: cn=Manager,dc=my-domain,dc=com
olcRootPW: {SSHA}q6sOQ5FGWkU6YE5H+awaGZj8UKpLVkBH
olcSyncUseSubentry: FALSE
olcMonitoring: TRUE
...................................

Next we need to configure our domain, you can do this by using the replace(replace dc=my-domain,dc=com) feature in your editor or you can use sed like I have done below:

 $ sed -i -e 's/dc=my-domain,dc=com/dc=clouddev,dc=lan/g' /etc/openldap/slapd.d/cn\=config/olcDatabase\={2}bdb.ldif
 $ sed -i -e 's/dc=my-domain,dc=com/dc=clouddev,dc=lan/g' /etc/openldap/slapd.d/cn\=config/olcDatabase\={1}monitor.ldif

Auto start OpenLDAP:

 $ chkconfig slapd start
 $ service slapd start

Populate it:

For this example I will only populate the root directory and will configure the rest later using phpldapadmin. If you wish you can create users/groups using ldapadd and ldif files.

 $ echo -e "dn: dc=clouddev,dc=lan\nobjectClass: dcObject\nobjectClass: organization\ndc: clouddev\no : clouddev" > /tmp/base.ldif
 $ ldapadd -f /tmp/base.ldif -D cn=Manager,dc=clouddev,dc=lan -w password

Configure iptables:

Since I'm just doing this for a development environment I just turned of iptables completely:

 $ iptables --flush
 $ service iptables stop
 $ chkconfig iptables off

If you wish to configure them simply insert:

-A INPUT -p tcp --dport 389 -j ACCEPT

into /etc/sysconfig/iptables

Finally test that your server is up and working by querying it:

 $ ldapsearch -h localhost -b dc=clouddev,dc=lan -xxx

I didn't wish to spend time doing ldif dumps and applying them with ldapadd/ldapdelete/ldapmodify so I opted to install phpldapadmin which supplies a web based interface for managing OpenLDAP.

Start by setting up the EPEL repos on CentOS:

 $ wget http://ftp.riken.jp/Linux/fedora/epel/RPM-GPG-KEY-EPEL-6
 $ rpm --import RPM-GPG-KEY-EPEL-6 
 $ wget http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
 $ rpm -ivh epel-release-6-8.noarch.rpm 

Continue on to install phpldapadmin:

 $ yum --enablerepo=epel install phpldapadmin

Configure the access you require to it by modifying the allow access from line in /etc/httpd/config.d/phpldapadmin.conf

Next we need to change an option in phpldapadmin's configuration file to use a dn for login instead of a uid.

 $ sed -i -e "s/$servers->setValue('login','attr','uid');/\/\/$servers->setValue('login','attr','uid');/g" -e "s/\/\/$servers->setValue('login','attr','dn');/$servers->setValue('login','attr','dn');/g" /etc/phpldapadmin/config.php

Finally restart the httpd:

 $ service httpd restart

browse to http://server-address/ldapadmin and login with
username: cn=Manager,dc=clouddev,dc=lan
password: password

and create your wanted OUs, users, groups, etc.