The Turnstone's Bill

Setting Up Jenkins

I’ve just finished setting up a Jenkins instance at home to automate aspects of my developent workflow. I haven’t lived with this system for long but my hope is at least to accomplish the following;

  1. Automated builds … so I can

    • provide an rss feed with regular a alpha or beta build for testers
    • know when a change I make to my AppStore version of DropSync breaks the direct download version or vice versa
  2. Automated tests … so I can

    • run tests that I can’t realistically run every time I hit build in xcode
    • run tests that launch my app and modify app data in a clean or specified environment
    • actually get some use out of my tests (you have to actually run them to do that)
  3. Solve all my problems and achieve a state of blissful enlightenment

Here’s how I set things up. Although I’ve got all this working I haven’t really road tested it yet so I might decide to change some of this down the road.

1. Create shell scripts

Generally speaking everything Jenkins does should be boiled down to commands that can be run in a shell script. I have two main jobs I want to run, building my app, and testing my app so I created shell scripts to perform these tasks that I could ask Jenkins to invoke. I added these to my source code repository.

My xcode build script is very simple it looks like this;

#!/bin/bash

#Scheme should be given as a commandline argument. eg build.sh DropSync\ App or build.sh DropSync
scheme=$1

schemebuildroot=$HOME/.xcbuildroot/$scheme
mkdir -p $schemebuildroot/dst
mkdir -p $schemebuildroot/sym
mkdir -p $schemebuildroot/obj

xcodebuild clean build archive -scheme $scheme DSTROOT=$schemebuildroot/dst SYMROOT=$schemebuildroot/sym OBJROOT=$schemebuildroot/obj 

The script takes one argument which is the scheme I want to build, so I can use the same script to build my app two different ways for example;

build.sh DropSync
build.sh DropSync\ App  # My Appstore version which has a different scheme

Another choice I made with this script was to use ~/.xcbuildroot/ as the build root. The reason for this choice is so that my built products will allways be in a predictable place on both my normal user account and for the Jenkins user. You may find that this isn’t necessary for your setup, but in my case I wanted to run Applescript and JSTalk based tests using my built app, and needed to be able to provide the path to the built app to those scripts. By default xcode places build products under a directory like ~/Library/Developer/Xcode/DerivedData/<ProjectName>-<hashstring>/ which means that finding paths to built executables requires some breakable logic (eg look for a directory that starts with ProjectName, but what if two exist and what if Apple changes this location). To avoid all this I just use a fixed directory relative to the user and make sure that I don’t use conflicting scheme names.

2. Setup Jenkins

I’m recording my setup steps here for posterity, as documentation for myself and in the hope that others find it useful. Bear in mind though that I’m pretty much a total novice Jenkins user so this might be a long way from the ideal setup. It seems to work for me so far though.

  1. Create a separate user for Jenkins and login as that user
  2. Download Jenkins and start it up

     mkdir ~/JenkinsApp
     cd ~/JenkinsApp
     curl -L http://mirrors.jenkins-ci.org/war/latest/jenkins.war -o jenkins.war
     java -jar jenkins.war
    
  3. Install the git, github and github OAuth plugins using Jenkins’ admin interface. I did this because I host my projects on Github and wanted Jenkins to kick off a build each time I push some changes.

  4. Login to github and

    • register Jenkins as an application so that the OAuth plugin can work.
    • setup a service hook for the Jenkins Github plugin on relevant github projects. The Hook Url should be something like http://yourdomain:8080/github-webhook/
  5. Enter the Jenkins configuration Menu (Jenkins -> Manage Jenkins -> Configure) and;

    • Enable Security
    • Use github Authentication and settings for the github Authentication plugin
    • Github committer authorization
    • Enable github webhook and allow Jenkins to manage the URLs

3. Create a Jenkins Job

I created a Jenkins job to simply build my app every time a change is pushed to github. Initially I used the Jenkins xcode plugin for this, but eventually migrated to simply running my own build script. I miss out on some goodness with this, especially the fact that Jenkins will create reports from test results. Ultimately though, I figure that if a test fails I simply need to know about it, and I can then investigate the issue directly in xcode without requiring Jenkins. The advantage of using my own build script is that I can use the same script on my development account as Jenkins uses, and this should make troubleshooting easier. Later I’ll talk about my whole-app testing strategy which will make use of this script as a precursor to running tests.

So that’s my Jenkins setup so far. At this point I haven’t really gained very much with my simple build job, but I hope to write another post soon about the next steps in my automation and testing strategy where I try to implement some Applescript whole-app tests.

Rsync and S3 With DropSync

Amazon s3 is a reliable and relatively cost effective way to backup data offsite. Although s3 isn’t a filesystem, Amazon provide an API for accessing s3 data which has allowed developers to create s3fs, a program which presents s3 data as if it existed as files on a network drive.

This means that in theory it is now possible to backup data directly to s3 using DropSync. At this stage though the setup is not for the faint of heart … and there are some serious caveats (see below).

Setting up s3fs

  1. Install xcode and xcode commandline tools
  2. Install Homebrew

    ruby <(curl -fsSk https://raw.github.com/mxcl/homebrew/go)
    
  3. Install s3fs

    brew install s3fs
    

    Then follow instructions from

    brew info fuse4x-kext
    
  4. Mount your bucket with s3fs

    echo "accessKeyId:secretAccessKey" > ~/.passwd-s3fs  
    chmod 600 ~/.passwd-s3fs  
    mkdir -p /path/to/mounts3  
    s3fs bucketname /path/to/mounts3
    
  5. Now your bucket should appear as an accessible shared drive. You can now use dropsync to access this as normal

Caveats

Even though s3fs does a good job of presenting s3 to you as a normal set of files on your filesystem it cannot escape the fact that s3 is not like a normal filesystem. Several issues come up;

  1. S3 has no inherent concept of directories. This means that different programs accessing s3 adopt different conventions about how to encode directory structure. s3fs uses a different convention from amazon’s management console which means that in most cases you will only see directories if they were created by s3fs itself.
  2. S3 is a distributed filesystem and can only guarantee “eventual consistency” which means that its entirely possible to write some data to your s3 bucket only to find that when you read it moments later you don’t see that data. If you wait long enough it should reappear.
  3. Network latency was a big problem for me. This might be because I’m in Australia and we don’t have any Amazon datacenters particularly close ( I think the closest is Singapore)

Steve Jobs

Yesterday, I learned that Steve Jobs’ had died. Strangely I didn’t discover it on my mac, or my iPhone but when my girlfriend told me … and she knew I’d be sad. She knew how much I loved Steve’s creations, the mac, the iPod, the iPhone, the iPad. Most days it’s a bit demeaning to be called a mac “fanboi”, but yesterday it felt like the world understood. I think we all realised that even though our only interactions with Steve Jobs were with his products we’d somehow gotten to know a little bit about the man who created them. We realised that those things we use aren’t just gadgets, they are products designed and crafted by people who care about every detail, who care about aesthetic beauty, good design and craftsmanship. Say what you like about the turtlenecks, but Steve Jobs had that style and he cared about what his company made. Steve’s legacy won’t just be the products though. He changed the way the world saw technology. All those years ago in his parent’s garage Steve Jobs could see that computers would be things that everyday people used, and he spent his life making that happen. Although we’ve come a long way, it still feels like the beginning of that revolution. There are still (and perhaps always will be) so many things that geeks do with technology that everyday people don’t know or care about. In memory of Steve, let’s continue to bring those things to light, and let’s do it with style.