In my last post I wrote about setting up a jenkins server and said that one of my major motivations for doing so would be to automate my whole application tests. In this post I’m going to describe how to write those tests using Ruby’s rspec testing framework and why I chose to do so.
From Applescript to Ruby
For quite a while I’ve used a set of whole-app tests based on applescript and written using the ASUnit testing framework. This approach is pretty appealing since Applescript is the standard way to script Cocoa apps and ensures that tests are based on the same scripting mechanism as is provided to users. Over time though I found that I simply wasn’t writing new tests, and a quick look at the tests I had would reveal that they hardly covered any of my App’s functionality. The main reasons for this (apart from my laziness) are that;
- Applescript feels verbose and its syntax is unfamiliar to most programmers
- ASUnit is a pretty bare-bones testing framework
- My tests were limited to the functionality I was willing to expose to scripters
The bottom line is that I simply wasn’t effectively testing my app. I felt like I needed a fresh approach, and chose a system based on Ruby and it’s popular rspec framework. After translating all my old tests to this new system I found that they were much more succinct and readable. I was also able to quickly add new tests and have finally started to see the benefits of better test coverage through early identification of bugs.
More readable tests with rspec
My goal was to make writing and reading my tests much more enjoyable. I was already familiar with the rspec framework for ruby and since I find ruby code enjoyable to write and read, I decided to start by rewriting all my Applescript tests using rspec (testing Cocoa apps via rspec isn’t limited to Applescript functionality though). My goal was to be able to write tests something like this;
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27
There are lots of nice things about rspec, (and probably lots more I don’t know about). The example above makes use of nested describe blocks reflecting the object hierachy I want to test
Another nice thing about rspec is that it provides lots of mechanisms for organising the code required to prepare objects for testing. This preparation phase is often fairly involved and in some cases it involves expensive operations such as launching the app or performing some file operations. Being able to use nested contexts in rspec means that I can organise my preparatory code in a hierarchy of expense. For example I want to relaunch the app with a clean database between each high level group of tests, but I don’t want to do this to test each property of an object (simply creating a fresh object would do).
Finally rspec (and ruby) provide lots of mechanisms for making tests read nicely. In particular it’s easy to write custom rspec matchers, so for example I can use the following highly readable syntax to test if files exist where I expect them to be (useful for DropSync syncing tests).
1 2 3 4 5 6 7 8
Ruby and Cocoa
There are two major projects that allow you to send messages to Cocoa objects using Ruby, RubyCocoa and MacRuby. RubyCocoa is the older of the two projects and works by creating proxy objects in Ruby that forward messages to Objective-C instances. MacRuby is quite different. It is a complete implementation of the Ruby language on top of the Objective-C runtime, which means that when you instantate an object from one of the Cocoa frameworks in MacRuby you’ll actually be working with the object itself and not a proxy. It also means that MacRuby comes with its own ruby interpreter
macruby, and its own rspec.
Since MacRuby is eventually supposed to replace RubyCocoa and seems to be under very active development I initially figured it would be the best choice (I found this post by Steve Madsen particularly helpful when getting started). While I was initially pretty happy with this approach I ran into some nasty runtime crashes when I tried to use some xml parsing gems, and when I tried to send messages over Distributed Objects (for JSTalk). I’m sure that a seasoned macruby user could work around these issues, but I found that I was easily able to switch all my tests to RubyCocoa so I did. I’ll most likely switch back to MacRuby one day as my test code is mostly independent of the approach chosen.
Apple provides two classes that allow you to send Applescript to Cocoa applications,
SBApplication lets you access all the methods in your application’s scripting dictionary, you don’t actually use Applescript syntax to do so.
NSApplescript on the other hand is able to take a string of applescript code and run it as though it were a standalone applescript. I use both classes in my tests.
SBApplication is particularly useful for inspecting object properties, whereas
NSApplescript is great if you actually want to test drive your app just as an Applescript user would.
SBApplication to launch and access my application like this
Then I manipulate my app with some actual Applescript code using
NSApplescript like this
1 2 3 4 5 6 7 8 9 10 11 12 13
And finally I use the application instance I created with
SBApplication to check that the
store object was actually created
JSTalk.ing to the rest of the app