The Turnstone's Bill

Testing Cocoa Apps With Ruby and Applescript

In my last post I wrote about setting up a jenkins server and said that one of my major motivations for doing so would be to automate my whole application tests. In this post I’m going to describe how to write those tests using Ruby’s rspec testing framework and why I chose to do so.

From Applescript to Ruby

For quite a while I’ve used a set of whole-app tests based on applescript and written using the ASUnit testing framework. This approach is pretty appealing since Applescript is the standard way to script Cocoa apps and ensures that tests are based on the same scripting mechanism as is provided to users. Over time though I found that I simply wasn’t writing new tests, and a quick look at the tests I had would reveal that they hardly covered any of my App’s functionality. The main reasons for this (apart from my laziness) are that;

  • Applescript feels verbose and its syntax is unfamiliar to most programmers
  • ASUnit is a pretty bare-bones testing framework
  • My tests were limited to the functionality I was willing to expose to scripters

The bottom line is that I simply wasn’t effectively testing my app. I felt like I needed a fresh approach, and chose a system based on Ruby and it’s popular rspec framework. After translating all my old tests to this new system I found that they were much more succinct and readable. I was also able to quickly add new tests and have finally started to see the benefits of better test coverage through early identification of bugs.

More readable tests with rspec

My goal was to make writing and reading my tests much more enjoyable. I was already familiar with the rspec framework for ruby and since I find ruby code enjoyable to write and read, I decided to start by rewriting all my Applescript tests using rspec (testing Cocoa apps via rspec isn’t limited to Applescript functionality though). My goal was to be able to write tests something like this;

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
include 'spec_helpers.rb'
describe "my app" do
  include_context :clean_startup
  include_context :clean_shutdown

  describe "object" do
      before :each do
          @the_object = applescript_create_the_object @app_context
      end

      describe "some action on the object" do
          before :each do
              # (eg set properties, call method etc)
          end

          subject {@the_object}
          its(:property1) { should == "expected value"}
          its(:property2) { should == "expected value"}
          #...
          #...
      end

      # Testing other aspects of the object

  end
  # Testing other objects
end

There are lots of nice things about rspec, (and probably lots more I don’t know about). The example above makes use of nested describe blocks reflecting the object hierachy I want to test App->Object->Property.

Another nice thing about rspec is that it provides lots of mechanisms for organising the code required to prepare objects for testing. This preparation phase is often fairly involved and in some cases it involves expensive operations such as launching the app or performing some file operations. Being able to use nested contexts in rspec means that I can organise my preparatory code in a hierarchy of expense. For example I want to relaunch the app with a clean database between each high level group of tests, but I don’t want to do this to test each property of an object (simply creating a fresh object would do).

Finally rspec (and ruby) provide lots of mechanisms for making tests read nicely. In particular it’s easy to write custom rspec matchers, so for example I can use the following highly readable syntax to test if files exist where I expect them to be (useful for DropSync syncing tests).

1
2
3
4
5
6
7
8
"/a/path".should exist_as_path

# exist_as_path is a custom matcher written as follows
RSpec::Matchers.define :exist_as_path do
  match do |path|
          Pathname.new(path).exist?
      end
end

Ruby and Cocoa

There are two major projects that allow you to send messages to Cocoa objects using Ruby, RubyCocoa and MacRuby. RubyCocoa is the older of the two projects and works by creating proxy objects in Ruby that forward messages to Objective-C instances. MacRuby is quite different. It is a complete implementation of the Ruby language on top of the Objective-C runtime, which means that when you instantate an object from one of the Cocoa frameworks in MacRuby you’ll actually be working with the object itself and not a proxy. It also means that MacRuby comes with its own ruby interpreter macruby, and its own rspec.

Since MacRuby is eventually supposed to replace RubyCocoa and seems to be under very active development I initially figured it would be the best choice (I found this post by Steve Madsen particularly helpful when getting started). While I was initially pretty happy with this approach I ran into some nasty runtime crashes when I tried to use some xml parsing gems, and when I tried to send messages over Distributed Objects (for JSTalk). I’m sure that a seasoned macruby user could work around these issues, but I found that I was easily able to switch all my tests to RubyCocoa so I did. I’ll most likely switch back to MacRuby one day as my test code is mostly independent of the approach chosen.

Talking Applescript

Apple provides two classes that allow you to send Applescript to Cocoa applications, SBApplication and NSApplescript. While SBApplication lets you access all the methods in your application’s scripting dictionary, you don’t actually use Applescript syntax to do so. NSApplescript on the other hand is able to take a string of applescript code and run it as though it were a standalone applescript. I use both classes in my tests. SBApplication is particularly useful for inspecting object properties, whereas NSApplescript is great if you actually want to test drive your app just as an Applescript user would.

I use SBApplication to launch and access my application like this

1
2
built_dropsync_url=NSURL.fileURLWithPath(built_dropsync_path)       
dropsync = SBApplication.applicationWithURL(built_dropsync_url)

Then I manipulate my app with some actual Applescript code using NSApplescript like this

1
2
3
4
5
6
7
8
9
10
11
12
13
# Returns result on success and a dictionary of errorInfo on failure
#
def send_applescript code
  as=NSAppleScript.new.initWithSource(code)
  result, errorInfo=as.executeAndReturnError_
  if ( result==nil)
      return errorInfo
  else
      return result
  end
end

send_applescript "tell application \"DropSync\" to make new store"

And finally I use the application instance I created with SBApplication to check that the store object was actually created

1
dropsync.stores.length.should eql 1

JSTalk.ing to the rest of the app

Creating an Applescript dictionary that allows users to comprehensively drive your app is alot of work. For my app I’ve created a scripting dictionary that tries to be reasonably comprehensive but is still limited to what users might realistically want scriptable. For testing purposes though, I want much more control, and that’s where JSTalk steps in. JSTalk is a project by Gus Mueller that vends a root object in your app and allows that object to be messaged using Distributed Objects. While JSTalk is primarily aimed at scripting apps with Javascript it’s easy to use ruby instead. I’m currently only using JSTalk in a few tests to fill gaps in my Applescript functionality but I expect to be using it more as my tests become more comprehensive. If you’re wondering how to use JSTalk with ruby just download the JSTalk project and checkout Gus’s examples. He’s got a handy JSTalk.py script in there that can easily be translated to Ruby.

Comments