Pivotal Labs

Main menu

Skip to primary content
Skip to secondary content
  • About
  • Case Studies
  • Team
    • Executives
    • Locations
      • San Francisco (HQ)
      • Boston
      • Boulder
      • Denver
      • London
      • Los Angeles
      • New York
  • Community
    • Blogs
    • Tech Talks
    • Events
  • Careers
    • Lifestyle
    • Principles & Practices
    • Benefits
    • FAQ
    • Apply
  • Contact
    • Press Room
    • Press Releases
    • In The News
    • Press Kit
  • All
  • Labs
  • Standup
  • Tracker
Adam Milligan

iPhone UI Automation tests with Jasmine

Adam Milligan
Sunday, July 18, 2010

Since the language of the new iPhone UI Automation component is JavaScript I figured the easiest way to organize tests is to use a JavaScript testing framework, such as Jasmine. So, I created jasmine-iphone, which is little more than a few simple scripts to make UI Automation and Jasmine play nice.

Once you clone jasmine-iphone from GitHub (it includes Jasmine as a submodule, so be sure to git submodule init && git submodule update) you can copy the example-suite.js file, import your spec files, point Instruments at your suite.js and go.

As an example, I set up a trivial example in the Cedar project. The directory structure looks like this:

Project Directory
- Spec
    - UIAutomation
        - jasmine-iphone     <--- submodule
            - jasmine        <--- nested submodule
        - suite.js
        - thing-spec.js
        - other-thing-spec.js

The suite.js file is relatively simple (note that I moved it up one directory from where the example-suite.js file is, so the #import statements are slightly different):

#import "jasmine-iphone/jasmine-uiautomation.js"
#import "jasmine-iphone/jasmine-uiautomation-reporter.js"

// Import your JS spec files here.
#import "thing-spec.js"
#import "other-thing-spec.js"

jasmine.getEnv().addReporter(new jasmine.UIAutomation.Reporter());
jasmine.getEnv().execute();

You can write the specs themselves the same way you’d write Jasmine specs for anything else. The UIAutomation subclass of the Jasmine Reporter takes care of marking the start of each spec, as well as reporting if it passes or fails. You’ll need to use the UIAutomation classes and methods for driving your application’s state, of course.

That’s it. Try it out and see what you think.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Adam Milligan

Cedar device specs and CI

Adam Milligan
Tuesday, July 6, 2010

One of the most common complaints I’ve read about OCUnit, the unit testing framework built into Xcode, is that the tests you write with it won’t run on the device. In addition, I personally have found the process of setting up a target for tests that depend on UIKit confusing and onerous. So, one of our goals for Cedar was to make testing UI elements easy (or easier), by making it easy to run specs in the simulator or on the device.

Probably the second most common complaint I’ve read about OCUnit is that the tests run as part of the build. This makes the test output difficult to separate from the build output, and makes it impossible to use the debugger when running tests. So, in addition to making it easy to run specs on the device, we wanted to be able to run them as a separate, debuggable executable.

Finally, we consider it important that our specs run in our CI system. That means we wanted to be able to run Cedar specs from the command line, and get an exit code signifying success or failure. At the same time, some of us appreciate the value of the green/red feedback for specs passing and failing, so sometimes we like a nice UI. As of today, Cedar will accommodate all of these various requirements.

If you have simple tests that don’t depend on iOS, you can simply link the Cedar dynamic framework into an OS X command line executable build target, write your specs, and run. Running from inside Xcode should give you output that looks something like this:
Xcode spec run pass

For specs that do depend on iOS frameworks, such as custom views or view controllers, you can link the Cedar static framework into an iOS executable, use the Cedar-specific application delegate (CedarAppDelegate), and you get something that looks like this:
Xcode UI spec run pass

The elements in the table represent the top-level specs that you have written. You can click on each one to drill into the state of each of its children, and its childrens’ children, etc.

Of course, those nice green bars aren’t terribly useful for the CI build, so if you’d prefer to skip the UI you can set the CEDAR_HEADLESS_SPECS environment variable, and then running the specs on the simulator (or device) reverts back to the dots:
Xcode headless UI spec run pass

Notice that we’ve created rake tasks for running the specs on the command line, for the sake of simplicity and ease of integration with CCRB. The Rakefile is available in the Cedar project, and does little more than set environment variables and execute bash shell commands. If you’d prefer to use your own bash script, or something similar, feel free to copy and paste your way to happiness.

I’ll spare you another screenshot, but you can see the results and build logs for the Cedar CI build here, and its current status is always displayed on our public CI Monitor page.

The last piece of the testing puzzle for iOS is full-stack integration and lifecycle testing (à la Selenium), which should be possible with Apple’s new UI Automation tool. Unfortunately, Apple hasn’t yet provided a way to run UI Automation tests, or get test results, from the command line. Hopefully we’ll find a way soon; their JavaScript DSL for iOS testing combined with Jasmine could be a potent combination.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Adam Milligan

Objective-C exceptions thrown inside methods invoked via NSInvocation are uncatchable

Adam Milligan
Thursday, July 1, 2010

Whether you’re using Cedar or not, if you’ve upgraded to the iOS 4.0 SDK you may have run into some odd behavior with exception handling blocks not catching exceptions. Strangely, the problem isn’t due to the exceptions themselves (at least not in any obvious way), but with how you call functions that raise exceptions. An exception thrown from within a method you invoke directly will function as expected. However, if you invoke that same function indirectly using NSInvocation any exception thrown becomes uncatchable, crashing the current process regardless of any exception handling code.

This happens only when running against the currently available iOS 4.0 SDK. Exception handling for both direct and indirect invocations performs as expected when using the OS X 10.6 SDK and previous versions of the iPhone SDK.

You can reproduce this problem yourself easily enough. Just create any old iPhone project, and add the following code in a method you know will run (e.g. viewDidLoad on the main controller):

@try {
    NSException *exception = [NSException exceptionWithName:@"foo" reason:@"bar" userInfo:nil];
    [exception raise];
} @catch (NSException *x) {
    NSLog(@"========================> %@", x);
}

Works, right? Now try this (which should be functionally identical):

@try {
    NSException *exception = [NSException exceptionWithName:@"foo" reason:@"bar" userInfo:nil];

    SEL selector = @selector(raise);
    NSMethodSignature *signature = [exception methodSignatureForSelector:selector];
    NSInvocation *invocation = [NSInvocation invocationWithMethodSignature:signature];

    [invocation setTarget:exception];
    [invocation setSelector:selector];
    [invocation invoke];
} @catch (NSException *x) {
    NSLog(@"========================> %@", x);
}

BOOM! Stack trace. Sad face.

Unfortunately, the OCHamcrest matcher library uses indirect method invocations to raise matcher failure exceptions. So, with iOS 4.0 rather than getting failure messages in your test output, you now get a stack trace and potentially a bunch of un-run tests.

Hopefully Apple will fix this soon.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Adam Milligan

Cedar has moved to Pivotal's GitHub account

Adam Milligan
Sunday, June 13, 2010

For anyone following the Cedar project on GitHub, I’ve moved it from my personal GitHub account to Pivotal’s GitHub account. This isn’t a fork, it’s a new repository that from here on out I’ll use as the main Cedar repository.

If you’re following Cedar, please follow the repo on the Pivotal account. If you’ve forked it please re-fork from the repo on the Pivotal account.

After enough time passes I’ll probably delete the repo on my account and recreate it as a fork of the Pivotal repo. Sorry for the inconvenience.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Adam Milligan

The run loop

Adam Milligan
Saturday, June 12, 2010

When I started writing Objective-C for iPhone and iPad projects I started to hear about these mystical and all-powerful run loops objects. Everything seems to relate to run loops, all the class references talk about how one object or another depends on run loops, the most basic iPhone application documentation even talks about how the device starts and maintains the main run loop for you because it’s so important. However, I found few straightforward descriptions of what a run loop is or how it works.

So, I read Apple’s Threading Programming Guide. I read about timers and ports and sources, about the starting and stopping and timing out of run loops. After this reading, and a bit of thought, I realized that all the sound and fury surrounding run loops blinded me to the fact that they are actually very, very simple. To wit, run loops are a mechanism for single-threaded asynchronous programming. Sound familiar? Sounds to me a lot like JavaScript.

A lot was made during the recent WWDC of the difference between synchronous and asynchronous network programming, so as an example let’s consider an AJAX request in JavaScript. To get some information from a server you’d send off a request in one function, passing in another function (or functions) for the system to call when it receives a response.

makeARequest({onSuccess: function(response) {
  // do some success stuff.
}});

Your initial function exits, and while the request is making its merry way through the series of tubes between the client and your server the system is happily going about its business. But, it has its digital ear to the ground for your response, and when that comes back it calls your function. Voilà, asynchronicity.

That’s a run loop. It has event sources, like some form of network socket on which it listens for your server’s response; it has timers, in the form of setTimeout() and clearTimeout(). The Apple Core Foundation run loops take a fair bit more setup to use — the network request performed by those three lines of JS would involve NSURLConnection, NSURLRequest, some form of delegate object, etc. — but it’s exactly the same idea. It’s waiting for something to happen while you take a nap; when that something happens it tells you about it.

There is one extremely significant difference between the JavaScript implementation and the Core Foundation implementation, at least in my mind: testing. In JavaScript this type of asynchronous code isn’t a run loop, it’s just JavaScript. As such, when you run your tests your code does everything you’d expect it to do; it’s the same interpreter running the tests as the production code, after all. So, when your test setup reaches into your AJAX library and stubs out the part that actually talks to the network everything continues to hum along. If you want, you step behind the curtain and send fake responses back through the AJAX object.

In comparison, a Core Foundation run loop is a special construct that the system sets up for you. When you run an app on your iPhone the device takes care of all the setup, care, and feeding of the main run loop. So, in tests, you if you want to run code that depends on a run loop you have to start the run loop yourself. Worse, starting a run loop blocks the execution of your tests until you stop the run loop. So, if you stub out the network code that actually talks to the network your code may sit and wait forever for a response that can’t possibly come. If you set up your code to return a fake response before starting the run loop your code will receive that response, but the run loop will continue to run, potentially blocking your test code forever.

This is, in my opinion, a significant problem for testing Cocoa projects. Asynchronous programming using run loops is simpler than mucking about with threads, and largely endorsed by Apple via their frameworks. But, there’s no reasonably simple way to test code written this way. You could set up a test run loop with test-specific run loop sources or timeouts to prevent it from blocking forever, but that seems immensely complex when compared to the JavaScript implementation. I would very much like to see some support from Apple in the form of a stub run loop for test environments that provides mechanisms for passing fake data to run loop sources, freezing and incrementing time to test timers and delayed invocations, etc.

I’m interested to hear other ideas you’ve had for how to get around this, or test mechanisms you’d like to see from Apple.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Adam Milligan

BDD-style testing for iPhone projects

Adam Milligan
Thursday, June 3, 2010

A little while ago I wrote about Cedar, a BDD-style testing framework for Objective-C. The responses I received nearly all went something along these lines: “That’s great! Too bad I can’t use it, since I’m writing an iPhone app.”

Hogwash.

I actually wrote Cedar specifically for testing iPhone OS projects we’re working on at Pivotal. To prove it, I’ve started a small public iPhone project that I’ve test-driven entirely with Cedar. You can get the project here (more on that in a bit); it should eventually allow you to log into Pivotal Tracker, see all the delivered stories in a given project, and accept or reject each one. At the moment it does little more than start up and display the Pivotal Chicken*, but it does contain Cedar specs that run on and off the device.

“How is this possible?” you ask. I’ve done two things to make this work:

  1. I separated out all classes that don’t depend on UIKit into a target that builds a static library. The specs for this target run as a console app using the OS X runtime, so no need to worry about runtime support for blocks (assuming you’re running 10.6). Also no need to incur the overhead of starting the emulator every time you run tests. This is a pattern I started using ages ago to make automated testing easier on Win32 client applications, and it works great for all the mobile platforms I’ve worked on. Framework independence means faster tests, and faster tests mean happier programmers. I recommend doing this whether you’re interested in testing with Cedar or not.

  2. Tests for the actual app, which does depend on UIKit and therefore must target the iPhone runtime, run on the emulator (or, in theory, a device) using the PLBlocks iPhone runtime for block support.

Getting the specs up and running takes a few steps, but as we all know good things come to those who wait.

  • Clone, or otherwise obtain the StoryAccepter project. It will probably have a number of missing library references.

  • You’ll need to build Cedar, both the dynamic framework (Cedar.framework) and the static iPhone library (libCedar-iPhone.a), as well as the OCHamcrest and OCMock frameworks. Fix the references in the StoryAccepter project to point to these libraries on your system.

  • If you’re running Leopard you’ll need to install PLBlocks 1.0 for Leopard, and you’ll need to include the runtime and set the compiler for both spec targets; the PLBlocks page has excellent instructions. If you’re running Snow Leopard the project should already contain the runtime, so you’ll just need to download and install the compiler for PLBlocks 1.0.1.

  • Select the DomainSpec target, and make sure you’ve selected the appropriate Mac OS X runtime for your system. Build and run; you should see dots appear in the console window as the specs run.

  • Select the StoryAccepterSpec target, and make sure you’ve selected an iPhone runtime (if you want to try running on a device you’ll have to set up the provisioning, of course). Build and run; the emulator should start up and try to run the app, which will simply run the specs and then exit. You should see dots in the console window, as before.

All of this still has some rough spots, especially the UIKit-dependent specs, but even so I’ve found test driving with hierarchical describe blocks far more pleasant than using OCUnit. Some things I hope to improve on:

  • I imagine the app that runs the UIKit-dependent specs showing a graphical display of test results, perhaps similar to what GHUnit displays when run on the emulator or device.

  • The iPhone doesn’t allow dynamic libraries, and I haven’t found a way to use OCHamcrest or OCMock for UIKit-dependent specs. The folks at Carbon Five describe using these libraries in their tests on the device; I’m curious to know how they pull that off.

  • Once iPhone SDK 4.0 comes out with support for blocks this should all work without the need for the PLBlocks runtime. That won’t help iPad development for the foreseeable future, though.

  • Full-stack integration testing à la Selenium.

  • Lots of other things. What would you like to see added to Cedar?

*Not an officially endorsed Pivotal mascot (yet).

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Adam Milligan

BDD-style testing using Objective-C

Adam Milligan
Sunday, May 2, 2010

As we’ve grown our mobile practice at Pivotal we’ve tried to apply to it the same principles and disciplines that have made our Rails practice successful. Often the one that we have the most difficulty translating is testing. In my experience the testing tools for Objective-C in particular are significantly wanting; there are some out there, but they’re hard to find, often hard to use, and occasionally defective in frustrating ways.

One of the things I found I miss most in testing Objective-C, Java, or C++, is the hierarchical structure for organizing tests that frameworks like RSpec or Jasmine provide. I find nested describes indispensable for managing orthogonal aspects of the classes under test, for handling preconditions, for eliminating redundant setup code, and for generally keeping my sanity. So, when I first heard about the addition of blocks in the GCC compiler for Objective-C the first application that came to mind was testing.

So, I wrote Cedar, a BDD-style framework for writing tests in Objective-C. The code is available here. Perhaps more importantly, Cedar is in its infancy so I’m interested in any suggestions and feedback. To that end, I created a public Tracker project for it here.

A minimal spec in Cedar looks like this:

// FooSpec.m

#include "SpecHelper.h"
SPEC_BEGIN(FooSpec)

describe(@"An example", ^{
  it(@"should be descriptive and helpful", ^{
    ...
  });
});

SPEC_END

A few things to note:

  • Unlike OCUnit, Cedar doesn’t run magically run as part of the build. You have to create an executable target for your specs and run it. I did this because I find looking through the build output for test logging and the like to be cumbersome, among other reasons. This may or may not have been a good choice.
  • Yes, those are C preprocessor macros surrounding the specs. Before you get out the torches and pitchforks keep in mind that Objective-C, unlike Ruby or JavaScript, is a compiled language. This means that all imperative code must be in function or class method of some kind. In order to remove code that provided a distraction from the specs themselves I wrapped as much boilerplate as possible in these macros. When expanded, the code looks like this:
// FooSpec.m

#include "SpecHelper.h"

@interface FooSpec : Spec
@end

@implementation FooSpec

- (void)declareBehaviors {
  describe(@"An example", ^{
    it(@"should be descriptive and helpful", ^{
      ...
    });
  });
}
  • Cedar has no matchers, other than the fail() method. Rather than reinvent the wheel I decided to support using the matchers from the Hamcrest library, available here. Note that you can only get the Objective-C port of Hamcrest by checking out the code from Subversion and building it yourself. I considered committing a pre-built version of the Hamcrest framework into the Cedar repository, but I’m not sure what the accepted approach is for including dependencies like that in Objective-C projects. Feedback welcome.
  • All of this obviously depends upon the support for blocks provided by the GCC compiler for Objective-C. Unfortunately, this means you can only use Cedar on a Mac. Far more unfortunately, it means you have to build your specs for a runtime that supports blocks; at the moment this is only the Mac OS X 10.6 runtime. The iPhone OS runtime doesn’t support blocks (although 4.0 may), and the Mac OS X 10.5 runtime doesn’t support blocks. However, all is not lost. Plausible Labs provides patched versions of the GCC compiler and runtimes for iPhone OS and Mac OS X 10.5. I built much of Cedar on a Leopard machine with the PLBlocks compiler; I haven’t tried building for iPhone OS yet, I look forward to hearing about any experiences.
  • Don’t try to mix blocks with Objective-C++, at least not yet. I tried it for some time and ran into any number of internal compiler errors. Hopefully this will improve in the future. As some will astutely point out, I could have used the anonymous functions introduced by C++0x (and supported by GCC). Unfortunately (from Wikipedia):

    If a closure object containing references to local variables is invoked after the innermost block scope of its creation, the behaviour is undefined.

  • There’s no need to provide a header file for your specs, since Cedar finds the specs by introspection.

As I’m sure will soon become entirely obvious this is very much a minimal viable product for Cedar. You can create and nest describe blocks, create examples and beforeEach blocks, and that’s about it. I’m curious to see if people will use something like this; if they do, I’m hoping for plenty of feedback. I’m attached to basically nothing about the framework at the moment (including the name), so please send me a note or join the Tracker project if there’s something you’d like to see added, removed, or changed.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter

Topics

  • agile (778)
  • rails (113)
  • testing (87)
  • ruby (83)
  • ruby on rails (70)
  • jobs (62)
  • javascript (54)
  • techtalk (44)
  • rspec (38)
  • activerecord (29)
  • productivity (29)
  • gogaruco (29)
  • ironblogger (29)
  • git (28)
  • nyc (27)
  • rubymine (25)
  • mobile (22)
  • bloggerdome (21)
  • cucumber (20)
  • process (19)
  • pivotal tracker (19)
  • jasmine (19)
  • design (18)
  • ios (18)
  • webos (17)
  • objective-c (17)
  • android (16)
  • palm (16)
  • "soft" ware (16)
  • fun (15)
  • tracker ecosystem (15)
  • ci (15)
  • cedar (15)
  • rails3 (14)
  • performance (14)
  • bdd (14)
  • gem (13)
  • tdd (13)
  • selenium (12)
  • css (12)
  • goruco (12)
  • bundler (12)
  • meetup (11)
  • railsconf (11)
  • nyc-standup (11)
  • capybara (10)
  • mac (10)
  • mojo (10)
  • chef (10)
  • api (10)
Subscribe to objective-c Feed
  1. ←
  2. 1
  3. 2
  • About
  • Case Studies
  • Team
  • Community
  • Careers
  • Contact
  • Labs
  • Events

Contact Us

contact@pivotallabs.com
+1 415-77-PIVOT
TwitterLinkedInFacebook

Pivotal Tracker

Tracker is the award-winning agile project management tool that enables real-time collaboration around a shared, prioritized backlog.
Visit pivotaltracker.com >