Pivotal Labs

Main menu

Skip to primary content
Skip to secondary content
  • About
  • Case Studies
  • Team
    • Executives
    • Locations
      • San Francisco (HQ)
      • Boston
      • Boulder
      • Denver
      • London
      • Los Angeles
      • New York
  • Community
    • Blogs
    • Tech Talks
    • Events
  • Careers
    • Lifestyle
    • Principles & Practices
    • Benefits
    • FAQ
    • Apply
  • Tools
  • Contact
    • Press Room
    • Press Releases
    • In The News
    • Press Kit
  • All
  • Labs
  • Standup
  • Tracker

Why not to use ARC

Adam Milligan
Sunday, October 7, 2012

If you have done any development for iOS in the past few years you have at least some familiarity with ARC. The overall response to ARC since Apple released it with iOS 5 has been little short of orgasmic. You can’t swing a dead internet cat without hitting a blog post from someone explaining how ARC saved his/her dying grandmother and if you’re not using it on every project you touch then you’re helping the Commies win.

I’ve seen some projects do perfectly well with ARC, but at the same time I feel it provides its own set of challenges which we should not overlook. Here are some reasons why you might want to consider not using ARC on your next Objective C project.

ARC probably isn’t solving the problem you need it to

Memory management. Programmers whisper these words in fearful tones, or brazenly avow the impossibility of doing it correctly. ARC takes care of memory management, thus solving one of the great problems of our generation, right?

As it turns out, memory management is actually quite simple; /relationship/ management presents the challenge. Memory problems are usually a result of poor relationship management. Object A leaks because objects B and C both have an ownership relationship to A, A has an ownership relationship to C, etc.

In simple cases ARC will clean up the unused memory for you, but you’re still left with a poorly designed object graph, and all its associated problems. In more complex cases you can have strong circular references — a common result of a messy object graph — and then even ARC can’t prevent the leaks.

ARC makes easy things easier, but difficult things more difficult

Here’s a simple example of a class that has a simple relationship to another class:

Class declaration without ARC:

@interface Person : NSObject
@property (nonatomic, retain) Wallet *wallet;
@end

Class declaration with ARC:

@interface Person : NSObject
@property (strong, nonatomic) Wallet *wallet;
@end

Not much difference here. Here’s a bit of the implementation:

Initializer without ARC:

- (id)init {
    if (self = [super init];) {
        self.wallet = [[[Wallet alloc] init] autorelease];
    }
    return self;
}

- (void)dealloc {
    self.wallet = nil;
    [super dealloc];
}

Initializer with ARC:

- (id)init {
    if (self = [super init];) {
        self.wallet = [[Wallet alloc] init];
    }
    return self;
}

In this simple case ARC makes life a little easier; you don’t have to type autorelease, and you don’t have to write a dealloc method to release owned objects. With a well designed object graph, these two things are the vast majority of memory management you need to do. They’re more tedious than difficult, but ARC makes them go away. That’s pretty handy, right?

Now let’s look at an example of retrieving a value from the iOS Keychain, which is a bit more complicated:

Without ARC:

NSMutableDictionary *query = [NSMutableDictionary dictionary];
[query setObject:kSecClassGenericPassword forKey:kSecClass];
[query setObject:[NSData dataWithBytes:SOME_ID length:ID_LENGTH] forKey:kSecAttrGeneric];
[query setObject:(id)kCFBooleanTrue forKey:kSecReturnData];

NSData *result = nil;
NSString *value = nil
if (errSecSuccess == SecItemCopyMatching((CFDictionaryRef)query, (CFTypeRef *)&result)) {
    value = [[[NSString alloc] initWithData:[result autorelease] encoding:NSUTF8StringEncoding] autorelease];
}

With ARC:

NSMutableDictionary *query = [NSMutableDictionary dictionary];
[query setObject:(__bridge id)kSecClassGenericPassword forKey:(__bridge id)kSecClass];
[query setObject:[NSData dataWithBytes:SOME_ID length:ID_LENGTH] forKey:(__bridge id)kSecAttrGeneric];
[query setObject:(id)kCFBooleanTrue forKey:(__bridge id)kSecReturnData];

CFDataRef result = NULL;
NSString *value = nil;
if (errSecSuccess == SecItemCopyMatching((__bridge CFDictionaryRef)query, (CFTypeRef *)&result)) {
    value = [[NSString alloc] initWithData:(__bridge_transfer NSData *)result encoding:NSUTF8StringEncoding];
}

ARC saves us from two autorelease calls, but at the cost of five __bridge casts and a __bridge_transfer cast. ARC won’t let you declare the result variable as an NSData, since ARC disallows casting indirect pointers entirely. Thus, when you receive the result in a CFDataRef you’re still responsible for releasing that memory. You could use CFRelease() on it, or, as shown here, __bridge_transfer the CFDataRef into an NSData * temporary that ARC cleans up at the end of the execution of the statement.

Not exactly simpler, is it?

Sometimes ARC doesn’t help at all

Do you see the memory leak in this code?

self.thing = [Thing thing];
self.thing.completeNotification = ^{
    [self spreadTheWord];
};
[self.thing startTask];

By default, Objective C blocks retain everything they refer to from the local scope, including the self pointer. In this case, self retains the Thing, the Thing retains the block, and the block retains self. This creates a circular reference and a memory leak, with or without ARC.

Here’s how you fix it without ARC:

__block ThisClass *this = self;
self.thing.completeNotification = ^{
    [this spreadTheWord];
};

and with ARC:

__weak ThisClass *this = self;
self.thing.completeNotification = ^{
    [this spreadTheWord];
};

Even with ARC you still have to think about managing memory in some cases. If you rely on ARC to handle all of your memory this type of leak is probably more likely to happen.

The ARC compiler is broken

So ARC may not be helping as much as you thought, but at least it’s not really hurting anything, right? Unfortunately, turning on ARC injects bugs into the compiler. Until LLVM 4.1 (in XCode 4.5) code containing C++ templates wouldn’t even compile with ARC enabled. While LLVM 4.1 is a significant improvement over its predecessors, now the code compiles but fails to work correctly. Here’s an example of a simple function template with a specialization:

class Thing {
public:
    template<typename T>
    void do_something(const T &);
};

template<typename T>
void Thing::do_something(const T &) {
    NSLog(@"Do something with generic type");
}

template<>
void Thing::do_something(UIView * const &) {
    NSLog(@"Do something with UIView *");
}
</typename></typename>

When invoked:

Thing thing;
thing.do_something(someViewController.view);

here is the output without ARC:

2012-10-07 20:17:57.387 Project[65248:c07] Do something with UIView *

and with ARC:

2012-10-07 20:18:54.400 Project[65984:c07] Do something with generic type

What does C++ template type resolution have to do with Objective C reference counting? Nothing, as far as I can tell. It seems the ARC compiler is working off a different branch than the non-ARC compiler; a branch with significant unrelated defects. Who knows what other problems lie in wait?

Sometimes ARC is really, really broken

If you try digging around in the more esoteric capabilities of Objective C, ARC will sometimes get confused and do the wrong thing without warning you. For instance, if you’d like to change the class of an object at runtime, for example to a proxy class, with object_setClass, ARC won’t make a peep. However, it can get confused and release the object when its class changes, leading to overrelease and EXC_BAD_ACCESS. Worst of all, since ARC is completely out of your control, there’s nothing you can do about it!

Weirdly, for something that is meant to operate 100% at compile time, this behavior changes depending on what platform you run on. It happens in some instances on the simulator for older iOS versions, in different instances on the iOS 6 simulator, and (mercifully) never on a device.

ARC doesn’t really improve performance

This isn’t really a problem with ARC, but I’ve seen a number of blog posts about how great ARC is because it improves performance. These blog posts are usually filled with breathless tales of tail call optimization and assembly language listings. Now, there’s nothing wrong with Objective C from a performance perspective, but if you want to get down to questions of how many processor cycles your code takes then Objective C is pretty slow. Removing a few processor cycles from your iOS app is not going to have a noticeable effect.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter

Cedar Expectations

Adam Milligan
Wednesday, November 2, 2011

As I wrote here we’ve used OCHamcrest matchers for some time for writing expectations in Cedar, but have found them unsatisfying. We wanted convenient matchers, like the ones Jasmine provides for JavaScript, but for Objective C. To that end, we added Cedar-specific expectation functions and matchers that specifically solve the problems we had with OCHamcrest.

To use Cedar’s expectations you need to make a couple small changes to your spec files:

1) Cedar matchers use C++ templates. Tell the compiler to expect some C++ code by changing the file extension on your spec files from .m to .mm.

2) Cedar matchers live in a C++ namespace. At the top of your spec file, after the includes, add this line:

using namespace Cedar::Matchers;

Features

Type Deduction

Cedar matchers determine the types of the values in the expectation, so you don’t have to. Rather than this:

assertThat(someObject, equalTo(anotherObject));
assertThatUnsignedInt(someUInt, equalToUnsignedInt(anotherUInt));

you write this:

expect(someObject).to(equal(anotherObject));
expect(someUInt).to(equal(anotherUInt));

We can also mix and match object and non-object types, in cases where such comparisons make sense. For instance, rather than this:

NSNumber *aNumber = [NSNumber numberWithFloat:1.7];
assertThat(aNumber, equalTo([NSNumber numberWithFloat:1.8));
assertThatFloat([aNumber floatValue], equalToFloat(1.8));  // equivalent

you write this:

NSNumber *aNumber = [NSNumber numberWithFloat:1.7];
expect(aNumber).to(equal(1.8));

Extensibilty

Cedar’s expectations use C++ templates not just for argument types, but for matcher objects as well. This means what goes inside the to() construct (e.g. expect(foo).to(equal(bar));) can be any object that responds to the appropriate methods (more on this later). If you want to write a custom matcher for your system (e.g. expect(user).to(be_logged_in());), no need to subclass anything or link against the Cedar library.

We also separated comparators from matchers for common expectations, such as equality. This means that adding equality comparison for a custom type involves only adding a comparator function for your type, with no need to modify the equality matcher itself. For example, here is the method for comparing two NSNumber objects:

bool compare_equal(NSNumber * const actualValue, NSNumber * const expectedValue) {
    return [actualValue isEqualToNumber:expectedValue];
}

Caveats

Several versions of LLVM have had trouble with properly compiling Objective C++ code that contains blocks, and GCC has never done so correctly. Versions of LLVM before 2.0 (Xcode 3.x) will fail to compile Cedar specs with the Internal Compiler Error” message. Version 2.0 of LLVM (Xcode 4.0.x) will successfully compile Cedar specs, but incorrectly handle reference to temporaries, leading to difficult to understand error messages. Versions 2.1 (Xcode 4.1) and 3.0 (Xcode 4.2) fix this problem, and will work properly. However, most version of Xcode still default to using the GCC/LLVM hybrid, so you may need to change this setting.

Also, C++ templates deduce types at compile time, which means they use the matcher function for the declared type of parameters. For instance, if you have an equality comparator specialized for MyType, which is a subclass of NSObject, the following code will invoke the equality comparator for NSObject, not MyType:

NSObject *foo = [[MyType alloc] init]; // refers to a MyType, but is declared as NSObject *
expect(foo).to(equal(someOtherObject));

In practice this seems to cause problems less frequently than I had expected. However, keep in mind that the declared type of nearly all Objective C initializers is id (NSNumber seems to be the exception to this rule, for some reason). So, if you pass the temporary result of an initializer to a matcher, it will most likely use the specialization of that matcher for id. For instance:

expect([NSString string]).to(equal(@""));  // Compares id to NSString * (works fine)
expect([[MyType alloc] init]).to(equal(@"")); // Compares id to NSString * (might work?)

Finally, because of the way C++ template work, the order of declaration of functions does matter. For instance, the equality matcher delegates comparison to the appropriate overload of the compare_equal function; this allows easy addition of custom comparators, as mentioned above. However, at the point where the compiler finds the equality matcher, it will only match the parameter types to comparator function overloads it has already seen. This mean you need to include your custom comparators before the declaration of the equality matcher. Since Cedar helpfully includes all matchers and comparators, you can do this in one of a few ways:

  1. Import your custom comparators before Cedar’s SpecHelper.h at the top of your file. This should work fine, since your custom matchers and comparators need not included dependencies from Cedar.

  2. Cedar checks a preprocessor value named CEDAR_CUSTOM_COMPARATORS. If you set this value Cedar will treat the value as a file path and #import it before the matcher library. Cedar also have CEDAR_CUSTOM_MATCHERS and CEDAR_CUSTOM_STRINGIFIERS (for providing custom output formats for your types).

  3. For new matchers and comparators for Cocoa types (CGRect, UIView, NSIndexPath, AVPlayer, etc.), feel free to contribute back to Cedar so they get included by default.

Please send any comments, questions, suggestions, feature request, pull requests, etc. to the Cedar discussion mailing list. Unfortunately, messages to the Pivotal account on GitHub get lost in the jillions of messages and notifications we receive.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter

The Trouble With Expectations in Objective C

Adam Milligan
Wednesday, November 2, 2011

At Pivotal we write a lot of tests, or specs if you prefer. We TDD nearly everything we write, in every language we write in and on every platform we write for, so we actively work to improve every aspect of our testing tools. Personally, as I’ve written tests in Objective C I’ve found that the syntax of expectations has left much to be desired.

Like many people who write Objective C, I’ve spent a fair bit of time with languages like Ruby and JavaScript. When I write specs I often yearn for the simplicity of the expectation syntax in those languages. For example, some simple expectations in JavaScript, using Jasmine:

expect(composer.name).toEqual("Ludwig");
expect(composer.symphonies.count).toEqual(9);
expect(composer.symphonies).toContain("Eroica");
expect(composer.symphonies).not.toContain("Appassionata");

In comparison, the same expectations in Objective C, using OCHamcrest:

assertThat(composer.name, equalTo(@"Ludwig"));
assertThatUnsignedInt([composer.symphonies count], equalToUnsignedInt(9u));
assertThat(composer.symphonies, hasItem(@"Eroica"));
assertThat(composer.symphonies, isNot(hasItem(@"Appassionata"));

The challenge

I want three primary attributes from matchers: readability, ease of use, and extensibility.

Readability

With the exception of the unfortunate mismatch between the negation function (isNot) and the containment matcher (hasItem), I find the OCHamcrest expectations reasonably readable. The distinction between the first equality matcher (assertThat/equalTo) and the second (assertThatUnsignedInt/equalToUnsignedInt) is a bit jarring, though.

Ease of use

That distinction is much more of a problem for usability. The problem comes from the Objective C type system’s split personality: some variables refer to Objective C objects (NSNumber *, NSString *, NSArray *, UIView *), while the others refer to built-in types and structs (int, char *, CFArray, CGRect), and never the twain shall meet. A simple equality match must use the isEqual: method for the former, but the == operator, or something more esoteric like strncmp, for the latter. Sadly, a single matcher function has no way to determine the correct equality mechanism to use. Worse, the matcher function declarations must specify the types of their parameters, so expectations for any non-pointer types must have separate declarations for each type. The number of functions for any general purpose expectation explodes:

  • equalTo
  • equalToInt
  • equalToUnsignedInt
  • equalToFloat
  • equalToDouble
  • etc.

I can’t describe how many times I’ve failed to have the type of the assertThatX function for an expectation match the equalToX function. Frustration ensues.

Extensibility

Adding new OCHamcrest matchers isn’t terribly difficult, but you have to derive your matcher object from the HCBaseMatcher class, which means any library that includes custom matchers must link against the OCHamcrest library. If your spec target links against OCHamcrest and your custom matcher library, and your custom matcher library also links against OCHamcrest, suddenly you’re in dependency management hell.

What to do about it?

We’ve used OCHamcrest on a number of projects now, but the usability issues have become an increasing burden.

Peter Kim wrote a nice matcher library named Expecta which solves the type differentiation problem using the Objective C @encode function. We’ve used this on a couple projects, and like it more than OCHamcrest. The above example would look like this in Expecta:

expect(composer.name).toEqual(@"Ludwig");
expect([composer.symphonies count]).toEqual(9);
expect(composer.symphonies).toContain(@"Eroica");
expect(composer.symphonies).Not.toContain(@"Appassionata");

At the same time, we added a set of built-in matchers to Cedar, which solve the type differentiation problem with C++ templates, and which we designed to be extremely extensible. We’re using these matchers on some of our projects as well, and we’ve found they work nicely. Here’s the above example using Cedar matchers:

expect(composer.name).to(equal(@"Ludwig"));
expect([composer.symphonies count]).to(equal(9));
expect(composer.symphonies).to(contain(@"Eroica"));
expect(composer.symphonies).to_not(contain(@"Appassionata"));

You can read more about the Cedar matches, including how to use and extend them, here.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter

Cedar vs. Xcode 4 (round one: the command line)

Adam Milligan
Tuesday, April 19, 2011

I’ve finally found a bit of time to update Cedar to work with Xcode 4, and I hope to have it working smoothly some time in the next few days. However, I’ve already come across my first significant issue with the Xcode 4 changes: the location of build products.

Not unexpectedly, the problem has to do with command line builds using xcodebuild. By default, Xcode 4 now puts build products into a project-specific directory in the “Derived Data” folder; this looks something like /Users/pivotal/Library/Developer/Xcode/DerivedData/Cedar-somegianthashstring/Build/Products/Debug-iphonesimulator/libCedar-StaticLib.a. This isn’t a problem, generally, because the BUILD_DIR compiler variable contains the build directory, should you need to find this location during the build process.

Sadly, when you build from the command line, using the xcodebuild command, the build products still go into the old Xcode 3 build location, but the BUILD_DIR compiler variable contains the new Xcode 4 build directory. This means any script that looks for the build results in the directory specified by BUILD_DIR won’t find anything.

The build target for Cedar’s static framework is simply a script that uses xcodebuild to build the static library for both the simulator and the device, and then uses lipo to make a fat binary from the results. Because it can’t find the build results at the location specified by BUILD_DIR it now fails messily.

The easiest workaround I’ve found is to change where build products go using the Locations setting in the Xcode 4 preferences (details below). Unfortunately, this isn’t a project-specific setting, so you’ll have to change your preferences similarly to make it work. I haven’t found any problems with changing the location of the build products, but this does mean the Cedar static framework (as well as the related static frameworks for OCHamcrest and OCMock) won’t build with the default settings. Unsatisfying.

The longer term solution is for Apple to act on the bug I filed. We’ll see how that goes.

UPDATE: Thanks to Christian Niles for pointing out the SYMROOT environment variable in a pull request. Setting this for command line builds forces Xcode to use the specified location for all build products, and updates the BUILD_DIR compiler variable.

Steps for changing the build location in Xcode 4:

  • Open Xcode preferences (Command-,)
  • Select the “Locations” tab
  • Change the “Build Location” drop down from “Place build products in derived data location” to “Place build products in locations specified by targets.”
  • 0 Shares
  • Share on Facebook
  • Share on Twitter

Colorized output for Cedar

Adam Milligan
Thursday, December 23, 2010

Thanks to Sean Moon and Sam Coward Cedar now has colorized output on the command line:

Colorized Cedar report

If you’d like to display colorized output like this you can specify the appropriate custom reporter class using the CEDAR_REPORTER_CLASS environment variable. We do this in our Rakefiles, like so:

task :specs => :build_specs do
  ENV["DYLD_FRAMEWORK_PATH"] = BUILD_DIR
  ENV["CEDAR_REPORTER_CLASS"] = "CDRColorizedReporter"
  system_or_exit(File.join(BUILD_DIR, SPECS_TARGET_NAME))
end

You can set the environment variable in whatever way works for you. You can also set it to any reporter class you choose, so customize away.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter

iPhone on blocks: UITextFields

Adam Milligan
Saturday, August 7, 2010

If you’ve ever used a UITextField in an iPhone project (or, I suppose, an NSTextField in a Cocoa project) you know that you pass it a delegate object in order to respond to events. Handling the “Return” key press from the on-screen keyboard may look something like this (probably implemented in your view controller):

- (BOOL)textFieldShouldReturn:(UITextField *)textField {
    if (0 == [textField.text length]) {
        return NO;
    }
    [self doSomethingWithText:textField.text];
    [textField resignFirstResponder];
    return YES;
}

The delegate pattern is de rigueur for Cocoa classes, so you’ve likely never given this much special thought. Unless, that is, you decided at some point to have two text fields on screen at once. With two text fields you need to handle two sets of callbacks. You have a couple options for how to do this:

  1. Use the same delegate to handle both sets of callbacks, and use conditionals or switch statements to differentiate between the text fields.
  2. Create UITextField subclasses for each text field, each of which knows how to handle its own events. Each subclass will need a reference to the view controller, and you’ll need to expose methods in the view controller’s public interface for the subclasses to call, in order to effect some change in the system.
  3. Create a separate delegate class for each text field. As in the previous option, each delegate class will need a reference to the view controller and a way to send it messages to effect changes in the system.

None of these options feel particularly satisfactory: the second overuses inheritance, which the delegate pattern exists largely to avoid; both the second and third can result in class explosion; and the first feels so… procedural. Isn’t Object Oriented Programming supposed to save us from problems like this?

Procedural or no, Apple suggests the first option in all of their documentation and example code. An if statement isn’t really a big deal for two text fields; but what about three? Five? Ten? Your delegate method could look something like this:

- (BOOL)textFieldShouldReturn:(UITextField *)textField {
    if (textField == self.fooTextField) {
        if (0 == [textField.text length]) {
            return NO;
        }
        // Do something with the text
        }
    } else if (textField == self.barTextField) {
        // different logic...
    } else if (textField == self.batTextField) {
        // more different logic...
    } else if (textField == self.bazTextField) {
        // yet more different logic...
    } else if (textField == self.wibbleTextField) {
        // still yet more different logic...
    } else if ...
    [textField resignFirstResponder];
    return YES;
}

Or, alternately, you could resort to the dreaded switch statement. Either way, ugh.

The fundamental problem here is that only one object (let’s say a view controller) knows what to do when events occur, but only the subordinate objects (the text fields) know when the events occur. Each text field object encapsulates the behavior of its particular on-screen representation, but can’t access the internal state or implementation details of the view controller in order to effect changes to the system.

The recent addition of blocks to iOS can help us work around this problem. Since blocks are closures they capture and maintain their surrounding state at the point they’re instantiated. We can use this to define an implementation, and capture the internal state of our view controller, and pass all of this to an individual text field. Once done, each text field object will manage its own behavior without any intervention from the view controller whatsoever.

To make this work you’d need one (and only one) new class (theoretically, you could make a subclass of UITextField and set it to be its own delegate; unfortunately, setting a UITextField to be its own delegate seems to create an infinite loop deep in the bowels of Cocoa):

@interface BlockTextFieldDelegate : NSObject <UITextFieldDelegate>
@property (nonatomic, copy) BOOL (^textFieldShouldReturn)(UITextField *textField);
@end

@implementation BlockTextFieldDelegate
@synthesize textFieldShouldReturn = textFieldShouldReturn_;

- (void)dealloc {
    self.textFieldShouldReturn = nil;
    [super dealloc];
}

- (BOOL)textFieldShouldReturn:(UITextField *)textField {
    if (self.textFieldShouldReturn) {
        return self.textFieldShouldReturn(textField);
    }
    return YES;
}

@end

Using the BlockTextFieldDelegate class might look like this (in your view controller):

- (void)viewDidLoad {
    [super viewDidLoad];
    self.textFieldDelegate.textFieldShouldReturn = ^ BOOL (UITextField *textField) {
        if (0 == [textField.text length]) {
            return NO;
        }
        [self doSomethingWithText:textField.text];
        [textField resignFirstResponder];
        return YES;
    };
}

The astute reader, armed with a passing familiarity with Apple’s Human Interface Guidelines, will point out that having several text fields on one screen may create a poor user experience. Certainly true for the iPhone, perhaps less true for the iPad’s larger screen. In any case, this technique works for any situation that requires multiple objects, not just UITextFields, reporting to a single delegate. Consider the case of multiple concurrent network requests, created by NSURLConnection objects, managed by a single view controller.

The astute reader will also point out that you may leave off the return type when defining blocks, assuming the compiler can infer it. I left the return type in to keep the example as explicit as possible, but the above block assignment could look like this (note the missing BOOL return declaration):

self.textFieldDelegate.textFieldShouldReturn = ^ (UITextField *textField) {

The block syntax isn’t beautiful, but you can use this technique to eliminate conditional chains, keep the number of classes and subclasses you create low, and avoid exposing the internal state of your objects; and that is beautiful.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter

iPhone UI Automation tests: a decent start

Adam Milligan
Tuesday, July 20, 2010

Apple’s inclusion of the UI Automation component in Instruments with iOS 4 is a definite step in the right direction. It’s the first reasonable way to write tests that externally exercise your actual app, rather than weirdly injecting test code into it. It’s also the only way to programmatically test lifecycle issues, such as how your app behaves when put in the background, when rotated, when the device locks, etc. Good stuff. Unfortunately, the current implementation of UI Automation also has some significant problems:

  1. There’s no way to run tests from the command line. The subtitle of the WWDC talk for UI Automation was “find bugs while you sleep;” unfortunately, you can’t find bugs while you sleep if you have to wake up to click the “Run” button.
  2. There’s no way to set up or reset state. The lack of fixtures which set up a known state at the beginning of iPhone tests has been a problem for unit testing (with OCUnit, Cedar, or what have you), particularly for apps that use CoreData. Now it’s worse than ever, because UI Automation manipulates the actual state of the app on the device, much like Selenium does in a browser. Sadly, UI Automation provides no method for reseting the device’s state, making it nigh impossible to prevent tests from affecting one another.
  3. Part of the previous problem is that UI Automation has no concept of discrete tests; it provides no form of organization for your test scripts. No test methods, no set up or tear down methods, just one big stream of consciousness line of execution. Obviously you can break this up into functions as you see fit, but why reinvent the wheel? Since the test script is JavaScript, I like the idea of using Jasmine for this.
  4. There’s no way to programmatically retrieve the results of the test run. You could debate the value of solving this issue at the moment, considering there’s no way to programmatically start the tests either. However, even if you were to write some clever AppleScript to kick off the tests automatically the only indication of the pass/fail status is in the Instruments UI, so you still have to wake up to check the results. I searched around a bit for information on deconstructing the protocol that UI Automation uses to talk to the device, but I came up empty.

I’ll definitely use UI Automation, particularly for app lifecycle testing. But, not being able to add those tests to a CI build definitely stings. I very much hope Apple keeps their momentum for automated testing and makes it more developer-friendly.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter

iPhone UI Automation tests with Jasmine

Adam Milligan
Sunday, July 18, 2010

Since the language of the new iPhone UI Automation component is JavaScript I figured the easiest way to organize tests is to use a JavaScript testing framework, such as Jasmine. So, I created jasmine-iphone, which is little more than a few simple scripts to make UI Automation and Jasmine play nice.

Once you clone jasmine-iphone from GitHub (it includes Jasmine as a submodule, so be sure to git submodule init && git submodule update) you can copy the example-suite.js file, import your spec files, point Instruments at your suite.js and go.

As an example, I set up a trivial example in the Cedar project. The directory structure looks like this:

Project Directory
- Spec
    - UIAutomation
        - jasmine-iphone     <--- submodule
            - jasmine        <--- nested submodule
        - suite.js
        - thing-spec.js
        - other-thing-spec.js

The suite.js file is relatively simple (note that I moved it up one directory from where the example-suite.js file is, so the #import statements are slightly different):

#import "jasmine-iphone/jasmine-uiautomation.js"
#import "jasmine-iphone/jasmine-uiautomation-reporter.js"

// Import your JS spec files here.
#import "thing-spec.js"
#import "other-thing-spec.js"

jasmine.getEnv().addReporter(new jasmine.UIAutomation.Reporter());
jasmine.getEnv().execute();

You can write the specs themselves the same way you’d write Jasmine specs for anything else. The UIAutomation subclass of the Jasmine Reporter takes care of marking the start of each spec, as well as reporting if it passes or fails. You’ll need to use the UIAutomation classes and methods for driving your application’s state, of course.

That’s it. Try it out and see what you think.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter

Cedar device specs and CI

Adam Milligan
Tuesday, July 6, 2010

One of the most common complaints I’ve read about OCUnit, the unit testing framework built into Xcode, is that the tests you write with it won’t run on the device. In addition, I personally have found the process of setting up a target for tests that depend on UIKit confusing and onerous. So, one of our goals for Cedar was to make testing UI elements easy (or easier), by making it easy to run specs in the simulator or on the device.

Probably the second most common complaint I’ve read about OCUnit is that the tests run as part of the build. This makes the test output difficult to separate from the build output, and makes it impossible to use the debugger when running tests. So, in addition to making it easy to run specs on the device, we wanted to be able to run them as a separate, debuggable executable.

Finally, we consider it important that our specs run in our CI system. That means we wanted to be able to run Cedar specs from the command line, and get an exit code signifying success or failure. At the same time, some of us appreciate the value of the green/red feedback for specs passing and failing, so sometimes we like a nice UI. As of today, Cedar will accommodate all of these various requirements.

If you have simple tests that don’t depend on iOS, you can simply link the Cedar dynamic framework into an OS X command line executable build target, write your specs, and run. Running from inside Xcode should give you output that looks something like this:
Xcode spec run pass

For specs that do depend on iOS frameworks, such as custom views or view controllers, you can link the Cedar static framework into an iOS executable, use the Cedar-specific application delegate (CedarAppDelegate), and you get something that looks like this:
Xcode UI spec run pass

The elements in the table represent the top-level specs that you have written. You can click on each one to drill into the state of each of its children, and its childrens’ children, etc.

Of course, those nice green bars aren’t terribly useful for the CI build, so if you’d prefer to skip the UI you can set the CEDAR_HEADLESS_SPECS environment variable, and then running the specs on the simulator (or device) reverts back to the dots:
Xcode headless UI spec run pass

Notice that we’ve created rake tasks for running the specs on the command line, for the sake of simplicity and ease of integration with CCRB. The Rakefile is available in the Cedar project, and does little more than set environment variables and execute bash shell commands. If you’d prefer to use your own bash script, or something similar, feel free to copy and paste your way to happiness.

I’ll spare you another screenshot, but you can see the results and build logs for the Cedar CI build here, and its current status is always displayed on our public CI Monitor page.

The last piece of the testing puzzle for iOS is full-stack integration and lifecycle testing (à la Selenium), which should be possible with Apple’s new UI Automation tool. Unfortunately, Apple hasn’t yet provided a way to run UI Automation tests, or get test results, from the command line. Hopefully we’ll find a way soon; their JavaScript DSL for iOS testing combined with Jasmine could be a potent combination.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter

Objective-C exceptions thrown inside methods invoked via NSInvocation are uncatchable

Adam Milligan
Thursday, July 1, 2010

Whether you’re using Cedar or not, if you’ve upgraded to the iOS 4.0 SDK you may have run into some odd behavior with exception handling blocks not catching exceptions. Strangely, the problem isn’t due to the exceptions themselves (at least not in any obvious way), but with how you call functions that raise exceptions. An exception thrown from within a method you invoke directly will function as expected. However, if you invoke that same function indirectly using NSInvocation any exception thrown becomes uncatchable, crashing the current process regardless of any exception handling code.

This happens only when running against the currently available iOS 4.0 SDK. Exception handling for both direct and indirect invocations performs as expected when using the OS X 10.6 SDK and previous versions of the iPhone SDK.

You can reproduce this problem yourself easily enough. Just create any old iPhone project, and add the following code in a method you know will run (e.g. viewDidLoad on the main controller):

@try {
    NSException *exception = [NSException exceptionWithName:@"foo" reason:@"bar" userInfo:nil];
    [exception raise];
} @catch (NSException *x) {
    NSLog(@"========================> %@", x);
}

Works, right? Now try this (which should be functionally identical):

@try {
    NSException *exception = [NSException exceptionWithName:@"foo" reason:@"bar" userInfo:nil];

    SEL selector = @selector(raise);
    NSMethodSignature *signature = [exception methodSignatureForSelector:selector];
    NSInvocation *invocation = [NSInvocation invocationWithMethodSignature:signature];

    [invocation setTarget:exception];
    [invocation setSelector:selector];
    [invocation invoke];
} @catch (NSException *x) {
    NSLog(@"========================> %@", x);
}

BOOM! Stack trace. Sad face.

Unfortunately, the OCHamcrest matcher library uses indirect method invocations to raise matcher failure exceptions. So, with iOS 4.0 rather than getting failure messages in your test output, you now get a stack trace and potentially a bunch of un-run tests.

Hopefully Apple will fix this soon.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Adam Milligan

Adam Milligan
New York

Subscribe to Adam's Feed

Author Topics

blocks (2)
cplusplus (4)
ios (10)
objective-c (13)
cedar (11)
testing (16)
opensource (5)
xcode4 (1)
rake (4)
access control (3)
ci (2)
jasmine (1)
javascript (3)
ie6 (1)
addiction (1)
activerecord (7)
nested_attributes (1)
rails (12)
actionpack (1)
refactoring (1)
agile (4)
ruby (8)
actionview (1)
threads (1)
functors (2)
brownbags (2)
solr (1)
  • About
  • Case Studies
  • Team
  • Community
  • Careers
  • Tools
  • Contact
  • Labs
  • Events

Contact Us

contact@pivotallabs.com
+1 415-77-PIVOT
TwitterLinkedInFacebook

Pivotal Tracker

Tracker is the award-winning agile project management tool that enables real-time collaboration around a shared, prioritized backlog.
Visit pivotaltracker.com >