Pivotal Labs

Main menu

Skip to primary content
Skip to secondary content
  • About
  • Case Studies
  • Team
    • Executives
    • Locations
      • San Francisco (HQ)
      • Boston
      • Boulder
      • Denver
      • London
      • Los Angeles
      • New York
  • Community
    • Blogs
    • Tech Talks
    • Events
  • Careers
    • Lifestyle
    • Principles & Practices
    • Benefits
    • FAQ
    • Apply
  • Contact
    • Press Room
    • Press Releases
    • In The News
    • Press Kit
  • All
  • Labs
  • Standup
  • Tracker
Sam Coward

Cedar target templates for Xcode 4.3.x

Sam Coward
Thursday, July 12, 2012

Announcing Cedar target templates for Xcode 4.3.x: Getting started with Cedar has never been easier!

These templates let you quickly add Cedar targets to your iOS or Mac OS X project. It’s super simple to build and install from source now:

git clone http://github.com/pivotal/cedar --recursive && cd cedar ; rake install

If you don’t like building from source, you can also install a build of the Cedar templates from the downloads page – just untar to your home directory.

Using Cedar Templates

Restart Xcode after installing, open your project and add a new target. You’ll find targets for both iOS and Mac OS X. There are 2 types of targets: Suites and Bundles. A Suite target will build your tests as a separate application which you then run to execute your tests. Bundle targets work just like OCUnit testing bundles, providing closer integration with Xcode. If you do create a bundle target, make sure to edit the scheme of your application and add the new target to the list of tests.

Every template comes with a simple example which should run out of the box to help you get running.

ARC: A word of caution

Automatic Reference Counting setting: Don’t use it for your spec targets. Your application is free to use ARC, just don’t use it for any Cedar targets. This is because there is a flaw in the compiler that ships with Xcode (see here for the details, and if you care please feel free to use this information to file a bug with apple).

Have fun speccing

We really hope these changes help you to get started writing specs and spending less time setting up Cedar. If you have any feedback, whether it’s about these templates or something else about Cedar, please feel welcome stop by the discussion group and share your thoughts.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Dmitriy Kalinin

Cedar and OCUnit

Dmitriy Kalinin
Wednesday, November 16, 2011

tl;dr Latest version of Cedar can nicely integrate with Xcode just like OCUnit.

Why

When creating new Xcode project you might have noticed that New Project dialog has ‘Include Unit Tests’ checkbox at the bottom. When that option is selected in addition to creating an app target Xcode will also create a separate target that is used for testing your application. This target most likely will be named [app]Tests. Convenient, huh! Since Xcode is such a nice IDE[1] it will also create an example test file that uses Xcode’s built-in unit testing framework — OCUnit (also known as SenTestingKit). If you never heard of it below is a screenshot of how that example OCUnit test file looks.


Because Apple ships OCUnit with Xcode it is tightly integrated into the IDE and that gives it somewhat of an advantage over other testing frameworks. Example of such unfair treatment is relatively nice looking red error markers (shown above) that show up next to failed test cases when application tests are run. If you have ever used Cedar before you would know that when Cedar specs fail we do not get those markers but rather we are presented with boring gray console output (we have dots though!). Unfortunately there is no documented way for custom testing frameworks to mimic OCUnit behavior without building on top of it. But even if do decide to use OCUnit as a base for your custom testing framework you will run into several problems. One of those problems, at least a year ago, was that you could not reliably use debugger with OCUnit tests. That was one of the main reasons why Cedar was not build on top of it. Recently, however, Xcode 4 was released with a lot of enhancements including better support for running OCUnit tests (with or without debugger) or so it seems. Red markers being a driving force, once again it was worth to investigate on how take advantage of OCUnit goodies from Cedar. Latest version of Cedar succeeds at that. What follows are the steps on how to setup Cedar for maximum Xcode integration.

Setup steps


Check ‘Include Unit Tests’ checkbox in New Project dialog. If you want to add application tests to an existing project here is a very nice tutorial: http://twobitlabs.com/2011/06/adding-ocunit-to-an-existing-ios-project-with-xcode-4/


After Xcode is done creating a new project file go to Build Settings for application tests target (in this case ExampleTests) and find Other Linker Flags settings. Add -ObjC, -all_load, and -lstdc++ to that setting and click Done.


Next step is to add Cedar static library to application tests target.


Now you are ready to create a Cedar spec file. Create a new empty file and end the filename with ‘Spec.mm’. ‘mm’ file extension is needed to be able to use built-in Cedar matcher library. Also since this file should only be run as part of application tests and not the actual application make sure that you only include it in application tests target.


Add a test or two (see above for example) to be able to test that Cedar is working correctly. Example above tries to assert that 2+2 equals 5 and that should hopefully fail. Notice that here I’m using recently added Cedar matchers should syntax. If you do not like that you can always rewrite that expectation as expect(2+2).to(equal(5));


Larger image.
And finally to run specs you just wrote select scheme for your app (in this case Example) from the drop down in the upper left corner and press ‘Cmd+U’. ‘Cmd+U’ in Xcode by default runs tests associated with selected scheme. That should first build your app (because application tests target lists the app as a target dependency), then build application tests target and then run the tests. Since 2+2 does not equal 5 you will see that Xcode will highlight that line in red and show that as an error in the errors panel on the left side. Fixing that expectation should make the test pass.

Running from the command line

If you want to run your application tests from the command line you can use rake ocunit:application to do that. Before doing so make sure you have updated APP_NAME and OCUNIT_APPLICATION_SPECS_TARGET_NAME in the Rakefile. Also you’ll need to use Debug for CONFIGURATION so that your tests are able to include classes from your application. What’s nice about this rake task is that it is ready to be used on your CI machine since it returns exit code 1 when any tests fail and 0 if they all succeed. It also prints in color.

How it actually works

If you are wondering how Cedar pretends to be OCUnit here is a short summary:

  • When Cedar is loaded it overrides runTests class method on SenTestProbe class (related file to look at is CDROTestIPhoneRunner.m)
  • SenTestProbe +runTests gets called by otest — an executable that Xcode uses to run tests after it injected the application tests bundle into TEST_HOST app
  • Since SenTestProbe +runTests was overridden with Cedar’s run method Cedar executes all of its specs following all the usual rules: focused, pended, etc.
  • When specs are running CDROTestReporter (source) gets notified which tests passed, failed, got skipped, etc. and prints out appropriate information which you see in console log
  • CDROTestReporter prints out failed tests in a way that is recognized by Xcode (OCUnit spits out same output for failed examples) e.g.:
Test Case '-[ExampleTests testExample]' started.
/Users/work/workspace/Example/ExampleTests/ExampleTests.m:16: error: -[ExampleTests testExample] : '4' should be equal to '5': 2+2 != 5
Test Case '-[ExampleTests testExample]' failed (0.000 seconds).
  • Xcode notices that pattern in the console output, parses it, gets filename and line number and marks that file location with a red marker to indicate a test failure

Drawbacks/Not implemented features

  • Debugger support is still a bit flaky but this is a problem with Xcode/OCUnit integration
  • I have not tried running it on the device but it should work
  • If you are used to specifying -SenTest option to run specific test case it will be ignored; however, you can use Cedar’s fit, fdescribe, and fcontext to focus on specific tests
  • Some third party OCUnit add-ons might not work (let us know about them)

Useful links

  • Cedar source code
  • Apple Unit Testing Overview
  • Official but outdated OCUnit and otest source code
  • More recent version of OCUnit source code
  • Public Pivotal Tracker project for Cedar to submit bugs and feature requests

[1] It’s not.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Dmitriy Kalinin

Too many parentheses

Dmitriy Kalinin
Wednesday, November 16, 2011

Cedar comes with its own built-in matcher library and here is Adam Milligan’s blog post that explains on how to use it. An example of expectation written using Cedar matcher library:

expect(controller.button).to_not(be_nil());

Even though this is much better than OCHamcrest it is still not satisfying (in my opinion) because there are so many parentheses involved. Since Cedar matcher library is written in C++ we can do some magic here. The result is you can write your expectations like this:

controller.button should_not be_nil;

So let’s dig into how Cedar’s should syntax is implemented.

  controller.button   should_not   be_nil
|-------------------|------------|--------|
    actual value       C macro     matcher

As you might have guessed should and should_not are C macros. Besides making possible for each expectation to know its filename and line number (with __FILE__ and __LINE__ preprocessor variables), they also hide the unnecessary complexity which makes this syntax possible. Here is how it looks expanded:

  controller.button   ,(ActualValueMarker){__FILE__, __LINE__},true,   be_nil
|-------------------|------------------------------------------------|--------|
    actual value                        mumbo jumbo                    matcher

So how does this mumbo jumbo work? Since Cedar matcher library was written in C++ we can overload the comma operator. Besides being a weird operator to overload, it has a useful property that we can exploit here. Comma operator has the lowest precedence among all other operators and that allows us to write 2+2 should equal(4) instead of (2+2) should equal(4). Now let’s break mumbo jumbo into pieces:

(1) <actual value>
    (ActualValueMarker){__FILE__, __LINE__} -> (2) ActualValue
                                                   true/false  -> (3) ActualValueProxy
                                                                      <matcher>

(1) This step captures actual value (can be anything since it is templated), filename and line number. It is implemented like this:

template<typename T>
const ActualValue<T> operator,(const T & actualValue, const ActualValueMarker & marker) {
    return ActualValue<T>(marker.fileName, marker.lineNumber, actualValue);
}

(2) Now we overload comma operator to match ActualValue (returned from step 1) and a bool that indicates whether matcher result should be negated.

template<typename T>
const ActualValueMatchProxy<T> operator,(const ActualValue<T> & actualValue, bool negate) {
    return negate ? actualValue.to_not : actualValue.to;
}

(3) And finally we overload comma operator again so that it matches ActualValueMatchProxy (returned from step 2) and anything else. Second argument will be considered to be a matcher.

template<typename T, typename MatcherType>
void operator,(const ActualValueMatchProxy<T> & matchProxy, const MatcherType & matcher) {
    matchProxy(matcher);
}

That’s pretty much all. Full implementation of should syntax (which is just several more lines more than what is shown above) can be found in ShouldSyntax.h (and corresponding spec file that makes sure syntax works). Everything above allows us to write expectations as follows:

controller.button should_not be_nil();

Since the goal is to get rid of as many parentheses as possible we still have some work to do. Nil matcher that comes with Cedar is implemented with BeNil class. The way it worked before was be_nil was a function defined in Cedar::Matchers namespace. Calling it each time would return a new instance of BeNil class. To make parentheses go away be_nil was turned into a static variable that is initialized only once to an instance of BeNil class. (be_nil matcher still supports being used with two parentheses for backwards compatibility by implementing call operator on BeNil class.) Bam.

controller.button should_not be_nil;

Usually it is not recommended to use operator overloading for non-operator related purposes (e.g. don’t use + operator for subtraction); however, in this case I think it’s fine to bend the rules just a bit to escape all those parentheses and provide clean expectation DSL. You can use should syntax with the latest Cedar master branch.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Adam Milligan

Cedar Expectations

Adam Milligan
Wednesday, November 2, 2011

As I wrote here we’ve used OCHamcrest matchers for some time for writing expectations in Cedar, but have found them unsatisfying. We wanted convenient matchers, like the ones Jasmine provides for JavaScript, but for Objective C. To that end, we added Cedar-specific expectation functions and matchers that specifically solve the problems we had with OCHamcrest.

To use Cedar’s expectations you need to make a couple small changes to your spec files:

1) Cedar matchers use C++ templates. Tell the compiler to expect some C++ code by changing the file extension on your spec files from .m to .mm.

2) Cedar matchers live in a C++ namespace. At the top of your spec file, after the includes, add this line:

using namespace Cedar::Matchers;

Features

Type Deduction

Cedar matchers determine the types of the values in the expectation, so you don’t have to. Rather than this:

assertThat(someObject, equalTo(anotherObject));
assertThatUnsignedInt(someUInt, equalToUnsignedInt(anotherUInt));

you write this:

expect(someObject).to(equal(anotherObject));
expect(someUInt).to(equal(anotherUInt));

We can also mix and match object and non-object types, in cases where such comparisons make sense. For instance, rather than this:

NSNumber *aNumber = [NSNumber numberWithFloat:1.7];
assertThat(aNumber, equalTo([NSNumber numberWithFloat:1.8));
assertThatFloat([aNumber floatValue], equalToFloat(1.8));  // equivalent

you write this:

NSNumber *aNumber = [NSNumber numberWithFloat:1.7];
expect(aNumber).to(equal(1.8));

Extensibilty

Cedar’s expectations use C++ templates not just for argument types, but for matcher objects as well. This means what goes inside the to() construct (e.g. expect(foo).to(equal(bar));) can be any object that responds to the appropriate methods (more on this later). If you want to write a custom matcher for your system (e.g. expect(user).to(be_logged_in());), no need to subclass anything or link against the Cedar library.

We also separated comparators from matchers for common expectations, such as equality. This means that adding equality comparison for a custom type involves only adding a comparator function for your type, with no need to modify the equality matcher itself. For example, here is the method for comparing two NSNumber objects:

bool compare_equal(NSNumber * const actualValue, NSNumber * const expectedValue) {
    return [actualValue isEqualToNumber:expectedValue];
}

Caveats

Several versions of LLVM have had trouble with properly compiling Objective C++ code that contains blocks, and GCC has never done so correctly. Versions of LLVM before 2.0 (Xcode 3.x) will fail to compile Cedar specs with the Internal Compiler Error” message. Version 2.0 of LLVM (Xcode 4.0.x) will successfully compile Cedar specs, but incorrectly handle reference to temporaries, leading to difficult to understand error messages. Versions 2.1 (Xcode 4.1) and 3.0 (Xcode 4.2) fix this problem, and will work properly. However, most version of Xcode still default to using the GCC/LLVM hybrid, so you may need to change this setting.

Also, C++ templates deduce types at compile time, which means they use the matcher function for the declared type of parameters. For instance, if you have an equality comparator specialized for MyType, which is a subclass of NSObject, the following code will invoke the equality comparator for NSObject, not MyType:

NSObject *foo = [[MyType alloc] init]; // refers to a MyType, but is declared as NSObject *
expect(foo).to(equal(someOtherObject));

In practice this seems to cause problems less frequently than I had expected. However, keep in mind that the declared type of nearly all Objective C initializers is id (NSNumber seems to be the exception to this rule, for some reason). So, if you pass the temporary result of an initializer to a matcher, it will most likely use the specialization of that matcher for id. For instance:

expect([NSString string]).to(equal(@""));  // Compares id to NSString * (works fine)
expect([[MyType alloc] init]).to(equal(@"")); // Compares id to NSString * (might work?)

Finally, because of the way C++ template work, the order of declaration of functions does matter. For instance, the equality matcher delegates comparison to the appropriate overload of the compare_equal function; this allows easy addition of custom comparators, as mentioned above. However, at the point where the compiler finds the equality matcher, it will only match the parameter types to comparator function overloads it has already seen. This mean you need to include your custom comparators before the declaration of the equality matcher. Since Cedar helpfully includes all matchers and comparators, you can do this in one of a few ways:

  1. Import your custom comparators before Cedar’s SpecHelper.h at the top of your file. This should work fine, since your custom matchers and comparators need not included dependencies from Cedar.

  2. Cedar checks a preprocessor value named CEDAR_CUSTOM_COMPARATORS. If you set this value Cedar will treat the value as a file path and #import it before the matcher library. Cedar also have CEDAR_CUSTOM_MATCHERS and CEDAR_CUSTOM_STRINGIFIERS (for providing custom output formats for your types).

  3. For new matchers and comparators for Cocoa types (CGRect, UIView, NSIndexPath, AVPlayer, etc.), feel free to contribute back to Cedar so they get included by default.

Please send any comments, questions, suggestions, feature request, pull requests, etc. to the Cedar discussion mailing list. Unfortunately, messages to the Pivotal account on GitHub get lost in the jillions of messages and notifications we receive.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Adam Milligan

The Trouble With Expectations in Objective C

Adam Milligan
Wednesday, November 2, 2011

At Pivotal we write a lot of tests, or specs if you prefer. We TDD nearly everything we write, in every language we write in and on every platform we write for, so we actively work to improve every aspect of our testing tools. Personally, as I’ve written tests in Objective C I’ve found that the syntax of expectations has left much to be desired.

Like many people who write Objective C, I’ve spent a fair bit of time with languages like Ruby and JavaScript. When I write specs I often yearn for the simplicity of the expectation syntax in those languages. For example, some simple expectations in JavaScript, using Jasmine:

expect(composer.name).toEqual("Ludwig");
expect(composer.symphonies.count).toEqual(9);
expect(composer.symphonies).toContain("Eroica");
expect(composer.symphonies).not.toContain("Appassionata");

In comparison, the same expectations in Objective C, using OCHamcrest:

assertThat(composer.name, equalTo(@"Ludwig"));
assertThatUnsignedInt([composer.symphonies count], equalToUnsignedInt(9u));
assertThat(composer.symphonies, hasItem(@"Eroica"));
assertThat(composer.symphonies, isNot(hasItem(@"Appassionata"));

The challenge

I want three primary attributes from matchers: readability, ease of use, and extensibility.

Readability

With the exception of the unfortunate mismatch between the negation function (isNot) and the containment matcher (hasItem), I find the OCHamcrest expectations reasonably readable. The distinction between the first equality matcher (assertThat/equalTo) and the second (assertThatUnsignedInt/equalToUnsignedInt) is a bit jarring, though.

Ease of use

That distinction is much more of a problem for usability. The problem comes from the Objective C type system’s split personality: some variables refer to Objective C objects (NSNumber *, NSString *, NSArray *, UIView *), while the others refer to built-in types and structs (int, char *, CFArray, CGRect), and never the twain shall meet. A simple equality match must use the isEqual: method for the former, but the == operator, or something more esoteric like strncmp, for the latter. Sadly, a single matcher function has no way to determine the correct equality mechanism to use. Worse, the matcher function declarations must specify the types of their parameters, so expectations for any non-pointer types must have separate declarations for each type. The number of functions for any general purpose expectation explodes:

  • equalTo
  • equalToInt
  • equalToUnsignedInt
  • equalToFloat
  • equalToDouble
  • etc.

I can’t describe how many times I’ve failed to have the type of the assertThatX function for an expectation match the equalToX function. Frustration ensues.

Extensibility

Adding new OCHamcrest matchers isn’t terribly difficult, but you have to derive your matcher object from the HCBaseMatcher class, which means any library that includes custom matchers must link against the OCHamcrest library. If your spec target links against OCHamcrest and your custom matcher library, and your custom matcher library also links against OCHamcrest, suddenly you’re in dependency management hell.

What to do about it?

We’ve used OCHamcrest on a number of projects now, but the usability issues have become an increasing burden.

Peter Kim wrote a nice matcher library named Expecta which solves the type differentiation problem using the Objective C @encode function. We’ve used this on a couple projects, and like it more than OCHamcrest. The above example would look like this in Expecta:

expect(composer.name).toEqual(@"Ludwig");
expect([composer.symphonies count]).toEqual(9);
expect(composer.symphonies).toContain(@"Eroica");
expect(composer.symphonies).Not.toContain(@"Appassionata");

At the same time, we added a set of built-in matchers to Cedar, which solve the type differentiation problem with C++ templates, and which we designed to be extremely extensible. We’re using these matchers on some of our projects as well, and we’ve found they work nicely. Here’s the above example using Cedar matchers:

expect(composer.name).to(equal(@"Ludwig"));
expect([composer.symphonies count]).to(equal(9));
expect(composer.symphonies).to(contain(@"Eroica"));
expect(composer.symphonies).to_not(contain(@"Appassionata"));

You can read more about the Cedar matches, including how to use and extend them, here.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Andrew Kitchen

Meanwhile, over in Objective-C land

Andrew Kitchen
Friday, July 1, 2011

Two new interesting open-source Objective-C frameworks bookended our week this week:

First, Peter Kim sent a message to the Cedar-discuss mailing list to introduce us to his new matcher framework Expecta. Now, instead of OCHamcrest’s assertThatBool(maybe, equalToBool(NO)) shenanigans one can simply expect(2+2).toEqual(4); There’s also a nice DSL for implementing custom matchers for objects.

Excited about the possibilities, Kurtis and I spiked on implementing a set of custom matchers for commonly-used Cocoa structs such as CGPoint, something we have found lacking in current tools and testing patterns. Our entire project team looks forward to writing more tests using Expecta.

Also, Twitter Engineering just announced TwUI, a new Core Animation-based UI framework for OS X. Fun times.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Adam Milligan

Cedar vs. Xcode 4 (round one: the command line)

Adam Milligan
Tuesday, April 19, 2011

I’ve finally found a bit of time to update Cedar to work with Xcode 4, and I hope to have it working smoothly some time in the next few days. However, I’ve already come across my first significant issue with the Xcode 4 changes: the location of build products.

Not unexpectedly, the problem has to do with command line builds using xcodebuild. By default, Xcode 4 now puts build products into a project-specific directory in the “Derived Data” folder; this looks something like /Users/pivotal/Library/Developer/Xcode/DerivedData/Cedar-somegianthashstring/Build/Products/Debug-iphonesimulator/libCedar-StaticLib.a. This isn’t a problem, generally, because the BUILD_DIR compiler variable contains the build directory, should you need to find this location during the build process.

Sadly, when you build from the command line, using the xcodebuild command, the build products still go into the old Xcode 3 build location, but the BUILD_DIR compiler variable contains the new Xcode 4 build directory. This means any script that looks for the build results in the directory specified by BUILD_DIR won’t find anything.

The build target for Cedar’s static framework is simply a script that uses xcodebuild to build the static library for both the simulator and the device, and then uses lipo to make a fat binary from the results. Because it can’t find the build results at the location specified by BUILD_DIR it now fails messily.

The easiest workaround I’ve found is to change where build products go using the Locations setting in the Xcode 4 preferences (details below). Unfortunately, this isn’t a project-specific setting, so you’ll have to change your preferences similarly to make it work. I haven’t found any problems with changing the location of the build products, but this does mean the Cedar static framework (as well as the related static frameworks for OCHamcrest and OCMock) won’t build with the default settings. Unsatisfying.

The longer term solution is for Apple to act on the bug I filed. We’ll see how that goes.

UPDATE: Thanks to Christian Niles for pointing out the SYMROOT environment variable in a pull request. Setting this for command line builds forces Xcode to use the specified location for all build products, and updates the BUILD_DIR compiler variable.

Steps for changing the build location in Xcode 4:

  • Open Xcode preferences (Command-,)
  • Select the “Locations” tab
  • Change the “Build Location” drop down from “Place build products in derived data location” to “Place build products in locations specified by targets.”
  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Adam Milligan

Colorized output for Cedar

Adam Milligan
Thursday, December 23, 2010

Thanks to Sean Moon and Sam Coward Cedar now has colorized output on the command line:

Colorized Cedar report

If you’d like to display colorized output like this you can specify the appropriate custom reporter class using the CEDAR_REPORTER_CLASS environment variable. We do this in our Rakefiles, like so:

task :specs => :build_specs do
  ENV["DYLD_FRAMEWORK_PATH"] = BUILD_DIR
  ENV["CEDAR_REPORTER_CLASS"] = "CDRColorizedReporter"
  system_or_exit(File.join(BUILD_DIR, SPECS_TARGET_NAME))
end

You can set the environment variable in whatever way works for you. You can also set it to any reporter class you choose, so customize away.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Adam Milligan

iPhone UI Automation tests with Jasmine

Adam Milligan
Sunday, July 18, 2010

Since the language of the new iPhone UI Automation component is JavaScript I figured the easiest way to organize tests is to use a JavaScript testing framework, such as Jasmine. So, I created jasmine-iphone, which is little more than a few simple scripts to make UI Automation and Jasmine play nice.

Once you clone jasmine-iphone from GitHub (it includes Jasmine as a submodule, so be sure to git submodule init && git submodule update) you can copy the example-suite.js file, import your spec files, point Instruments at your suite.js and go.

As an example, I set up a trivial example in the Cedar project. The directory structure looks like this:

Project Directory
- Spec
    - UIAutomation
        - jasmine-iphone     <--- submodule
            - jasmine        <--- nested submodule
        - suite.js
        - thing-spec.js
        - other-thing-spec.js

The suite.js file is relatively simple (note that I moved it up one directory from where the example-suite.js file is, so the #import statements are slightly different):

#import "jasmine-iphone/jasmine-uiautomation.js"
#import "jasmine-iphone/jasmine-uiautomation-reporter.js"

// Import your JS spec files here.
#import "thing-spec.js"
#import "other-thing-spec.js"

jasmine.getEnv().addReporter(new jasmine.UIAutomation.Reporter());
jasmine.getEnv().execute();

You can write the specs themselves the same way you’d write Jasmine specs for anything else. The UIAutomation subclass of the Jasmine Reporter takes care of marking the start of each spec, as well as reporting if it passes or fails. You’ll need to use the UIAutomation classes and methods for driving your application’s state, of course.

That’s it. Try it out and see what you think.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Adam Milligan

Cedar device specs and CI

Adam Milligan
Tuesday, July 6, 2010

One of the most common complaints I’ve read about OCUnit, the unit testing framework built into Xcode, is that the tests you write with it won’t run on the device. In addition, I personally have found the process of setting up a target for tests that depend on UIKit confusing and onerous. So, one of our goals for Cedar was to make testing UI elements easy (or easier), by making it easy to run specs in the simulator or on the device.

Probably the second most common complaint I’ve read about OCUnit is that the tests run as part of the build. This makes the test output difficult to separate from the build output, and makes it impossible to use the debugger when running tests. So, in addition to making it easy to run specs on the device, we wanted to be able to run them as a separate, debuggable executable.

Finally, we consider it important that our specs run in our CI system. That means we wanted to be able to run Cedar specs from the command line, and get an exit code signifying success or failure. At the same time, some of us appreciate the value of the green/red feedback for specs passing and failing, so sometimes we like a nice UI. As of today, Cedar will accommodate all of these various requirements.

If you have simple tests that don’t depend on iOS, you can simply link the Cedar dynamic framework into an OS X command line executable build target, write your specs, and run. Running from inside Xcode should give you output that looks something like this:
Xcode spec run pass

For specs that do depend on iOS frameworks, such as custom views or view controllers, you can link the Cedar static framework into an iOS executable, use the Cedar-specific application delegate (CedarAppDelegate), and you get something that looks like this:
Xcode UI spec run pass

The elements in the table represent the top-level specs that you have written. You can click on each one to drill into the state of each of its children, and its childrens’ children, etc.

Of course, those nice green bars aren’t terribly useful for the CI build, so if you’d prefer to skip the UI you can set the CEDAR_HEADLESS_SPECS environment variable, and then running the specs on the simulator (or device) reverts back to the dots:
Xcode headless UI spec run pass

Notice that we’ve created rake tasks for running the specs on the command line, for the sake of simplicity and ease of integration with CCRB. The Rakefile is available in the Cedar project, and does little more than set environment variables and execute bash shell commands. If you’d prefer to use your own bash script, or something similar, feel free to copy and paste your way to happiness.

I’ll spare you another screenshot, but you can see the results and build logs for the Cedar CI build here, and its current status is always displayed on our public CI Monitor page.

The last piece of the testing puzzle for iOS is full-stack integration and lifecycle testing (à la Selenium), which should be possible with Apple’s new UI Automation tool. Unfortunately, Apple hasn’t yet provided a way to run UI Automation tests, or get test results, from the command line. Hopefully we’ll find a way soon; their JavaScript DSL for iOS testing combined with Jasmine could be a potent combination.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter

Topics

  • agile (780)
  • rails (113)
  • testing (88)
  • ruby (83)
  • ruby on rails (70)
  • jobs (62)
  • javascript (55)
  • techtalk (44)
  • rspec (38)
  • ironblogger (32)
  • productivity (30)
  • activerecord (29)
  • gogaruco (29)
  • git (28)
  • nyc (27)
  • rubymine (26)
  • bloggerdome (23)
  • mobile (22)
  • process (21)
  • pivotal tracker (20)
  • cucumber (20)
  • jasmine (19)
  • design (18)
  • ios (18)
  • webos (17)
  • objective-c (17)
  • android (16)
  • palm (16)
  • "soft" ware (16)
  • fun (15)
  • tracker ecosystem (15)
  • ci (15)
  • cedar (15)
  • rails3 (14)
  • performance (14)
  • bdd (14)
  • gem (13)
  • css (13)
  • tdd (13)
  • selenium (12)
  • goruco (12)
  • bundler (12)
  • meetup (11)
  • railsconf (11)
  • nyc-standup (11)
  • capybara (10)
  • mac (10)
  • mojo (10)
  • chef (10)
  • api (10)
Subscribe to cedar Feed
  1. 1
  2. 2
  3. →
  • About
  • Case Studies
  • Team
  • Community
  • Careers
  • Contact
  • Labs
  • Events

Contact Us

contact@pivotallabs.com
+1 415-77-PIVOT
TwitterLinkedInFacebook

Pivotal Tracker

Tracker is the award-winning agile project management tool that enables real-time collaboration around a shared, prioritized backlog.
Visit pivotaltracker.com >