If you’re interested in following the growth of Cedar, or just testing iPhone projects in general, you can join the public Cedar discussion group or follow Cedar on Twitter at @cedarbdd.
I’ll continue to write out my thoughts on the topic here, but I’d prefer to have a method of sharing information that anyone can post to.
When I started writing Objective-C for iPhone and iPad projects I started to hear about these mystical and all-powerful run loops objects. Everything seems to relate to run loops, all the class references talk about how one object or another depends on run loops, the most basic iPhone application documentation even talks about how the device starts and maintains the main run loop for you because it’s so important. However, I found few straightforward descriptions of what a run loop is or how it works.
So, I read Apple’s Threading Programming Guide. I read about timers and ports and sources, about the starting and stopping and timing out of run loops. After this reading, and a bit of thought, I realized that all the sound and fury surrounding run loops blinded me to the fact that they are actually very, very simple. To wit, run loops are a mechanism for single-threaded asynchronous programming. Sound familiar? Sounds to me a lot like JavaScript.
A lot was made during the recent WWDC of the difference between synchronous and asynchronous network programming, so as an example let’s consider an AJAX request in JavaScript. To get some information from a server you’d send off a request in one function, passing in another function (or functions) for the system to call when it receives a response.
makeARequest({onSuccess: function(response) {
// do some success stuff.
}});
Your initial function exits, and while the request is making its merry way through the series of tubes between the client and your server the system is happily going about its business. But, it has its digital ear to the ground for your response, and when that comes back it calls your function. Voilà, asynchronicity.
That’s a run loop. It has event sources, like some form of network socket on which it listens for your server’s response; it has timers, in the form of setTimeout() and clearTimeout(). The Apple Core Foundation run loops take a fair bit more setup to use — the network request performed by those three lines of JS would involve NSURLConnection, NSURLRequest, some form of delegate object, etc. — but it’s exactly the same idea. It’s waiting for something to happen while you take a nap; when that something happens it tells you about it.
There is one extremely significant difference between the JavaScript implementation and the Core Foundation implementation, at least in my mind: testing. In JavaScript this type of asynchronous code isn’t a run loop, it’s just JavaScript. As such, when you run your tests your code does everything you’d expect it to do; it’s the same interpreter running the tests as the production code, after all. So, when your test setup reaches into your AJAX library and stubs out the part that actually talks to the network everything continues to hum along. If you want, you step behind the curtain and send fake responses back through the AJAX object.
In comparison, a Core Foundation run loop is a special construct that the system sets up for you. When you run an app on your iPhone the device takes care of all the setup, care, and feeding of the main run loop. So, in tests, you if you want to run code that depends on a run loop you have to start the run loop yourself. Worse, starting a run loop blocks the execution of your tests until you stop the run loop. So, if you stub out the network code that actually talks to the network your code may sit and wait forever for a response that can’t possibly come. If you set up your code to return a fake response before starting the run loop your code will receive that response, but the run loop will continue to run, potentially blocking your test code forever.
This is, in my opinion, a significant problem for testing Cocoa projects. Asynchronous programming using run loops is simpler than mucking about with threads, and largely endorsed by Apple via their frameworks. But, there’s no reasonably simple way to test code written this way. You could set up a test run loop with test-specific run loop sources or timeouts to prevent it from blocking forever, but that seems immensely complex when compared to the JavaScript implementation. I would very much like to see some support from Apple in the form of a stub run loop for test environments that provides mechanisms for passing fake data to run loop sources, freezing and incrementing time to test timers and delayed invocations, etc.
I’m interested to hear other ideas you’ve had for how to get around this, or test mechanisms you’d like to see from Apple.
A little while ago I wrote about Cedar, a BDD-style testing framework for Objective-C. The responses I received nearly all went something along these lines: “That’s great! Too bad I can’t use it, since I’m writing an iPhone app.”
Hogwash.
I actually wrote Cedar specifically for testing iPhone OS projects we’re working on at Pivotal. To prove it, I’ve started a small public iPhone project that I’ve test-driven entirely with Cedar. You can get the project here (more on that in a bit); it should eventually allow you to log into Pivotal Tracker, see all the delivered stories in a given project, and accept or reject each one. At the moment it does little more than start up and display the Pivotal Chicken*, but it does contain Cedar specs that run on and off the device.
“How is this possible?” you ask. I’ve done two things to make this work:
I separated out all classes that don’t depend on UIKit into a target that builds a static library. The specs for this target run as a console app using the OS X runtime, so no need to worry about runtime support for blocks (assuming you’re running 10.6). Also no need to incur the overhead of starting the emulator every time you run tests. This is a pattern I started using ages ago to make automated testing easier on Win32 client applications, and it works great for all the mobile platforms I’ve worked on. Framework independence means faster tests, and faster tests mean happier programmers. I recommend doing this whether you’re interested in testing with Cedar or not.
Tests for the actual app, which does depend on UIKit and therefore must target the iPhone runtime, run on the emulator (or, in theory, a device) using the PLBlocks iPhone runtime for block support.
Getting the specs up and running takes a few steps, but as we all know good things come to those who wait.
Clone, or otherwise obtain the StoryAccepter project. It will probably have a number of missing library references.
You’ll need to build Cedar, both the dynamic framework (Cedar.framework) and the static iPhone library (libCedar-iPhone.a), as well as the OCHamcrest and OCMock frameworks. Fix the references in the StoryAccepter project to point to these libraries on your system.
If you’re running Leopard you’ll need to install PLBlocks 1.0 for Leopard, and you’ll need to include the runtime and set the compiler for both spec targets; the PLBlocks page has excellent instructions. If you’re running Snow Leopard the project should already contain the runtime, so you’ll just need to download and install the compiler for PLBlocks 1.0.1.
Select the DomainSpec target, and make sure you’ve selected the appropriate Mac OS X runtime for your system. Build and run; you should see dots appear in the console window as the specs run.
Select the StoryAccepterSpec target, and make sure you’ve selected an iPhone runtime (if you want to try running on a device you’ll have to set up the provisioning, of course). Build and run; the emulator should start up and try to run the app, which will simply run the specs and then exit. You should see dots in the console window, as before.
All of this still has some rough spots, especially the UIKit-dependent specs, but even so I’ve found test driving with hierarchical describe blocks far more pleasant than using OCUnit. Some things I hope to improve on:
I imagine the app that runs the UIKit-dependent specs showing a graphical display of test results, perhaps similar to what GHUnit displays when run on the emulator or device.
The iPhone doesn’t allow dynamic libraries, and I haven’t found a way to use OCHamcrest or OCMock for UIKit-dependent specs. The folks at Carbon Five describe using these libraries in their tests on the device; I’m curious to know how they pull that off.
Once iPhone SDK 4.0 comes out with support for blocks this should all work without the need for the PLBlocks runtime. That won’t help iPad development for the foreseeable future, though.
Full-stack integration testing à la Selenium.
Lots of other things. What would you like to see added to Cedar?
*Not an officially endorsed Pivotal mascot (yet).
Many people use the ultra popular Paperclip library to handle file attachments in Rails. Unfortunately the Paperclip documentation does not cover how to stub out calls to ImageMagick in your test suite. Without the proper stubs in place a test suite that uses Paperclip will take much, much longer to run.
In the grease your suite presentation by Nick Gauthier it has a slide titled Quickerclip that describes what needs to be done to spend up Paperclip in tests, basically you need to keep it from shelling out to ImageMagick. Alas, the presentation does include code for how to achieve Quickerclip.
As the presentation shows Paperclip.run is the method that needs to be changed. The first parameter passed to Paperclip.run is the ImageMagick command be executed. Paperclip uses the identify and convert commands. The identify command is used to determine the dimensions of an image. The convert command is the really heavy one that does image manipulation and thumbnail generation. Here is a redefinition of Paperclip.run with sensible behavior for tests.
module Paperclip
def self.run cmd, params = "", expected_outcodes = 0
case cmd
when "identify"
return "100x100"
when "convert"
return
else
super
end
end
end
class Paperclip::Attachment
def post_process
end
end
Redefining post_process in Paperclip::Attachment is an optional additional optimization. In Paperclip, post_process eventually calls Paperclip.run("convert") and by short-circuiting the method earlier in the chain we save a few cycles.
As we’ve grown our mobile practice at Pivotal we’ve tried to apply to it the same principles and disciplines that have made our Rails practice successful. Often the one that we have the most difficulty translating is testing. In my experience the testing tools for Objective-C in particular are significantly wanting; there are some out there, but they’re hard to find, often hard to use, and occasionally defective in frustrating ways.
One of the things I found I miss most in testing Objective-C, Java, or C++, is the hierarchical structure for organizing tests that frameworks like RSpec or Jasmine provide. I find nested describes indispensable for managing orthogonal aspects of the classes under test, for handling preconditions, for eliminating redundant setup code, and for generally keeping my sanity. So, when I first heard about the addition of blocks in the GCC compiler for Objective-C the first application that came to mind was testing.
So, I wrote Cedar, a BDD-style framework for writing tests in Objective-C. The code is available here. Perhaps more importantly, Cedar is in its infancy so I’m interested in any suggestions and feedback. To that end, I created a public Tracker project for it here.
A minimal spec in Cedar looks like this:
// FooSpec.m
#include "SpecHelper.h"
SPEC_BEGIN(FooSpec)
describe(@"An example", ^{
it(@"should be descriptive and helpful", ^{
...
});
});
SPEC_END
A few things to note:
// FooSpec.m
#include "SpecHelper.h"
@interface FooSpec : Spec
@end
@implementation FooSpec
- (void)declareBehaviors {
describe(@"An example", ^{
it(@"should be descriptive and helpful", ^{
...
});
});
}
fail() method. Rather than reinvent the wheel I decided to support using the matchers from the Hamcrest library, available here. Note that you can only get the Objective-C port of Hamcrest by checking out the code from Subversion and building it yourself. I considered committing a pre-built version of the Hamcrest framework into the Cedar repository, but I’m not sure what the accepted approach is for including dependencies like that in Objective-C projects. Feedback welcome.Don’t try to mix blocks with Objective-C++, at least not yet. I tried it for some time and ran into any number of internal compiler errors. Hopefully this will improve in the future. As some will astutely point out, I could have used the anonymous functions introduced by C++0x (and supported by GCC). Unfortunately (from Wikipedia):
If a closure object containing references to local variables is invoked after the innermost block scope of its creation, the behaviour is undefined.
There’s no need to provide a header file for your specs, since Cedar finds the specs by introspection.
As I’m sure will soon become entirely obvious this is very much a minimal viable product for Cedar. You can create and nest describe blocks, create examples and beforeEach blocks, and that’s about it. I’m curious to see if people will use something like this; if they do, I’m hoping for plenty of feedback. I’m attached to basically nothing about the framework at the moment (including the name), so please send me a note or join the Tracker project if there’s something you’d like to see added, removed, or changed.
“What’s the best way to test that an array contains another array?”
A custom matcher using array - other_array was suggested.
The team that was having problems with Prototype response codes found a partial solution.
They sometimes call abort() on the request which triggers the callbacks with a 0 response code. They found that if they set transport.onreadystatechange = function() {}; before calling abort() they could skip the callbacks, and manually call whatever clean-up they needed. That Prototype interprets 0 as a success still sounds strange.
Javascript tests are good, but manually-maintained HTML fixtures are painful. It’s time consuming to keep fixture markup in sync with the actual markup produced by your app. Despite best efforts, deviations arise, leading to bugs and false positives in tests.
For the past few months on Mavenlink, we’ve been pre-generating real-life fixture markup and making it available in our javascript tests. Results have been positive.
The basic approach is simple:
1) Pre-generate real-life markup using any convenient mechanism.
2) When javascript tests run, load the pre-baked markup into the DOM for access by JS code.
For #1, we use a small set of RSpec controller specs that exist solely to generate fixtures. For #2, we retrieve fixture markup from our Jasmine server via ajax, and inject it into the DOM with jQuery. While our approach leans on the various bits of our stack, any part of it could be swapped out to adapt to different tools.
Big thanks to Jonathan Barnes for the code that got us started.
Taking a closer look, here’s an RSpec test that generates an HTML fixture file named ‘workspace-page’:
describe WorkspacesController do
it "generates a workspace page" do
@workspace = create_workspace
log_in @workspace.creator
get :show, :id => @workspace.to_param
response.should be_success
save_fixture(html_for('body'), 'workspace-page')
end
end
We create a workspace, hit the WorkspaceController‘s show method, and save the full text of the response’s <body> element to a file. This spec lives in a file with others like it, separate from the real controller specs. We have about a dozen of these specs, all in the same file.
We added save_fixture() and html_for() to ControllerExampleGroup to help with fixture generation. This gist has implementation details.
Our javascript tests sometimes require that we load different versions of the same page. We generate a different fixture for each version, giving them meaningful names like ‘busy-workspace-page’ and ‘empty-workspace-page’.
When we change markup consumed by our javascript tests, we re-run the controller specs that generate fixtures. Changes are picked up the next time we run our javascript tests. Fixture generation is hooked into continuous integration, so our javascript tests see the latest markup when running in CI.
On the javascript side, fixture loading is handled by the code in this gist. Of note:
We use two methods for making fixture content available to our tests. loadFixture() inserts the fixture markup into the DOM for use by code that accesses the DOM. readFixture() returns fixture content as text; we use it to test our ajax callback methods.
Within the same test run, we cache fixture text in javascript to avoid multiple requests to the server for the same markup. Across test runs, we ensure our markup is fresh by appending a cache busting timestamp to our request path.
Here’s how we typically use loadFixture() within our Jasmine tests:
describe('the status module', function() {
it('switches tabs', function() {
spec.loadFixture('workspace-page');
var $tabContainer = $('#jasmine_content').find('.tab-container');
expect($tabContainer).toHaveSelectedTab('team');
$tabContainer.find('li[tab=schedule]').click();
expect($tabContainer).toHaveSelectedTab('schedule');
});
});
loadFixture() inserts markup into the #jasmine_content div. Then we examine the DOM, do stuff to it, and inspect it again. toHaveSelectedTab() is a custom Jasmine matcher. Jasmine matchers are super easy to write. We love them.
You may be wondering how we’ve established our event bindings. jQuery’s html() method, used within loadFixture(), executes any scripts in the markup passed to it. If you’ve bound events in your fixture markup, within a $(document).ready() or not, they will execute when you call loadFixture(). This is really nice, because it means the same mechanism used for binding events in real life is also used within tests, keeping our tests that much closer to reality.
If, on the other hand, you bind events not within fixture markup, but instead within a script loaded once per suite globally, you’ll have to invoke your event binding code explicitly before each test.
Speaking of event bindings, you’ll need to clean them up properly between tests. For example, jQuery live events are bound to document. We clear them in a global beforeEach():
beforeEach(function() {
$('#jasmine_content').empty();
spec.clearLiveEventBindings();
jasmine.Clock.useMock();
});
spec.clearLiveEventBindings = function() {
var events = jQuery.data(document, "events");
for (prop in events) {
delete events[prop];
}
};
Any events bound on elements within #jasmine_content are cleared out by jQuery when we call $('#jasmine_content').empty(), which also wipes the DOM clean between our test runs.
Managing html fixtures like this has been a big win. In a recent team retrospective, consensus was “I can’t imagine doing it any other way.”
Building fixtures for tests is often simple as writing a 5-line controller spec. Maintaining fixtures just means re-running the specs, or perhaps enhancing the specs with additional data. We don’t see any production bugs from fixture markup vs. real markup discrepancies, and we spend very little time dealing with the fixtures.
We sometimes forget to re-generate fixtures after we change our markup, but, by now, realizing our mistake is a reflex action.
Test speed was one of our concerns going into this. Loading the fixtures takes time, of course. In practice, the runtime hit is dwarfed by the time saved by automating fixtures.
We have 224 javascript tests, and 200 of them load an html fixture. Our suite runs in 38 seconds in Chrome on a 2.4 GHz Core 2 iMac. (Other browsers are considerably slower). Of that 38 seconds, 9 seconds are spent loading fixtures and re-binding events.
We end up running our tests on fairly large DOM. We could probably save some running time by more narrowly focusing the markup that we generate and load. However, we haven’t felt it worth the additional development overhead. Additionally, running tests on our full DOM has an advantage – it exposes event handlers that conflict with each other.
Nested describes are another great feature of Jasmine. It can be difficult, though, to keep track of where you’ve called loadFixture() within your hierarchy of describes. We sometimes found that we were calling loadFixure() twice in the same spec. To prevent that, we now keep track of how many times loadFixture() is called within a single test, and fail the test if the count exceeds 1.
At one point, we noticed that our javascript suite was consuming hundreds of megabytes of memory as it executed. We traced the problem to two jQuery plugins. Each time we loaded a fixture, the plugins claimed memory that never got freed. Now we mock out the plugins where we’re not explicitly testing them.
“Anyone have good strategies for using S3 as a content delivery network for static files?”
Using S3 as a CDN is pretty common. S3 is certainly cheap, and fairly easy to set up. However, latency can be large – S3 isn’t built to act as a CDN, so the performance can be lacking. In addition, you need to work out your pathing in your CSS files to find background images correctly. Relative paths are a common technique here.
The performance of files in your public directory is much better. Amazon’s Cloudfront is another (expensive) option.
Note observation #4 in this blog article: link
“I can’t get ImageMagick to work on Snow Leopard. What gives?”
A brief look online shows several step-by-step instructions. It’s unclear what this particular problem is about.
“After upgrading to the latest version of Mocha, any_instance doesn’t clean up after itself. Why?”
Mocha‘s any_instance stubbing is one of the few features that distinguish Mocha from other mock frameworks.
One suggestion was to update rspec as well.
“How do I set up a Compact Privacy Policy? How does this funky syntax work?”
Definition of compact privacy policy
More info about the CPP workflow
In short, work with your client’s legal staff to get a policy and convert it to P3P and compact privacy policy form.
“Heroku 1.5.3 isn’t letting me use heroku rake commands. What can I do?”
Upgrade to Heroku 1.5.6.
EY Cloud’s slave database functionality is broken right now. It’s supposed to be fixed this afternoon.
Amazon restricts you to 20 EBS volumes/EC2 instances per account by default. The trick here is that deleting volumes does not immediately free up space. Volumes stay in a ‘Deleting volume’ state for an indefinite amount of time before they are truly free, making it hard to allot space for them. Finding these deleting instances can be a real challenge – the AWS API can find them, but not the EY cloud GUI.
If you need to manipulate AWS credentials for EY Cloud, it’s fairly easy to go to the machine and find the appropriate file – /etc/.mysql.backups.yml
Yesterday I modified an old app to turn off a feature for records that were more than a certain number of days old. This is an old enough app that the test data is still yaml fixture files, and in the fixtures I see records with attributes like so:
published_at: 2006-08-25 00:00:00 -07:00
created_at: 2006-08-24 00:00:00 -07:00
updated_at: 2006-08-25 00:00:00 -07:00
And of course, the tests code also makes assumptions about absolute dates…
get :show, :year => '2006', :month => '08', :day => '24'
assert_not_nil assigns[:article]
That makes adding tests that need to check for dates relative to the current date very difficult to integrate into the test suite. It also means that over time, your fixture data gets “stale” and you don’t have any data that appears to be recent, which may or may not matter to your application.
One way to deal with this is to mock time in your test code, so that tests always run at the same effective time, so the hardwired absolute dates in fixtures are always relatively the same age. I think that’s a good Plan B, but I prefer Plan A.
Plan A is to create your test data with dates and times relative to when the tests are being run. You can do that in yaml fixtures by embedding ruby in the yaml:
published_at: <%= 2.days.ago.to_s(:db) %>
created_at: <%= 3.days.ago.to_s(:db) %>
updated_at: <%= 2.days.ago.to_s(:db) %>
And in your test code, be flexible too:
pub = article.published_at.to_date
get :show, :year => pub.year.to_s, :month => pub.month.to_s, :day => pub.mday.to_s
assert_not_nil assigns[:article]
Keep your test dates relative, and a year from now you’ll thank yourself (or somebody else will).