Javascript tests are good, but manually-maintained HTML fixtures are painful. It’s time consuming to keep fixture markup in sync with the actual markup produced by your app. Despite best efforts, deviations arise, leading to bugs and false positives in tests.
For the past few months on Mavenlink, we’ve been pre-generating real-life fixture markup and making it available in our javascript tests. Results have been positive.
The basic approach is simple:
1) Pre-generate real-life markup using any convenient mechanism.
2) When javascript tests run, load the pre-baked markup into the DOM for access by JS code.
For #1, we use a small set of RSpec controller specs that exist solely to generate fixtures. For #2, we retrieve fixture markup from our Jasmine server via ajax, and inject it into the DOM with jQuery. While our approach leans on the various bits of our stack, any part of it could be swapped out to adapt to different tools.
Big thanks to Jonathan Barnes for the code that got us started.
Generating the fixtures
Taking a closer look, here’s an RSpec test that generates an HTML fixture file named ‘workspace-page’:
describe WorkspacesController do
it "generates a workspace page" do
@workspace = create_workspace
log_in @workspace.creator
get :show, :id => @workspace.to_param
response.should be_success
save_fixture(html_for('body'), 'workspace-page')
end
end
We create a workspace, hit the WorkspaceController‘s show method, and save the full text of the response’s <body> element to a file. This spec lives in a file with others like it, separate from the real controller specs. We have about a dozen of these specs, all in the same file.
We added save_fixture() and html_for() to ControllerExampleGroup to help with fixture generation. This gist has implementation details.
Our javascript tests sometimes require that we load different versions of the same page. We generate a different fixture for each version, giving them meaningful names like ‘busy-workspace-page’ and ‘empty-workspace-page’.
When we change markup consumed by our javascript tests, we re-run the controller specs that generate fixtures. Changes are picked up the next time we run our javascript tests. Fixture generation is hooked into continuous integration, so our javascript tests see the latest markup when running in CI.
Loading fixtures into the javascript test DOM
On the javascript side, fixture loading is handled by the code in this gist. Of note:
We use two methods for making fixture content available to our tests. loadFixture() inserts the fixture markup into the DOM for use by code that accesses the DOM. readFixture() returns fixture content as text; we use it to test our ajax callback methods.
Within the same test run, we cache fixture text in javascript to avoid multiple requests to the server for the same markup. Across test runs, we ensure our markup is fresh by appending a cache busting timestamp to our request path.
Ready to test!
Here’s how we typically use loadFixture() within our Jasmine tests:
describe('the status module', function() {
it('switches tabs', function() {
spec.loadFixture('workspace-page');
var $tabContainer = $('#jasmine_content').find('.tab-container');
expect($tabContainer).toHaveSelectedTab('team');
$tabContainer.find('li[tab=schedule]').click();
expect($tabContainer).toHaveSelectedTab('schedule');
});
});
loadFixture() inserts markup into the #jasmine_content div. Then we examine the DOM, do stuff to it, and inspect it again. toHaveSelectedTab() is a custom Jasmine matcher. Jasmine matchers are super easy to write. We love them.
You may be wondering how we’ve established our event bindings. jQuery’s html() method, used within loadFixture(), executes any scripts in the markup passed to it. If you’ve bound events in your fixture markup, within a $(document).ready() or not, they will execute when you call loadFixture(). This is really nice, because it means the same mechanism used for binding events in real life is also used within tests, keeping our tests that much closer to reality.
If, on the other hand, you bind events not within fixture markup, but instead within a script loaded once per suite globally, you’ll have to invoke your event binding code explicitly before each test.
Speaking of event bindings, you’ll need to clean them up properly between tests. For example, jQuery live events are bound to document. We clear them in a global beforeEach():
beforeEach(function() {
$('#jasmine_content').empty();
spec.clearLiveEventBindings();
jasmine.Clock.useMock();
});
spec.clearLiveEventBindings = function() {
var events = jQuery.data(document, "events");
for (prop in events) {
delete events[prop];
}
};
Any events bound on elements within #jasmine_content are cleared out by jQuery when we call $('#jasmine_content').empty(), which also wipes the DOM clean between our test runs.
Results
Managing html fixtures like this has been a big win. In a recent team retrospective, consensus was “I can’t imagine doing it any other way.”
Building fixtures for tests is often simple as writing a 5-line controller spec. Maintaining fixtures just means re-running the specs, or perhaps enhancing the specs with additional data. We don’t see any production bugs from fixture markup vs. real markup discrepancies, and we spend very little time dealing with the fixtures.
We sometimes forget to re-generate fixtures after we change our markup, but, by now, realizing our mistake is a reflex action.
Test speed was one of our concerns going into this. Loading the fixtures takes time, of course. In practice, the runtime hit is dwarfed by the time saved by automating fixtures.
We have 224 javascript tests, and 200 of them load an html fixture. Our suite runs in 38 seconds in Chrome on a 2.4 GHz Core 2 iMac. (Other browsers are considerably slower). Of that 38 seconds, 9 seconds are spent loading fixtures and re-binding events.
We end up running our tests on fairly large DOM. We could probably save some running time by more narrowly focusing the markup that we generate and load. However, we haven’t felt it worth the additional development overhead. Additionally, running tests on our full DOM has an advantage – it exposes event handlers that conflict with each other.
Other gotchas
Nested describes are another great feature of Jasmine. It can be difficult, though, to keep track of where you’ve called loadFixture() within your hierarchy of describes. We sometimes found that we were calling loadFixure() twice in the same spec. To prevent that, we now keep track of how many times loadFixture() is called within a single test, and fail the test if the count exceeds 1.
At one point, we noticed that our javascript suite was consuming hundreds of megabytes of memory as it executed. We traced the problem to two jQuery plugins. Each time we loaded a fixture, the plugins claimed memory that never got freed. Now we mock out the plugins where we’re not explicitly testing them.
